New enhancements for moving data between Amazon FSx for Lustre and Amazon S3

Amazon FSx for Lustre works natively with Amazon S3, making it easy for you to process cloud data sets with high performance file systems. When linked to an S3 bucket, an FSx for Lustre file system transparently presents S3 objects as files and allows you to write results back to S3. Until today, Amazon FSx provided file system commands to export new or changed files from FSx to S3, but these commands did not copy file permissions or provide the ability to monitor or cancel the transfer.

Today, Amazon FSx for Lustre is launching an AWS application programming interface (API) that allows you to easily export files from FSx to S3. With the new API, you can easily initiate, monitor, and cancel writing new or changed files to S3. Since it’s an AWS-native API, you can use it to easily orchestrate data export tasks from cloud native workflows such as Lambda-based serverless applications. In addition to transferring file data and file permissions, this API also allows you to transfer symbolic links, file ownership metadata, and file time stamps to S3. File permissions and other file metadata are stored in S3 in the same format used by AWS DataSync and AWS Storage Gateway.

Additionally, FSx has quadrupled the speed at which file metadata is imported from S3, allowing you to launch S3-backed FSx file systems up to four times faster.



https://aws.amazon.com/about-aws/whats-new/2019/12/new-enhancements-moving-data-amazon-fsx-lustre-amazon-s3/