Amazon SageMaker Now Works With Amazon FSx For Lustre and Amazon EFS, Accelerating And Simplifying Model Training

Until today, Amazon SageMaker transparently downloaded a full training set from Amazon S3 to local file storage at the start of a training job, when using the File input mode. Now with Amazon FSx for Lustre, customers can accelerate their File mode training jobs by avoiding the initial Amazon S3 download time. When Amazon FSx for Lustre file system is linked to Amazon S3 buckets, it automatically copies objects from Amazon S3 to the file system when objects are accessed for the first time. The same FSx file system can also be used across multiple SageMaker jobs, preventing repeated downloading of common objects.

Also until today, customers could only use Amazon SageMaker with training sets stored on Amazon S3. Now, customers can also use training sets that are stored on Amazon EFS. Amazon SageMaker interacts directly with Amazon EFS, eliminating the need to copy data sets from Amazon EFS to Amazon S3 for use with Amazon SageMaker.

Most Amazon SageMaker built-in machine learning algorithms support EFS and FSx for Lustre as input data source. This feature is available in all regions where the respective file systems are available. For details on region availability please check the AWS region table.



https://aws.amazon.com/about-aws/whats-new/2019/08/amazon-sagemaker-works-with-amazon-fsx-lustre-amazon-efs-model-training/