You can pick the Apache Spark infrastructure that you want your AWS Glue scripts to run on by choosing a Glue version. Glue development endpoints that were created without specifying a Glue version are defaulted to Glue version 0.9. Glue development endpoints with Glue version of 1.0 will run on Apache Spark 2.4.3. In addition to supporting the latest version of Spark, you will also have the ability to choose between Python 2 and Python 3. Previously, you were able to choose Glue versions only for Glue ETL jobs.
To learn more about how you can take advantage of this feature, please visit our documentation.
This feature is now available in all the AWS regions where AWS Glue is available except AWS GovCloud (US-East) and AWS GovCloud (US-West).