Amazon Web Services has expanded the geographical footprint of its serverless data integration platform, making AWS Glue 5.1 globally available following its deployment in the AWS European Sovereign Cloud Region. This rollout brings the latest iteration of the scalable data preparation and discovery service to strict regulatory environments, aligning regional operations with the complete feature set previously launched across standard commercial and AWS GovCloud infrastructure.
Core Engine Upgrades and Modernized Runtimes
At the heart of the AWS Glue 5.1 release is a modernization of its underlying execution engines and runtime environments. Data engineers and analytics teams working within sovereign boundaries can now leverage performance boosts, long-term support cycles, and modern programming language features without sacrificing regulatory compliance.
The upgrade package modernizes the foundational software stack with the following core components:
- Apache Spark 3.5.6: Provides foundational performance improvements, enhanced query planning, and modern execution capabilities for large-scale data processing workloads.
- Python 3.11: Delivers faster execution speeds and improved error handling for custom PySpark scripts and data transformation pipelines.
- Scala 2.12.18: Offers developers robust language compatibility and optimized compilation options for enterprise Scala-based applications.
Advanced Open Table Format Support
Modern data lakehouse architectures rely on open table formats to provide ACID transactions, time travel, and schema evolution on top of cloud object storage. AWS Glue 5.1 deepens its native integration with leading open-source table formats, ensuring organizations operating in the European Sovereign Cloud can build architectures using industry standards.
The platform introduces critical ecosystem updates across multiple storage frameworks:
- Apache Iceberg 1.10.0: Brings updated tooling and optimizations for managing massive analytical datasets.
- Apache Iceberg Format Version 3.0: Introduces advanced capabilities such as default column values, deletion vectors for optimized merge-on-read tables, multi-argument transforms, and granular row lineage tracking.
- Apache Hudi 1.0.2: Enhances high-frequency data ingestion and incremental processing performance.
- Delta Lake 3.3.2: Delivers enhanced reliability and compatibility for Delta-based lakehouse pipelines.
Granular Security and Access Control
Security and governance remain paramount for workloads deployed in sovereign cloud environments. AWS Glue 5.1 addresses these concerns by expanding the scope of AWS Lake Formation fine-grained access control. Previously restricted to read-only operations, security administrators can now enforce fine-grained access rules across write operations—encompassing both Data Manipulation Language (DML) and Data Definition Language (DDL) executions—for Spark DataFrames and Spark SQL.
Furthermore, the release introduces full-table access control capabilities within Apache Spark specifically for Apache Hudi and Delta Lake tables. These additions close historical governance gaps, providing a framework to secure data pipelines from ingestion through transformation and final storage.
Availability and Getting Started
With this regional expansion, AWS Glue 5.1 achieves universal availability across all AWS commercial regions, AWS GovCloud (US) regions, and the AWS European Sovereign Cloud. Development and data engineering teams can provision and manage workloads using standard automation and deployment channels.
Engineers can initialize and configure AWS Glue 5.1 workflows using the AWS APIs, AWS CLI, AWS SDK, or the visual interface of AWS Glue Studio. Detailed integration guides and comprehensive release notes are accessible via official AWS technical documentation channels.
Source: Original Article
