AWS Glue

Uses generative AI to quickly identify and resolve issues in Spark jobs by analyzing job metadata, execution logs, and configurations for root cause analysis and recommendations

Create ETL jobs using natural language descriptions. Automatically generates Apache Spark code that can be customized, tested, and deployed as production jobs

Generative AI automatically analyzes Spark jobs and generates upgrade plans to newer versions, reducing time and effort to keep jobs modern, secure and performant

Flexible execution job class for non-time-sensitive workloads that reduces costs up to 35% for preproduction jobs, testing, and data loads

Interactive point-and-click visual interface for cleaning and normalizing data without writing code. Includes over 250 built-in transformations to combine, pivot, and transpose data

Deduplicates and finds imperfect matching records using machine learning without requiring ML expertise. Learns from labeled examples to identify matches across databases

Enables developers to scale existing Python code and popular Python libraries using Ray.io open-source compute framework. Serverless with no infrastructure management required

Natively supports Apache Hudi, Apache Iceberg, and Delta Lake for transactional consistency in Amazon S3 based data lakes with read, insert, update, and delete operations

Automatically measures, monitors, and manages data quality in data lakes and pipelines. Computes statistics, recommends rules, monitors quality metrics, and alerts on deterioration

Defines, identifies, and processes sensitive data in pipelines and data lakes. Remediates PII and sensitive data by redacting, replacing, or reporting on personally identifiable information

Serverless feature that validates and controls evolution of streaming data using registered Apache Avro schemas with integration for Kafka, Amazon MSK, Kinesis, Apache Flink, and AWS Lambda

Integrates with Git version control system including GitHub and AWS CodeCommit for maintaining change history and applying DevOps practices. Works with automation tools like Jenkins and AWS CodeDeploy

Serverless notebooks with minimal setup in AWS Glue Studio for quick developer onboarding. Built-in interface for Interactive Sessions with ability to save and schedule notebook code as AWS Glue jobs

Allows data engineers to write and share business-specific Apache Spark logic as reusable visual transforms available across all jobs in account

Serverless feature for interactively developing ETL code using IDEs or notebooks of choice. Allows engineers to explore, experiment on, and process data interactively with custom readers, writers, and transformations

Crawlers connect to data stores, determine schema using prioritized classifiers, and create metadata in Data Catalog. Can be run on schedule, on demand, or triggered by events

Persistent metadata store for all data assets with table definitions, job definitions, schemas, automatic statistics computation, partition registration, and comprehensive schema version history

Visual interface for authoring highly scalable ETL jobs without becoming Apache Spark expert. Automatically generates code in Scala or Python for extract, transform, and load processes

AWS Glue is accessible in next generation of Amazon SageMaker for managing and building workloads in one place with cost-effective serverless data integration

Connects multiple data sources including DynamoDB, SaaS applications (Salesforce, SAP, ServiceNow), and self-managed databases to Amazon Redshift or SageMaker data lakehouse without operational overhead

Provides access to analytics on transactional data by replicating data from Oracle, SQL Server, MySQL, or PostgreSQL to Amazon Redshift within minutes with data filtering capabilities

All logs and notifications are pushed to Amazon CloudWatch for centralized monitoring and alerting

Jobs can be invoked on schedule, on demand, or based on events. Supports parallel job execution, inter-job dependencies, bad data filtering, and automatic retries

AWS Glue 5.0+ provides table, column, and row level permissions for Apache Spark jobs accessing Apache Iceberg, Apache Hudi, and Delta tables for simplified security and governance

Continuously consumes data from streaming sources like Amazon Kinesis and Amazon MSK, cleans and transforms it in-flight. Supports event data enrichment, aggregation, and complex analytics operations