Google Cloud Dataflow

Simplifies deployment and management of complete ML pipelines with ready-to-use patterns for personalized recommendations, fraud detection, threat prevention

Preprocess data and focus on transforming data without writing complex code or managing underlying libraries

Make predictions to generative AI models with streaming data

Use streaming AI and ML to power real-time ML models with low latency predictions, inferences, personalization, threat detection, and fraud prevention

Build streaming AI with Vertex AI, Gemini models, and Gemma models

Bring in streaming data for real-time analytics and operational pipelines with integration of streaming data sources like Pub/Sub, Kafka, CDC events, user clickstream, logs, and sensor data

Measures billing based on total resources used by jobs instead of data processed volume

Combines regular and preemptible VMs for batch processing with about 40% discount on vCPU and memory costs and delays job execution within 6-hour window

Ability to write streaming data to multiple storage locations in parallel

Process and write data immediately into BigQuery, Google Cloud Storage, Spanner, Bigtable, SQL stores, Splunk, Datadog and more for rapid analysis and decision-making

Write processed data from BigQuery back to OLTP stores for fast lookups and serving end users

Uses open source Apache Beam SDK to enable advanced streaming use cases at enterprise scale with rich capabilities for state and time transformations

Highly scalable feature that shuffles data outside of workers for batch pipelines with volume-based pricing

Enable parallel ingestion and transformation of multimodal data like images, text, and audio with specialized feature extraction and unified representation

Integrated User Defined Function builder to add custom logic to template jobs

Iteratively build pipelines with the latest data science frameworks and deploy with the Dataflow runner

Provides visibility into Dataflow usage and answers who did what, where, and when for better governance

Supports configurable persistent disk allocation for worker VMs

Allows creating snapshots of pipeline state for recovery and management

Allows observing data at each pipeline step for debugging and monitoring

Offers recommendations for job improvements based on pipeline analysis

UI for easy cost estimation and tracking of Dataflow job expenses

Provides job graphs, execution details, metrics, autoscaling dashboards, and logging capabilities

Automatically identifies performance bottlenecks in data pipelines

Enhance MLOps and ML job efficiency with GPU support and right-fitting capabilities

Moves streaming shuffle and state processing out of worker VMs into the Dataflow service backend for improved performance

Premium data processing platform that builds on Dataflow with improvements in resource utilization and distributed diagnostics using Data Compute Units (DCUs)

Scales to 4K workers per job with automatic scaling for optimal resource utilization in both batch and streaming pipelines

Encrypts data in use with confidential VM support for enhanced security

Allows customers to manage their own encryption keys for data protection

Ability to turn off public IPs for enhanced security

Integrates with VPC Service Controls for network security and access control

Pre-designed blueprints for stream and batch processing optimized for efficient CDC and BigQuery data integration that can be deployed in a few clicks without code

Visual UI for building and running Dataflow pipelines in the Google Cloud console without writing code

Real-time analysis of user interactions on websites and apps for personalization, A/B testing, and funnel optimization

Replicates Google Cloud logs to third-party platforms like Splunk for near real-time log processing with centralized management and compliance capabilities

Analyzes current market, customer, and competitor data for quick informed decisions with omnichannel marketing, CRM personalization, and competitive intelligence