Should You Replace Apache Airflow with Google Cloud Workflows and Cloud Run?

Executive Summary & Diagnostic Context

Data engineering infrastructures default to Apache Airflow (managed on GCP as Cloud Composer) for orchestrating data pipelines. While Airflow remains the industry standard for Python-based Directed Acyclic Graphs (DAGs), its monolithic architecture introduces high baseline costs, continuous compute overhead, and maintenance friction. As serverless paradigms mature, replacing Airflow’s always-on control plane with the event-driven, scale-to-zero model of Google Cloud Workflows and Cloud Run Jobs is an increasingly viable architectural pivot.

This analysis deconstructs both paradigms strictly through engineering constraints, quantitative cost models, and empirical Developer Experience (DX) metrics. The objective is to determine when this migration is mathematically and operationally justified, and when it leads to structural failure.

Core Paradigms & System Decomposition

Understanding this shift requires deconstructing the underlying state machines and execution models.

  • Apache Airflow (Cloud Composer): A declarative, poll-based state machine. Logic is defined in Python. It relies on a centralized metadata database (PostgreSQL), a persistent scheduler, and distributed workers. It is highly optimized for predictable, batch-oriented, and historically dependent data processing (ETL/ELT).
  • Workflows + Cloud Run: An imperative, push-based orchestration mesh. Workflows maintains state transition logic via YAML or JSON, executing API calls or triggering containerized Cloud Run Jobs. It operates with microsecond latency, requires no persistent infrastructure, and is engineered for event-driven, highly concurrent microservice chaining.

Architectural Trajectories (From Simple to Complex)

In accordance with strict engineering discipline, we evaluate the serverless alternatives starting from the lowest architectural complexity to the highest.

Level 1: Cron-Triggered Cloud Run Jobs

  • Architecture: Google Cloud Scheduler directly invokes a standalone Cloud Run Job via an HTTP trigger.
  • Implementation: A single Docker container executes a self-contained script (e.g., fetching daily exchange rates from a REST API and writing to BigQuery).
  • Risks & Bottlenecks: Zero state management across multiple tasks. This architecture fails entirely if the pipeline requires sequence integrity (e.g., Step B must strictly execute only after Step A succeeds).

Level 2: Linear State Machines (Workflows + Cloud Run)

  • Architecture: Cloud Workflows acts as the sequential engine. It triggers Cloud Run Job A, waits for completion, parses the execution exit code, and conditionally triggers Cloud Run Job B.
  • Implementation: Complex dependencies are isolated within polyglot containers (e.g., Python 3.11 for data extraction, F# / .NET 8 for high-performance mathematical transformations), while Workflows handles the orchestration layer.
  • Risks & Bottlenecks: YAML string manipulation is rigid. Dynamic task generation (dynamic DAGs based on runtime variables) becomes highly unreadable and difficult to maintain when modeled in YAML.

Level 3: Asynchronous Event-Driven Mesh

  • Architecture: Eventarc detects an object creation in Cloud Storage, pushing an event to Workflows. Workflows executes a parallel fan-out (using parallel step iterations) to multiple concurrent Cloud Run Jobs for data normalization, aggregates the callback payloads, and executes a native BigQuery SQL merge via the Workflows GCP Connector.
  • Implementation: Zero polling. Compute is only utilized exactly when data arrives.
  • Risks & Bottlenecks: Holistic debugging becomes severely fragmented. A failure in one parallel branch requires custom error handling inside the YAML schema; there is no visual “Retry from failure point” UI inherent to this architecture.

Mathematical Cost Modeling & TCO Assessment

To evaluate financial viability, we construct a deterministic cost model comparing the Total Cost of Ownership (TCO) for a standard data team executing 100 ETL pipelines daily. Each pipeline consists of 3 discrete tasks, running for 5 minutes (300 seconds) each, requiring 1 vCPU and 2 GB RAM.

Model A: Managed Apache Airflow (Cloud Composer 3 – Small)

  • Baseline Infrastructure: Cloud Composer requires a persistent GKE cluster underlying architecture, Cloud SQL, and environment orchestration. The minimum monthly baseline for a continuously running small environment is approximately $350 to $450, assuming minimal idle capacity.
  • Compute Costs: Included within the baseline up to a threshold, scaling dynamically under load.
  • Total Monthly Cost: ~$400 (The baseline infrastructure tax dominates for sparse workloads).

Model B: Cloud Workflows + Cloud Run Jobs

  • Baseline Infrastructure: $0.00 (Scales completely to zero).
  • Cloud Run Jobs Pricing:
    • vCPU cost: $0.000018 per vCPU-second.
    • Memory cost: $0.000002 per GiB-second.
    • Cost per task execution: (300s * $0.000018) + (300s * $0.000004) = $0.0054 + $0.0012 = $0.0066.
  • Total Monthly Compute: 100 pipelines × 3 tasks × 30 days × $0.0066 = $59.40.
  • Cloud Workflows Pricing:
    • $0.01 per 1,000 internal steps; $0.025 per 1,000 external steps. The Google Cloud Free Tier covers the first 5,000 internal and 2,000 external steps.
    • Cost for this workload is statistically negligible (under $1).
  • Total Monthly Cost: ~$60.

Quantitative Verdict: For sparse, discrete workloads, migrating to Workflows and Cloud Run yields an 85% reduction in direct infrastructure costs. However, the intersection point shifts under extreme load. For dense workloads exceeding 50,000 tasks per month, Airflow’s persistent connection pooling, shared state memory, and centralized worker nodes become more computationally and financially efficient per execution.

Infrastructure & Networking Bottlenecks

A critical engineering constraint often overlooked during migration is network isolation.

  • Airflow: Operates within a dedicated VPC. It natively interacts with internal on-premise databases, Redis caches, or private IP instances without traversing the public internet.
  • Serverless (Cloud Run): Operates on Google’s shared serverless infrastructure. To achieve identical network isolation, Cloud Run requires configuring Direct VPC Egress or Serverless VPC Access Connectors. These connectors introduce base costs (minimum ~$15-$30/month) and potential cold-start latency, which structurally degrades the absolute “scale-to-zero” financial advantage.

Real-World Case Studies

Positive Case: Unpredictable Third-Party API Ingestion

  • Context: A marketing analytics infrastructure ingesting webhooks and API payloads from AppsFlyer and PostHog experienced highly unpredictable delivery schedules.
  • Legacy Airflow State: Relied on Airflow HttpSensor continuously polling endpoints. This consumed persistent worker slots and incurred high CPU utilization strictly for idle waiting.
  • Migration Implementation: Replaced with Eventarc pushing directly to Cloud Workflows, dynamically allocating Cloud Run Jobs scaled precisely to the payload size.
  • Empirical Results: Latency dropped from an average of 3 minutes (the Airflow poll interval) to ~200 milliseconds. Compute costs dropped by 92%. The push-based model eliminated all idle compute cycles.

Negative Case: ML Training Pipelines & Historical Backfilling

  • Context: A data science team required complex machine learning pipelines involving 50+ dependent steps, dynamic hyperparameter branching, and frequent business requirements to “re-run all transformations from January 1st to March 15th” (backfilling).
  • Migration Implementation: Attempted to model the pipeline using nested Workflows YAML and Cloud Run containers.
  • Failure Mechanics: Workflows lacks the fundamental concept of a logical_date or historical execution context. The YAML configuration ballooned to over 2,000 lines just to handle error states, retries, and manual date overrides. When failures occurred deep in the execution tree, engineers had to write custom API scripts to inject state back into the workflow to resume it.
  • Empirical Results: Catastrophic degradation in DataOps velocity. System observability was lost. The architecture was ultimately rolled back to Airflow, where historical backfilling and visual state management exist as native primitives.

Orchestration Patterns and Anti-Patterns

Verified Patterns:

  1. Delegate Compute, Centralize Orchestration: Workflows must strictly handle state transitions (if/then logic, parallel waits, retries). It should never perform data manipulation. Pass only minimal JSON metadata between steps; keep physical data payloads in Cloud Storage or BigQuery.
  2. Native Connectors over Containers: If a step merely executes a BigQuery SQL script or triggers a Dataflow job, utilize the Workflows HTTP GCP connector. Do not provision a Cloud Run Job solely to execute a bq query command. This eliminates container boot time and compute billing.
  3. Strict Idempotency: Cloud Run Jobs must be strictly idempotent. Because Workflows implements automatic retries on 5xx errors, containers must be designed to fail and restart without corrupting database state (e.g., utilizing SQL MERGE instead of raw INSERT).

Critical Anti-Patterns:

  1. YAML Programming: Attempting to write complex business logic, nested iterative loops, or heavy string manipulation inside Workflows YAML. If orchestration logic cannot be comprehended visually within 100 lines, it fundamentally belongs in Python (Airflow).
  2. Synchronous Waiting in Containers: Engineering a Cloud Run Job that triggers an external system and subsequently sleep()s while waiting for a response. This incurs active charges for vCPU and RAM while doing zero compute. Instead, utilize Workflows callback endpoints to suspend the workflow without compute charges until the external system replies.

Developer Experience (DX), CI/CD & Usability Metrics

The most significant risk in deprecating Airflow lies in the degradation of observability frameworks.

  • Deployment Cycle (CI/CD): Workflows and Cloud Run are vastly superior. A Workflows schema is updated via API instantaneously. Cloud Run image deployments take seconds. Cloud Composer requires CI/CD to push .py files to Cloud Storage, which the GKE cluster syncs sequentially—a cycle taking 1-3 minutes, with the structural risk of a Python parsing error crashing the entire scheduler.
  • Local Testing (The Bottleneck): Airflow allows engineers to execute DAGs locally via Docker Compose, perfectly mimicking production. Workflows relies heavily on GCP IAM permissions and direct API integrations. Local emulation is rudimentary, forcing developers to test in a live cloud sandbox environment, significantly increasing iteration time.
  • Observability: Airflow provides a unified UI featuring Gantt charts, historical success/failure trees, and one-click node restarts. Workflows relies entirely on Cloud Logging and Cloud Trace. Visualizing the historical state of a complex pipeline in Workflows requires building custom dashboards—a severe step backward in out-of-the-box DataOps usability.

Final Decision Matrix

The architectural decision is binary, governed by the workload’s core characteristics.

Retain Apache Airflow (Cloud Composer) if:

  1. The primary workload is historical batch processing reliant on execution dates and complex backfilling.
  2. Pipelines exceed 15-20 dependent nodes with dynamic, runtime conditional branching.
  3. The engineering team relies heavily on Python-native programmatic DAG generation.
  4. Visual observability and UI-driven task restarts are critical to your team’s SLAs.

Migrate to Workflows + Cloud Run if:

  1. Workloads are strictly event-driven (triggering immediately on webhooks, file uploads, or Pub/Sub events).
  2. Pipelines are simple, linear sequences or fan-out/fan-in microservice architectures.
  3. Execution patterns are sparse, highly variable, and the business requires eliminating the $400/month baseline idle tax.
  4. Execution engines require disparate languages (e.g., orchestrating Python, Rust, and F# in the same pipeline), easily compartmentalized into standard Cloud Run containers.

If business requirements mandate the removal of Airflow due to infrastructure cost constraints, but the core engineering problem consists entirely of SQL transformations executed inside a data warehouse, neither Workflows nor Cloud Run is the optimal architectural solution.

In this scenario, deploying dbt Core (scheduled minimally via Cloud Run) or utilizing Google Cloud Dataform is the correct systemic path. These tools solve the exact problem of Directed Acyclic Graphs (DAGs), dependency resolution, and state management natively for SQL. They bypass the Python-heavy overhead of Airflow and eliminate the YAML rigidity of Workflows, providing a purpose-built, highly optimized orchestration layer strictly engineered for BigQuery transformations.

Similar Posts