Google Cloud Data Platform Architecture: A Practical Guide for Real‑World Production Systems – Google Cloud Solutions. From idea to cloud architecture.

Introduction. The Architecture of Failure: Why Monoliths Lose Money

In today’s mid‑market e‑commerce platforms and B2B SaaS solutions, data is not just an analytical asset — it is the operational core of the business. Our case is based on a classic scaling problem: the company begins receiving an irregular, unpredictable stream of critical webhooks from external systems such as Stripe, Salesforce, and logistics providers.

The symptoms of a degrading monolithic system appear clearly:

during peak loads like Black Friday or large marketing campaigns, the old architecture collapses;
incoming HTTP requests fail with timeouts and generate 5xx errors;
transactional data is lost permanently;
financial dashboards begin to show critical discrepancies with actual revenue;
cloud infrastructure costs grow unpredictably due to attempts to handle the load.

This situation requires a fundamental architectural shift. Trying to balance unpredictable traffic spikes with traditional compute resources means burning budget and engineering time on maintaining unstable infrastructure.

Engineering Goal: The Zero‑Ops Paradigm

To solve this business problem, we must design a new Engineering Blueprint for a resilient data pipeline. The target architecture must meet the following SLA and technical requirements:

Zero Data Loss — 100% delivery guarantee: every bit of information must be preserved.
Zero‑Ops — automatic scaling: the system must absorb traffic spikes instantly without DevOps involvement.
Data Quality — strict validation: protection against silent API changes on the partner side.
FinOps limits — predictable costs: transparent cost forecasting and automatic budget enforcement.

C4 Diagram: Conceptual Architecture of the Serverless Solution

To move away from the monolith, we adopt an event‑driven serverless architecture on Google Cloud Platform (GCP), deployed entirely through Infrastructure‑as‑Code (Terraform).

According to the C4 model, the solution is decomposed into isolated layers, each of which will be implemented in the following modules of this technical blueprint:

Security & Identity Layer: Access isolation through Workload Identity Federation (WIF) and Secret Manager, eliminating vulnerable static JSON keys.

Ingestion & Buffer Layer: Cloud Run acts as a webhook receiver, providing instant responses to partners and offloading traffic into the asynchronous Cloud Pub/Sub bus. The bus includes a Dead Letter Queue (DLQ) to protect storage from overload and isolate corrupted messages.

Storage & FinOps Layer: BigQuery with enforced partitioning and clustering to minimize scan costs. Data is written via the high‑performance Storage Write API.

DataOps Layer: GCP Dataform configured through Terraform. It handles transformations and quality control using SQLX Assertions, detecting duplicates and null values in raw data.

Observability & FinOps Automation: End‑to‑end monitoring via Cloud Monitoring with alerts for DLQ growth and 5xx errors, plus strict budget limits through Cloud Billing Budgets API.

CI/CD Pipeline: GitHub Actions YAML for seamless deployment of infrastructure and Docker images.

2. Security & Identity Layer: Zero‑Key Platform on GCP

Zero‑Ops begins with how the platform manages access to infrastructure and data. If the identity and permissions layer is designed incorrectly, every other engineering decision loses meaning: any leaked CI/CD key becomes a full project compromise, and any permission mistake exposes data to people who should not see it.

In the classic GCP model, infrastructure is managed through service account keys — static JSON files stored in GitHub Secrets, GitLab Variables, Jenkins Credentials, Ansible Vault, and similar systems. This is convenient at the start but does not scale:

keys live for years without rotation;
they are copied manually between environments;
they get lost, logged, or emailed;
they cannot be centrally controlled.

A Zero‑Ops architecture eliminates this model. The goal is a platform where no pipeline and no service stores GCP keys, and access is built through identity federation and short‑lived tokens.

2.1. Workload Identity Federation as the Core Pattern

Instead of storing GCP secrets in CI/CD, we use Workload Identity Federation (WIF). The idea is simple:

CI/CD platforms (GitHub, GitLab) issue OIDC tokens describing which repository and workflow is running.
GCP accepts these tokens and exchanges them for temporary access to a service account.
No JSON keys, no long‑lived secrets — only short‑lived tokens.

Architecturally, this works as follows:

A Workload Identity Pool is created in GCP — a space for external identities.
Inside it, a Provider is created for the specific CI/CD platform (e.g., GitHub OIDC).
Terraform defines the service account used for infrastructure deployment.
IAM bindings allow only specific repositories and workflows to “assume” this service account.
The CI/CD pipeline requests an OIDC token and uses WIF to obtain temporary access to GCP.

The key effect: CI/CD stores no GCP secrets. Even full access to repository settings will not reveal JSON keys.

2.2. Terraform Model for Security & Identity

2.2.1. Workload Identity Pool

hcl

resource "google_iam_workload_identity_pool" "ci_pool" {
  workload_identity_pool_id = "ci-pool"
  display_name              = "CI/CD Identity Pool"
  description               = "Federated identities for CI/CD pipelines"
}

This resource creates the base boundary for all external identities. It is not tied to a specific project or service — it is a platform‑level component that can be reused across multiple projects.

2.2.2. Workload Identity Provider (GitHub OIDC)

hcl

resource "google_iam_workload_identity_pool_provider" "github_provider" {
  workload_identity_pool_id          = google_iam_workload_identity_pool.ci_pool.workload_identity_pool_id
  workload_identity_pool_provider_id = "github-oidc"
  display_name                       = "GitHub Actions OIDC"

  oidc {
    issuer_uri = "https://token.actions.githubusercontent.com"
  }

  attribute_mapping = {
    "google.subject"       = "assertion.sub"
    "attribute.repository" = "assertion.repository"
    "attribute.workflow"   = "assertion.workflow"
    "attribute.branch"     = "assertion.ref"
  }
}

This defines how GCP interprets GitHub OIDC tokens. attribute.repository, attribute.workflow, and attribute.branch allow precise rules: which repository, which workflow, and which branch may work with infrastructure.

2.2.3. Terraform Service Account

hcl

resource "google_service_account" "terraform_sa" {
  account_id   = "terraform-deployer"
  display_name = "Terraform Deployer"
}

This service account is the single entry point for infrastructure changes. All terraform plan/apply operations must run under it. This provides:

a single object for audit;
a single object for permission management;
a clear boundary between “who can change infrastructure” and “who can only read.”

2.2.4. WIF → Service Account Binding

hcl

resource "google_service_account_iam_binding" "terraform_wif_binding" {
  service_account_id = google_service_account.terraform_sa.name
  role               = "roles/iam.workloadIdentityUser"

  members = [
    "principalSet://iam.googleapis.com/${google_iam_workload_identity_pool.ci_pool.name}/attribute.repository/my-org/infra-repo"
  ]
}

This means: “Any OIDC token from ci-pool with attribute.repository == "my-org/infra-repo" may use the terraform-deployer service account.”

Effects:

only one specific repository can deploy infrastructure;
copying Terraform code into another repository does not grant access to GCP;
permissions are tied to identity, not secrets.

2.2.5. Minimal Roles for Terraform

hcl

resource "google_project_iam_member" "terraform_infra_admin" {
  project = var.project_id
  role    = "roles.editor"
  member  = "serviceAccount:${google_service_account.terraform_sa.email}"
}

In real platforms, roles.editor is usually replaced with a set of narrower roles (run.admin, pubsub.admin, bigquery.admin, storage.admin, etc.). The principle remains: the Terraform account receives only the permissions required for infrastructure management and does not access data unless necessary.

2.3. GitHub Actions: Access to GCP Without Keys

The CI/CD pipeline does not store JSON keys, does not use GOOGLE_CREDENTIALS, and does not read GCP secrets from GitHub. Instead, it requests an OIDC token and uses WIF to obtain temporary access.

yaml

name: Terraform Apply

on:
  workflow_dispatch:
  push:
    branches:
      - main

jobs:
  terraform:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read

    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Authenticate to GCP via WIF
        uses: google-github-actions/auth@v2
        with:
          workload_identity_provider: "projects/123456789/locations/global/workloadIdentityPools/ci-pool/providers/github-oidc"
          service_account: "terraform-deployer@my-project.iam.gserviceaccount.com"

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v3

      - name: Terraform Init
        run: terraform init

      - name: Terraform Apply
        run: terraform apply -auto-approve

Key points:

permissions.id-token: write — GitHub allows the workflow to obtain an OIDC token.
google-github-actions/auth exchanges the token for temporary access.
No GCP keys are stored in secrets.

Security implications:

even full access to repository settings cannot reveal a GCP key;
access to GCP exists only during the job and only within the short‑lived token;
GCP audit logs show all operations as actions of terraform-deployer, not random users.

2.4. Secret Manager: Secrets Exist, but Not GCP Keys

Zero‑Ops does not eliminate secrets — it eliminates GCP secrets outside the cloud.

External API keys, payment provider tokens, and webhook keys must live in Secret Manager, not in CI/CD.

hcl

resource "google_secret_manager_secret" "stripe_api_key" {
  secret_id = "stripe-api-key"

  replication {
    automatic = true
  }
}

resource "google_secret_manager_secret_version" "stripe_api_key_v1" {
  secret      = google_secret_manager_secret.stripe_api_key.id
  secret_data = var.stripe_api_key
}

Cloud Run services that process webhooks receive roles/secretmanager.secretAccessor and read secrets directly from Secret Manager. CI/CD does not know, store, or log these secrets.

2.5. Organization Policies: Enforcing a Ban on Keys

To prevent the platform from “sliding back” to JSON keys, an organization‑level policy is enabled:

hcl

resource "google_org_policy_policy" "disable_sa_keys" {
  name   = "constraints/iam.disableServiceAccountKeyCreation"
  parent = "organizations/${var.org_id}"

  spec {
    rules {
      enforce = true
    }
  }
}

After this:

attempts to create keys in the GCP console fail;
attempts to create keys via gcloud iam service-accounts keys create are blocked;
even habitual attempts to return to the old model are technically impossible.

2.6. Result of the Security & Identity Layer

This layer provides:

a platform without GCP service account keys outside the cloud;
CI/CD that works through identity federation, not secrets;
a single Terraform service account with minimal permissions;
centralized storage of external secrets in Secret Manager;
an organizational ban on legacy authentication models.

This is the foundation of the Zero‑Ops Data Platform: infrastructure managed as code, access built through identity rather than secrets, and security that requires no ongoing manual maintenance.

3. Ingestion & Buffer Layer: How the Platform Survives Traffic Spikes Without Losing Events

The ingestion layer is the first line of defense. This is where external webhooks arrive: payments, delivery statuses, order updates, CRM events. If this layer is designed poorly, the platform breaks exactly here: during peak load it stops responding on time, partners start retrying requests, and some events are simply lost.

A Zero‑Ops architecture solves this through a clear pattern:

entry point — Cloud Run;
buffering and delivery guarantee — Pub/Sub;
protection from corrupted messages — Dead Letter Queue (DLQ).

Below is a step‑by‑step explanation of why this pattern works and how it is implemented in Terraform, code, and configuration.

3.1. Why an Independent Ingestion Layer Is Necessary

If webhooks are handled directly by the service that writes to the database or BigQuery, that service inevitably becomes a bottleneck:

under peak load it processes requests more slowly;
partners see timeouts and errors;
retry logic on the partner side activates;
load increases even more, pushing the system into a feedback loop of overload.

A dedicated ingestion layer solves this by:

separating request reception from processing;
ensuring the entry point responds quickly and predictably;
moving heavy processing into asynchronous components;
preventing the platform from becoming hostage to traffic spikes.

The idea is simple: everything coming from outside first goes into a buffer, and only then gets processed.

3.2. Why Cloud Run Is the Entry Point

Cloud Run is a fully managed container runtime that scales automatically. For Zero‑Ops, this matters for three reasons:

No servers. No need to manage VMs, clusters, autoscalers, or health checks. You deploy a container, and the platform handles scaling.
Direct HTTP entry. Cloud Run accepts HTTP requests directly, which is ideal for webhooks: external systems simply send POST requests to a Cloud Run URL.
Automatic scaling. When load increases, Cloud Run starts more instances; when load decreases, it scales down — without engineer involvement.

In the ingestion layer, Cloud Run has one job: accept the request quickly, validate it minimally, and hand off responsibility to Pub/Sub.

Terraform: Deploying the Cloud Run Service

hcl

resource "google_cloud_run_service" "webhook_receiver" {
  name     = "webhook-receiver"
  location = var.region

  template {
    spec {
      containers {
        image = "${var.region}-docker.pkg.dev/${var.project_id}/webhooks/webhook:latest"

        env {
          name  = "PUBSUB_TOPIC"
          value = google_pubsub_topic.ingestion_topic.name
        }
      }
    }
  }

  traffic {
    percent         = 100
    latest_revision = true
  }
}

Key points:

image — the container that receives webhooks;
PUBSUB_TOPIC — the Pub/Sub topic where events will be published;
traffic — all traffic goes to the latest revision.

Cloud Run must be allowed to publish messages to Pub/Sub:

hcl

resource "google_project_iam_member" "run_pubsub_publisher" {
  project = var.project_id
  role    = "roles/pubsub.publisher"
  member  = "serviceAccount:${google_cloud_run_service.webhook_receiver.status[0].service_account_email}"
}

Without this role, Cloud Run cannot publish messages.

3.3. Webhook Contract: What Cloud Run Actually Does

Imagine the platform receives payment events. A webhook arrives as an HTTP POST with a JSON body.

Cloud Run must:

accept the request;
check that key fields exist (event_id, type, created_at);
pack the event into a Pub/Sub message;
publish it to the queue;
return HTTP 200 to the partner.

It must not:

write directly to BigQuery;
perform complex transactions;
call external APIs;
execute heavy business logic.

All of that happens later in subscription handlers.

Example Cloud Run Container (Python)

python

import json
from flask import Flask, request
from google.cloud import pubsub_v1

app = Flask(__name__)
publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path("my-project", "ingestion-topic")

@app.post("/webhook")
def webhook():
    data = request.json

    # basic validation
    if not isinstance(data, dict):
        return "bad request", 400
    if "event_id" not in data or "type" not in data:
        return "bad request", 400

    payload = json.dumps(data).encode("utf-8")
    publisher.publish(topic_path, payload)

    return "ok", 200

This shows:

minimal format validation;
JSON packaging;
publishing to Pub/Sub;
fast response to the partner.

3.4. Why Pub/Sub Is a Mandatory Buffer

If Cloud Run writes directly to BigQuery or another downstream service, it becomes a bottleneck again: under peak load it waits for storage responses, increasing response time for partners.

Pub/Sub solves this:

Cloud Run publishes a message and responds immediately;
Pub/Sub stores messages in a queue;
subscription handlers process messages at a controlled rate;
if downstream services are temporarily unavailable, messages are not lost.

Terraform: Main Pub/Sub Topic

hcl

resource "google_pubsub_topic" "ingestion_topic" {
  name = "ingestion-topic"
}

This topic is the central buffer for all incoming events.

3.5. Why Dead Letter Queue (DLQ) Is Required

In real systems, some messages will be “bad”:

unexpected JSON structure;
missing required fields;
invalid values;
logic the current handler cannot process.

If such messages are retried endlessly, they:

clog the queue;
slow down processing of valid events;
create noise in logs.

DLQ is a separate topic where messages go after a defined number of failed attempts. This provides:

a way to analyze problematic events separately;
no infinite retries;
predictable system behavior during errors.

Terraform: DLQ Topic

hcl

resource "google_pubsub_topic" "dlq_topic" {
  name = "ingestion-dlq"
}

Terraform: Subscription with DLQ

hcl

resource "google_pubsub_subscription" "ingestion_sub" {
  name  = "ingestion-sub"
  topic = google_pubsub_topic.ingestion_topic.name

  dead_letter_policy {
    dead_letter_topic     = google_pubsub_topic.dlq_topic.name
    max_delivery_attempts = 5
  }

  retry_policy {
    minimum_backoff = "10s"
    maximum_backoff = "600s"
  }

  ack_deadline_seconds = 20
}

Key points:

max_delivery_attempts = 5 — after five failures, the message goes to DLQ;
retry_policy — defines retry intervals;
dead_letter_topic — the DLQ destination.

3.6. Full Data Flow

Putting everything together:

External service sends HTTP POST to Cloud Run.
Cloud Run validates fields, publishes to ingestion-topic, returns 200 OK.
Pub/Sub stores the message.
The subscription (ingestion-sub) delivers the message to handlers.
If processing succeeds — the message is acknowledged.
If processing fails — the message is retried.
After 5 failures — the message moves to ingestion-dlq.

Example Subscription Handler

python

from google.cloud import pubsub_v1

subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path("my-project", "ingestion-sub")

def process_event(raw_data: bytes):
    import json
    event = json.loads(raw_data)
    # business logic here

def callback(message):
    try:
        process_event(message.data)
        message.ack()
    except Exception:
        message.nack()

subscriber.subscribe(subscription_path, callback=callback)

3.7. Actual Role of Each Component

Component	Responsibility
Cloud Run (webhook)	Accepts HTTP request, validates fields, publishes to Pub/Sub, responds to partner
Pub/Sub ingestion	Buffers events, guarantees delivery, decouples reception from processing
Subscription (ingestion-sub)	Delivers messages to handlers, manages retries and DLQ routing
DLQ topic	Stores messages that cannot be processed, enables separate analysis
Consumer	Implements business logic: storage writes, downstream calls, aggregations

This separation allows:

independent scaling of request intake (Cloud Run);
independent scaling of processing (consumer);
independent handling of errors (DLQ).

3.8. Why This Pattern Matches Zero‑Ops

For a CTO or Lead Data Engineer, this layer provides key advantages:

no manual server management — Cloud Run and Pub/Sub are fully managed;
no blocking logic at the entry point — webhooks always receive fast responses;
a buffer between external systems and internal processing — the platform does not break under spikes;
predictable error handling — DLQ prevents the system from getting stuck on problematic events;
clear separation of responsibilities — each component does its job.

This is Zero‑Ops in the ingestion layer: the platform survives traffic spikes, does not lose events, and does not require constant manual intervention to “clear the queue” or “restart the service.”

4. Data Warehouse & FinOps Layer: BigQuery as the Core of the Platform and Strict Cost Control

The ingestion layer solves the problem of incoming pressure, but it does not answer the main question: where the data is stored, how durability is guaranteed, and how the platform remains cost‑efficient at scale. These tasks are handled by the storage layer — BigQuery — and a set of engineering decisions around it: Storage Write API, strict schemas, partitioning, clustering, and FinOps constraints.

Below is a practical, step‑by‑step breakdown of why and how this layer is built.

4.1. Why BigQuery in a Zero‑Ops Architecture

BigQuery is not just an analytical warehouse. In a Zero‑Ops platform, it serves as:

the primary storage for events coming from the ingestion layer;
the consistency point for downstream services;
the foundation for Dataform models and SQL transformations;
a system that requires no DevOps maintenance (no servers, no clusters, no manual scaling).

Why BigQuery fits Zero‑Ops:

No servers. No CPU, RAM, autoscalers, or cluster management — Google handles everything.
Guaranteed durability. Data is stored in a highly resilient distributed system.
Storage Write API enables streaming ingestion without batch delays.
Partitioning and clustering keep costs predictable.
IAM separation ensures ingestion writes while analysts read.

4.2. Why Storage Write API Instead of INSERT or Streaming API

BigQuery supports several write methods:

Method	Problems
SQL INSERT	slow, expensive, unsuitable for high load
Streaming API	high cost, risk of data loss under overload
Storage Write API	streaming, high throughput, low cost, delivery guarantee

Storage Write API is the modern way to write data into BigQuery:

uses gRPC;
supports batching;
guarantees delivery;
costs less than Streaming API;
handles high throughput.

In a Zero‑Ops architecture, this is the only correct choice.

4.3. Terraform: Creating the Dataset and Tables

First, create a dataset for raw events:

hcl

resource "google_bigquery_dataset" "raw_events" {
  dataset_id                  = "raw_events"
  location                    = var.region
  delete_contents_on_destroy  = false
  description                 = "Raw ingestion events from Pub/Sub"
}

Now create the events table. It must be partitioned by the event date, not the ingestion date:

hcl

resource "google_bigquery_table" "events" {
  dataset_id = google_bigquery_dataset.raw_events.dataset_id
  table_id   = "events"

  time_partitioning {
    type  = "DAY"
    field = "event_date"
  }

  clustering = ["event_type"]

  schema = <<EOF
[
  {"name": "event_id", "type": "STRING", "mode": "REQUIRED"},
  {"name": "event_type", "type": "STRING", "mode": "REQUIRED"},
  {"name": "event_date", "type": "DATE", "mode": "REQUIRED"},
  {"name": "payload", "type": "JSON", "mode": "NULLABLE"},
  {"name": "received_at", "type": "TIMESTAMP", "mode": "REQUIRED"}
]
EOF
}

Why this design:

partitioning by event_date → cheap date‑range queries;
clustering by event_type → cheap type‑specific queries;
payload as JSON → flexibility without losing structure;
received_at → precise ingestion timestamp.

4.4. How Data Reaches BigQuery: Storage Write API

The subscription handler (consumer) reads messages from Pub/Sub and writes them to BigQuery via Storage Write API.

Example (Python):

python

from google.cloud import bigquery_storage_v1
import json
import datetime

bqs = bigquery_storage_v1.BigQueryWriteClient()

parent = bqs.table_path("my-project", "raw_events", "events")

def write_event(event):
    row = {
        "event_id": event["event_id"],
        "event_type": event["type"],
        "event_date": event["created_at"].split("T")[0],
        "payload": json.dumps(event),
        "received_at": datetime.datetime.utcnow().isoformat()
    }

    proto_rows = bigquery_storage_v1.types.ProtoRows()
    proto_rows.serialized_rows.append(
        bigquery_storage_v1.types.ProtoRows.serialize(row)
    )

    request = bigquery_storage_v1.types.AppendRowsRequest(
        write_stream="projects/my-project/datasets/raw_events/tables/events/_default",
        rows=proto_rows
    )

    bqs.append_rows(request)

Key points:

writing to the default write stream → streaming ingestion;
ProtoRows → fast binary serialization;
no SQL → cheaper and faster.

4.5. FinOps: How We Control BigQuery Costs

BigQuery can be cheap or extremely expensive — architecture determines everything. A Zero‑Ops platform must be cost‑predictable.

We use three mechanisms:

1. Partitioning

Queries scan only relevant partitions. A query for “yesterday” scans exactly one day.

2. Clustering

Queries filtered by event_type scan only relevant clusters.

3. Budget Alerts

Terraform:

hcl

resource "google_billing_budget" "bq_budget" {
  billing_account = var.billing_account_id
  display_name    = "BigQuery Budget"

  amount {
    specified_amount {
      currency_code = "USD"
      units         = "300"
    }
  }

  threshold_rules {
    threshold_percent = 0.5
  }

  threshold_rules {
    threshold_percent = 0.9
  }
}

This provides:

alert at 50% spend;
alert at 90% spend;
ability to stop heavy pipelines when limits are exceeded.

4.6. Why BigQuery + Storage Write API Are Ideal for Zero‑Ops

From a CTO / Lead Data Engineer perspective:

no servers → no DevOps load;
no batch delays → data appears immediately;
no data loss → Storage Write API guarantees delivery;
predictable costs → partitioning + clustering;
simple ingestion pipeline → ingestion → Pub/Sub → Storage Write API.

BigQuery becomes the core of the platform: the ingestion layer guarantees delivery, and the warehouse guarantees durability, availability, and cost predictability.

5. DataOps Layer: Dataform, SQLX Models, and Real‑Time Data Quality Control

The Data Warehouse layer solves storage and cost, but it does not solve data quality, structure, transformation, validation, cataloging, or controlled schema evolution. These tasks belong to the DataOps layer — the set of processes and tools that turn raw events into reliable, validated, analytics‑ready models.

In a Zero‑Ops architecture, DataOps is built around Dataform — a declarative tool for managing SQL transformations in BigQuery. It works like Terraform, but for data: models are defined as code, tested as code, deployed through CI/CD, and stored in Git.

Below is a step‑by‑step breakdown of why and how this layer is built.

5.1. Why Dataform in a Zero‑Ops Platform

Raw events in BigQuery are only the beginning. To make data useful, it must be:

normalized;
enriched;
aggregated;
validated;
aligned to a stable schema;
made available for BI, ML, and downstream services.

Doing this manually — through SQL scripts, cron jobs, Airflow, or Dataflow — quickly leads to chaos: scripts live in different places, schemas evolve without control, tests are missing, and errors are discovered weeks later.

Dataform solves this:

all models are defined as code (SQLX);
dependencies are declared explicitly;
data quality tests run automatically;
all transformations run inside BigQuery, with no servers;
the entire DataOps workflow lives in Git and deploys through CI/CD.

This fully aligns with Zero‑Ops: no servers, no manual pipeline management, no hidden scripts.

5.2. Why SQLX Instead of Plain SQL

SQLX is an extension of SQL that adds:

declarative dependency definitions;
automatic DAG generation;
data quality tests;
model parameters;
automatic materialization of tables and views.

Example: instead of writing SQL and manually creating a table, we write a SQLX model:

sql

config {
  type: "table",
  schema: "analytics",
  name: "orders_daily",
  description: "Daily aggregated orders"
}

select
  order_id,
  customer_id,
  date(created_at) as order_date,
  amount
from
  raw_events.events
where
  event_type = "order_created"

Dataform automatically:

creates analytics.orders_daily;
tracks dependencies;
rebuilds the model when upstream tables change;
runs tests.

5.3. Terraform: Deploying the Dataform Project

Dataform is a GCP resource that can be managed via Terraform.

hcl

resource "google_dataform_repository" "repo" {
  name         = "zeroops-dataform"
  project      = var.project_id
  region       = var.region
  display_name = "Zero-Ops Dataform Repository"
}

Create a workspace:

hcl

resource "google_dataform_workspace" "workspace" {
  name       = "main"
  repository = google_dataform_repository.repo.name
}

Connect a GitHub repository:

hcl

resource "google_dataform_repository_git_remote_settings" "git" {
  repository     = google_dataform_repository.repo.name
  url            = "https://github.com/my-org/dataform.git"
  default_branch = "main"
}

Dataform will now automatically pull code from GitHub.

5.4. Structure of a Dataform Project

Typical structure:

Код

dataform/
  definitions/
    orders_daily.sqlx
    customers.sqlx
    revenue_by_day.sqlx
  includes/
    macros.sqlx
  tests/
    orders_daily_tests.sqlx
  dataform.json

dataform.json:

json

{
  "warehouse": "bigquery",
  "defaultSchema": "analytics",
  "assertionSchema": "assertions"
}

5.5. Example SQLX Model: Normalizing Events

Raw events contain a lot of noise. Below is a model that normalizes order_created events:

sql

config {
  type: "table",
  schema: "analytics",
  name: "orders_normalized",
  description: "Normalized order events"
}

select
  payload.order_id as order_id,
  payload.customer_id as customer_id,
  payload.amount as amount,
  payload.currency as currency,
  date(payload.created_at) as order_date,
  timestamp(payload.created_at) as created_at
from
  raw_events.events
where
  event_type = "order_created"

Here:

payload is the JSON field from raw events;
the model converts JSON into structured fields;
the result is a normalized table.

5.6. Example SQLX Aggregation: Daily Revenue

sql

config {
  type: "table",
  schema: "analytics",
  name: "revenue_daily"
}

select
  order_date,
  sum(amount) as total_revenue,
  count(*) as orders_count
from
  analytics.orders_normalized
group by
  order_date

This model:

aggregates data by day;
produces a table for BI dashboards;
rebuilds automatically when upstream models change.

5.7. Data Quality Tests (Assertions)

Dataform allows writing tests directly in SQLX.

Example: ensure order_id is not null.

sql

config {
  type: "assertion",
  schema: "assertions",
  name: "orders_order_id_not_null"
}

select
  order_id
from
  analytics.orders_normalized
where
  order_id is null

If rows are returned, the test fails.

Example: ensure amount > 0.

sql

config {
  type: "assertion",
  schema: "assertions",
  name: "orders_amount_positive"
}

select
  amount
from
  analytics.orders_normalized
where
  amount <= 0

These tests run automatically on every deployment.

5.8. CI/CD for Dataform: Automatic Model Deployment

Example GitHub Actions workflow:

yaml

name: Dataform Deploy

on:
  push:
    branches:
      - main

jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read

    steps:
      - uses: actions/checkout@v4

      - name: Authenticate to GCP
        uses: google-github-actions/auth@v2
        with:
          workload_identity_provider: ${{ secrets.WIF_PROVIDER }}
          service_account: ${{ secrets.DATAFORM_SA }}

      - name: Install Dataform CLI
        run: npm install -g @dataform/cli

      - name: Deploy Dataform
        run: dataform run

Dataform CLI:

pulls models from Git;
builds the DAG;
rebuilds models;
runs tests;
writes results to BigQuery.

5.9. Why Dataform Is Ideal for Zero‑Ops

From an architectural perspective:

no servers — Dataform runs entirely in BigQuery;
no manual ETL scripts — everything is defined as code;
no chaos — DAG and dependencies are managed automatically;
no hidden errors — data quality tests run on every deployment;
no uncontrolled schema changes — everything is versioned in Git;
no DevOps load — CI/CD manages the entire process.

The DataOps layer transforms raw events into reliable analytical models ready for BI, ML, product reporting, and downstream services.

6. Observability Layer: How the Platform Sees Itself, Detects Problems, and Reacts Without Manual Micromanagement

A Zero‑Ops platform cannot rely on engineers’ intuition. It must know what is happening inside: how many events arrive, how many are lost, where latency grows, where errors appear, when BigQuery becomes too expensive, and when Cloud Run hits its limits.

These tasks are handled by the Observability layer — logging, metrics, dashboards, and alerts. Below is a step‑by‑step breakdown of how this layer is built using Cloud Logging and Cloud Monitoring.

6.1. Why Observability Is Required in a Zero‑Ops Platform

Without observability, the platform becomes a “black box”:

partners complain about errors while everything looks “green” internally;
analysts see data gaps but nobody knows where they originated;
DevOps learn about issues from chat messages instead of monitoring alerts.

The Observability layer enables the platform to:

see the event flow from ingestion to storage;
detect anomalies (error spikes, throughput drops, latency increases);
react automatically (alerts, notifications, protective mechanisms);
provide transparency for CTOs and Lead Data Engineers.

Zero‑Ops means the platform signals problems itself instead of waiting for someone to “notice something is wrong.”

6.2. Logging: What We Write to Cloud Logging and Why

Each layer writes its own logs:

Cloud Run — HTTP requests, webhook processing errors;
Pub/Sub — delivery errors, DLQ events;
BigQuery — query errors, heavy jobs;
Dataform — transformation and test results.

Logging must be intentional: we decide what to log and why.

Cloud Run: Logging Incoming Webhooks

Purpose:

see the volume of incoming requests;
see distribution by event types;
see validation and Pub/Sub publishing errors.

Example structured log:

json

{
  "severity": "INFO",
  "service": "webhook-receiver",
  "event_id": "evt_123",
  "event_type": "order_created",
  "source": "stripe",
  "received_at": "2026-09-07T19:22:00Z"
}

On error:

json

{
  "severity": "ERROR",
  "service": "webhook-receiver",
  "error": "pubsub_publish_failed",
  "event_id": "evt_123",
  "event_type": "order_created"
}

These logs allow:

building metrics on event volume;
identifying event types that fail most often;
correlating ingestion errors with downstream issues.

6.3. Metrics: Turning Logs Into Numbers

Logs alone are not useful for alerts. We need metrics that can be aggregated and analyzed.

Cloud Monitoring supports log‑based metrics — metrics derived from logs.

Example: Metric for Successful Webhooks

Terraform:

hcl

resource "google_logging_metric" "webhook_success_count" {
  name   = "webhook_success_count"
  filter = <<EOF
resource.type="cloud_run_revision"
logName="projects/${var.project_id}/logs/run.googleapis.com%2Fstdout"
jsonPayload.service="webhook-receiver"
jsonPayload.severity="INFO"
EOF

  metric_descriptor {
    metric_kind = "DELTA"
    value_type  = "INT64"
    unit        = "1"
  }

  label_extractors = {
    "event_type" = "jsonPayload.event_type"
  }
}

This metric:

counts successful webhooks;
enables graphs by event type;
shows how load changes over time.

Metric for Pub/Sub Publishing Errors

hcl

resource "google_logging_metric" "webhook_error_count" {
  name   = "webhook_error_count"
  filter = <<EOF
resource.type="cloud_run_revision"
jsonPayload.service="webhook-receiver"
jsonPayload.error="pubsub_publish_failed"
EOF

  metric_descriptor {
    metric_kind = "DELTA"
    value_type  = "INT64"
    unit        = "1"
  }
}

This metric:

tracks publishing errors;
enables alerts when errors increase.

6.4. SLO and Alerts: How the Platform Says “I’m Not OK”

Observability without alerts is just pretty graphs. Zero‑Ops requires the platform to notify engineers when something goes wrong.

We define SLOs (Service Level Objectives) and build alerts around them.

Example SLO: Success Rate of Webhooks

Goal: at least 99% of webhooks must be processed without errors.

We compute:

webhook_success_count
webhook_error_count
their ratio

Terraform: Alert for Error Growth

hcl

resource "google_monitoring_alert_policy" "webhook_errors_alert" {
  display_name = "Webhook Errors Alert"
  combiner     = "OR"

  conditions {
    display_name = "High webhook error rate"

    condition_threshold {
      filter = "metric.type=\"logging.googleapis.com/user/webhook_error_count\""
      duration = "300s"
      comparison = "COMPARISON_GT"
      threshold_value = 10

      aggregations {
        alignment_period = "60s"
        per_series_aligner = "ALIGN_RATE"
      }
    }
  }

  notification_channels = [var.notification_channel_id]
}

This alert:

triggers when errors exceed a threshold;
sends notifications (email, Slack, PagerDuty);
signals ingestion problems early.

6.5. Observability for Pub/Sub and DLQ

We must see what happens inside Pub/Sub:

number of undelivered messages;
number of DLQ messages;
retry frequency.

Cloud Monitoring provides built‑in metrics:

pubsub.googleapis.com/subscription/num_undelivered_messages
pubsub.googleapis.com/subscription/num_dead_letter_messages

Terraform: Alert for DLQ Growth

hcl

resource "google_monitoring_alert_policy" "dlq_growth_alert" {
  display_name = "DLQ Growth Alert"
  combiner     = "OR"

  conditions {
    display_name = "High DLQ messages"

    condition_threshold {
      filter = "metric.type=\"pubsub.googleapis.com/subscription/num_dead_letter_messages\" AND resource.label.subscription_id=\"ingestion-sub\""
      duration = "600s"
      comparison = "COMPARISON_GT"
      threshold_value = 100

      aggregations {
        alignment_period = "300s"
        per_series_aligner = "ALIGN_MEAN"
      }
    }
  }

  notification_channels = [var.notification_channel_id]
}

This alert:

triggers when DLQ grows rapidly;
indicates handler failures or schema changes;
allows early intervention.

6.6. Observability for BigQuery and Dataform

We must track:

data ingestion volume;
query volume;
query errors;
Dataform test failures.

Cloud Monitoring provides BigQuery metrics:

bigquery.googleapis.com/query/count
bigquery.googleapis.com/query/errors
bigquery.googleapis.com/storage/bytes_used

Dataform logs contain assertion results.

Log‑Based Metric for Failed Dataform Assertions

hcl

resource "google_logging_metric" "dataform_assertion_failed" {
  name   = "dataform_assertion_failed"
  filter = <<EOF
logName="projects/${var.project_id}/logs/dataform.googleapis.com%2Fassertions"
jsonPayload.status="FAILED"
EOF

  metric_descriptor {
    metric_kind = "DELTA"
    value_type  = "INT64"
    unit        = "1"
  }
}

Alert for Failed Assertions

hcl

resource "google_monitoring_alert_policy" "dataform_assertions_alert" {
  display_name = "Dataform Assertions Failed"
  combiner     = "OR"

  conditions {
    display_name = "Assertions failure"

    condition_threshold {
      filter = "metric.type=\"logging.googleapis.com/user/dataform_assertion_failed\""
      duration = "300s"
      comparison = "COMPARISON_GT"
      threshold_value = 0
    }
  }

  notification_channels = [var.notification_channel_id]
}

If any data quality test fails, the platform reports it immediately.

6.7. Dashboards: How CTOs and Lead Data Engineers See the Platform

A Cloud Monitoring dashboard provides a unified view:

incoming webhook volume by event type;
ingestion errors;
Pub/Sub queue size and DLQ size;
BigQuery storage usage;
failed Dataform tests;
SLO status.

This allows:

understanding platform health in 30 seconds;
identifying where issues originate;
making architectural decisions instead of chasing isolated errors.

6.8. Why the Observability Layer Is Critical for Zero‑Ops

Without observability, Zero‑Ops becomes an illusion: the platform appears automated, but nobody knows when or why it breaks.

Observability provides:

transparency — visibility across all layers;
early detection — alerts fire before users notice issues;
controlled response — engineers receive precise signals;
SLO/SLA foundation — formal definitions of “healthy platform.”

Zero‑Ops means the platform tells you when it is unhealthy and provides enough context to fix it without chaotic log digging.

7. FinOps Automation Layer: How the Platform Automatically Controls Budget, Limits Spending, and Prevents Financial Incidents

Zero‑Ops is not only about eliminating DevOps overhead — it also eliminates manual financial oversight. The platform must monitor its own spending, signal budget overruns, restrict dangerous operations, and prevent financial incidents, which occur in the cloud more often than technical failures.

The FinOps layer automates financial discipline: budgeting, cost monitoring, alerts, protective mechanisms, and architectural decisions that make spending predictable.

Below is a step‑by‑step breakdown of how this layer is built using Billing API, Budget Alerts, IAM constraints, and architectural patterns.

7.1. Why FinOps Is Required in a Zero‑Ops Platform

Cloud flexibility is powerful but dangerous: a single incorrect BigQuery query can cost hundreds of dollars, a careless Dataform job can rebuild an entire dataset, and a runaway consumer can generate millions of Pub/Sub messages.

If the platform does not control its own spending, it becomes unpredictable.

FinOps ensures:

prevention of financial incidents rather than reaction;
restriction of dangerous operations via IAM and architecture;
automatic signaling of budget overruns;
transparency for CTOs and Data Engineers;
predictable costs regardless of load.

Zero‑Ops means the platform monitors its own finances.

7.2. Why BigQuery Is the Main Source of Financial Risk

In data platforms, 70–90% of costs typically come from BigQuery. Reasons:

queries can scan terabytes of data;
analysts may run heavy jobs without understanding cost;
Dataform may rebuild models too frequently;
ingestion may generate excessive raw events.

FinOps begins with BigQuery.

7.3. Architectural Decisions That Reduce BigQuery Costs

Before automation, the warehouse must be designed correctly.

Partitioning by event date Reduces query cost dramatically.
Clustering by event type Queries scan only relevant clusters.
JSON payload instead of wide tables Wide tables → expensive scans. JSON → cheap targeted queries.
Storage Write API instead of Streaming API Streaming API is more expensive and less reliable.
Materializing Dataform models as tables only when needed Some models should be views, not tables.

These decisions reduce cost before automation even begins.

7.4. Budget Alerts: Automatic Budget Control

The platform must signal budget overruns before money is spent.

Terraform:

hcl

resource "google_billing_budget" "bq_budget" {
  billing_account = var.billing_account_id
  display_name    = "BigQuery Budget"

  amount {
    specified_amount {
      currency_code = "USD"
      units         = "300"
    }
  }

  threshold_rules {
    threshold_percent = 0.5
  }

  threshold_rules {
    threshold_percent = 0.9
  }
}

This provides:

alert at 50% spend — early warning;
alert at 90% spend — critical warning;
ability to automatically disable heavy jobs.

Budget Alerts are the first line of defense.

7.5. Automatic Shutdown of Heavy Dataform Pipelines

If the budget approaches its limit, the platform must stop heavy transformations automatically.

Example mechanism:

Budget Alert → Pub/Sub → Cloud Function → Dataform API

The Cloud Function switches Dataform into “tests‑only” mode:

python

from googleapiclient.discovery import build

def stop_dataform(event, context):
    dataform = build("dataform", "v1beta1")
    dataform.projects().locations().repositories().workspaces().execute(
        name="projects/my-project/locations/eu/repositories/zeroops-dataform/workspaces/main",
        body={"actions": ["disable_compilation"]}
    ).execute()

This protects the platform from runaway pipelines.

7.6. IAM Restrictions: Who Can Run Expensive Queries

FinOps is not only automation — it is architectural discipline.

We separate roles:

Role	Permissions
`bq.dataEditor`	write data
`bq.dataViewer`	read data
`bq.jobUser`	run queries
`bq.admin`	modify schemas, create tables

Analysts receive only:

bq.jobUser
bq.dataViewer

DataOps team receives:

bq.dataEditor
bq.jobUser

Infrastructure receives:

bq.admin

Analysts cannot:

create tables;
modify schemas;
run Dataform;
write raw events.

This reduces the risk of costly mistakes.

7.7. Limiting Query Cost via Query Settings

BigQuery allows setting query cost limits.

Example:

sql

DECLARE max_bytes INT64 DEFAULT 10000000000; -- 10 GB

SELECT *
FROM analytics.orders_daily
OPTIONS (max_bytes_billed = max_bytes)

If the query exceeds the limit, it fails.

In Dataform, this can be set globally:

json

{
  "defaultConfig": {
    "bigquery": {
      "maxBytesBilled": 10000000000
    }
  }
}

This protects the platform from accidental full‑table scans.

7.8. Automatic Control of Storage Write API

Storage Write API is cheap, but runaway ingestion can generate millions of rows.

We create a metric:

hcl

resource "google_logging_metric" "storage_write_volume" {
  name   = "storage_write_volume"
  filter = <<EOF
logName="projects/${var.project_id}/logs/bigquery.googleapis.com%2Fstorage_write"
EOF

  metric_descriptor {
    metric_kind = "DELTA"
    value_type  = "INT64"
  }
}

And an alert:

hcl

resource "google_monitoring_alert_policy" "storage_write_alert" {
  display_name = "High Storage Write Volume"

  conditions {
    display_name = "Write volume spike"

    condition_threshold {
      filter = "metric.type=\"logging.googleapis.com/user/storage_write_volume\""
      duration = "300s"
      comparison = "COMPARISON_GT"
      threshold_value = 1000000
    }
  }

  notification_channels = [var.notification_channel_id]
}

If ingestion generates too many events, the platform signals immediately.

7.9. Why the FinOps Layer Is Critical for Zero‑Ops

Without FinOps, Zero‑Ops becomes expensive automation: everything works, but costs grow uncontrollably.

FinOps provides:

predictable cost — CTOs know the platform’s financial profile;
automatic protection — runaway load does not cause financial incidents;
architectural discipline — roles and limits prevent mistakes;
transparency — dashboards show where costs grow;
control — the platform regulates its own financial behavior.

Zero‑Ops means the platform not only operates autonomously — it also manages its own money.

8. CI/CD Layer: Full Zero‑Ops Pipeline for Terraform, Dataform, and Cloud Run via Workload Identity Federation

The CI/CD layer is where infrastructure, data, and application code converge into a single controlled process. If this layer is poorly designed, the platform becomes chaotic: environments drift, deployments break each other, tests don’t run, and errors reach production.

A Zero‑Ops architecture requires CI/CD that:

stores no GCP secrets (no JSON keys);
authenticates via Workload Identity Federation;
deploys Terraform infrastructure as code;
deploys Dataform models as code;
builds and deploys Cloud Run containers;
runs data quality tests automatically;
behaves identically across dev/stage/prod;
requires no DevOps intervention.

Below is a complete, practical CI/CD layer: why it exists, how it works, and how each step is implemented.

8.1. Why CI/CD Is Required in a Zero‑Ops Platform

Without CI/CD, the platform becomes unpredictable:

infrastructure is changed manually in the console;
Dataform models are deployed locally;
containers are built on laptops;
data quality tests run “when someone remembers”;
environments drift over time.

CI/CD solves this:

everything that changes the platform goes through Git;
every deployment is reproducible;
every step is logged and transparent;
every component is tested automatically;
every deployment uses short‑lived WIF tokens.

Zero‑Ops means the platform evolves autonomously, without manual control.

8.2. Why Workload Identity Federation Is the Foundation of CI/CD

Traditional CI/CD stores GCP secrets in GitHub Secrets or GitLab Variables. This is dangerous:

secrets can be stolen;
secrets can be logged accidentally;
secrets may not be rotated;
secrets grant full project access.

WIF solves this completely:

CI/CD receives an OIDC token from GitHub/GitLab;
GCP validates the token and issues temporary access to a service account;
no secrets, no JSON keys.

This makes CI/CD secure and fully Zero‑Key.

8.3. Terraform Pipeline: Infrastructure as Code

Terraform manages:

Cloud Run
Pub/Sub
BigQuery
Dataform
IAM
Budget Alerts
Secret Manager

The pipeline must:

authenticate via WIF;
run terraform init;
run terraform plan;
run terraform apply;
publish logs and artifacts.

GitHub Actions: Terraform Apply

yaml

name: Terraform Apply

on:
  push:
    branches:
      - main

jobs:
  terraform:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read

    steps:
      - uses: actions/checkout@v4

      - name: Authenticate to GCP via WIF
        uses: google-github-actions/auth@v2
        with:
          workload_identity_provider: ${{ secrets.WIF_PROVIDER }}
          service_account: ${{ secrets.TERRAFORM_SA }}

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v3

      - name: Terraform Init
        run: terraform init

      - name: Terraform Plan
        run: terraform plan

      - name: Terraform Apply
        run: terraform apply -auto-approve

Key points:

id-token: write — GitHub issues an OIDC token;
auth@v2 — exchanges the token for temporary access;
no GCP secrets.

8.4. CI/CD for Cloud Run: Building and Deploying Containers

Cloud Run is the ingestion entry point. Containers must be built automatically.

GitHub Actions: Build & Deploy Cloud Run

yaml

name: Deploy Cloud Run

on:
  push:
    branches:
      - main

jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read

    steps:
      - uses: actions/checkout@v4

      - name: Authenticate to GCP
        uses: google-github-actions/auth@v2
        with:
          workload_identity_provider: ${{ secrets.WIF_PROVIDER }}
          service_account: ${{ secrets.CLOUDRUN_SA }}

      - name: Configure Docker
        run: gcloud auth configure-docker ${{ env.REGION }}-docker.pkg.dev

      - name: Build image
        run: |
          docker build -t ${{ env.REGION }}-docker.pkg.dev/${{ env.PROJECT }}/webhooks/webhook:latest .

      - name: Push image
        run: |
          docker push ${{ env.REGION }}-docker.pkg.dev/${{ env.PROJECT }}/webhooks/webhook:latest

      - name: Deploy Cloud Run
        run: |
          gcloud run deploy webhook-receiver \
            --image=${{ env.REGION }}-docker.pkg.dev/${{ env.PROJECT }}/webhooks/webhook:latest \
            --region=${{ env.REGION }} \
            --platform=managed

Key points:

container builds happen in CI/CD, not locally;
Cloud Run deploys via WIF;
images are stored in Artifact Registry.

8.5. CI/CD for Dataform: Data Transformations as Code

Dataform manages:

SQLX models;
DAG dependencies;
data quality tests;
table materialization.

The pipeline must:

authenticate via WIF;
install Dataform CLI;
run tests;
run transformations.

GitHub Actions: Dataform Deploy

yaml

name: Dataform Deploy

on:
  push:
    branches:
      - main

jobs:
  dataform:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read

    steps:
      - uses: actions/checkout@v4

      - name: Authenticate to GCP
        uses: google-github-actions/auth@v2
        with:
          workload_identity_provider: ${{ secrets.WIF_PROVIDER }}
          service_account: ${{ secrets.DATAFORM_SA }}

      - name: Install Dataform CLI
        run: npm install -g @dataform/cli

      - name: Run Dataform
        run: dataform run

Key points:

Dataform runs entirely in BigQuery;
data quality tests run automatically;
no Airflow, no servers.

8.6. CI/CD for Storage Write API Consumers

Pub/Sub consumers must:

be built as containers;
be deployed to Cloud Run or Cloud Functions;
be tested automatically.

Example Pipeline

yaml

name: Deploy Consumer

on:
  push:
    branches:
      - main

jobs:
  consumer:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read

    steps:
      - uses: actions/checkout@v4

      - name: Authenticate to GCP
        uses: google-github-actions/auth@v2
        with:
          workload_identity_provider: ${{ secrets.WIF_PROVIDER }}
          service_account: ${{ secrets.CONSUMER_SA }}

      - name: Build
        run: docker build -t consumer:latest .

      - name: Push
        run: docker push ${{ env.REGION }}-docker.pkg.dev/${{ env.PROJECT }}/consumers/consumer:latest

      - name: Deploy
        run: |
          gcloud run deploy consumer \
            --image=${{ env.REGION }}-docker.pkg.dev/${{ env.PROJECT }}/consumers/consumer:latest \
            --region=${{ env.REGION }}

8.7. Full CI/CD Flow: How Everything Works Together

Below is the actual sequence of actions for any change in the repository:

Step	What Happens	Why It Matters
1	Git push → main	CI/CD entry point
2	GitHub issues OIDC token	No secrets
3	WIF → temporary access	Zero‑Key
4	Terraform Apply	Infrastructure as code
5	Build & Deploy Cloud Run	Update ingestion layer
6	Build & Deploy Consumers	Update processing layer
7	Dataform Run	Update data models
8	Dataform Assertions	Data quality validation
9	Cloud Monitoring	Metrics and alerts
10	Dashboards update automatically	Full transparency

Every deployment is:

secure
reproducible
transparent
tested
free of DevOps overhead

8.8. Why This CI/CD Layer Matches Zero‑Ops

From a CTO / Lead Data Engineer perspective:

no GCP secrets — only temporary tokens;
no manual deployments — everything goes through Git;
no CI/CD servers — GitHub/GitLab handle everything;
no infrastructure drift — Terraform manages resources;
no data chaos — Dataform manages models;
no hidden errors — data quality tests run automatically;
no unpredictable changes — every deployment is logged and reproducible.

The CI/CD layer turns the platform into a fully managed system where every component evolves as code and the entire architecture lives in Git.

9. Final Architecture: A Unified Zero‑Ops Platform — Principles, Diagrams, and CTO Recommendations

Zero‑Ops Data Platform is not a collection of isolated services. It is a single engineering system where each layer plays a specific role, and all layers together deliver:

resilience
predictability
manageability
low cost
no DevOps overhead
full transparency for CTOs and Lead Data Engineers

This section assembles the entire architecture: how it works end‑to‑end, why it is stable, why it is cost‑efficient, why it is Zero‑Ops, and what decisions CTOs must make to keep it that way.

9.1. Architecture Diagram: End‑to‑End Data Flow

Below is the logical diagram of the entire platform, showing how data travels from external webhooks to analytical models and alerts.

Код

┌──────────────────────────┐
│   External Webhooks      │
│ (Stripe, CRM, Logistics) │
└──────────────┬───────────┘
               │ HTTP POST
               ▼
┌──────────────────────────┐
│      Cloud Run           │
│  Webhook Receiver        │
│  - fast response         │
│  - minimal validation    │
│  - publish to Pub/Sub    │
└──────────────┬───────────┘
               │ publish()
               ▼
┌──────────────────────────┐
│        Pub/Sub           │
│   ingestion-topic        │
│   - buffer               │
│   - delivery guarantee   │
└──────────────┬───────────┘
               │ subscribe()
               ▼
┌──────────────────────────┐
│   Consumers (Cloud Run)  │
│   - Storage Write API    │
│   - write to BigQuery    │
└──────────────┬───────────┘
               │ rows.append()
               ▼
┌──────────────────────────┐
│        BigQuery          │
│   raw_events.events      │
│   - partitions           │
│   - clustering           │
└──────────────┬───────────┘
               │ SQLX DAG
               ▼
┌──────────────────────────┐
│        Dataform          │
│   - SQLX models          │
│   - data tests           │
│   - materialization      │
└──────────────┬───────────┘
               │ BI / ML
               ▼
┌──────────────────────────┐
│   Analytics Layer        │
│   - dashboards           │
│   - ML pipelines         │
└──────────────┬───────────┘
               │ metrics
               ▼
┌──────────────────────────┐
│  Observability Layer     │
│  - Logging               │
│  - Monitoring            │
│  - SLO / Alerts          │
└──────────────┬───────────┘
               │ budget signals
               ▼
┌──────────────────────────┐
│     FinOps Layer         │
│  - Budget Alerts         │
│  - IAM restrictions      │
│  - auto‑stop pipelines   │
└──────────────────────────┘

This is not a set of services — it is a closed loop where each layer reinforces the others.

9.2. Architectural Principles: Why It Is Resilient

Zero‑Ops architecture stands on five fundamental principles:

1. Zero‑Key Security

No component stores GCP secrets. CI/CD, Cloud Run, Dataform — all authenticate via WIF.

This eliminates:

key leaks
CI/CD compromise
manual secret rotation

2. Asynchronous Ingestion

Load reception and data processing are fully decoupled.

Cloud Run → Pub/Sub → Consumers → BigQuery

This eliminates:

overload at the entry point
data loss during spikes
dependency on downstream services

3. Serverless Everywhere

No VMs, no clusters, no autoscalers.

Cloud Run, Pub/Sub, BigQuery, Dataform — all fully managed.

This eliminates:

DevOps overhead
manual scaling
capacity planning issues

4. Data as Code

Dataform turns SQL models into code:

versioning
tests
DAG
CI/CD

This eliminates:

transformation chaos
hidden errors
uncontrolled schema changes

5. FinOps Automation

The platform controls its own cost:

Budget Alerts
IAM restrictions
automatic Dataform shutdown
BigQuery limits

This eliminates:

financial incidents
runaway workloads
unpredictable spending

9.3. Responsibility Map of All Layers

Layer	Responsibility	Why It Matters
Security & Identity	WIF, IAM, Zero‑Key	Secure without secrets
Ingestion	Cloud Run → Pub/Sub	Lossless load intake
Buffer	Pub/Sub + DLQ	Delivery guarantee
Warehouse	BigQuery + Storage Write API	Durability + low cost
DataOps	Dataform + SQLX	Controlled transformations
Observability	Logging + Monitoring	Visibility + alerts
FinOps	Budget Alerts + IAM	Cost control
CI/CD	Terraform + Dataform + Cloud Run	Managed evolution

This table is the “passport” of the architecture.

9.4. How the Platform Responds to Incidents (Real Scenario)

Imagine a real incident: a partner starts sending corrupted webhooks.

What happens:

Cloud Run receives the webhook and publishes it to Pub/Sub.
Consumers try to process the event → error.
Pub/Sub retries 5 times → message goes to DLQ.
DLQ grows → Monitoring alert triggers.
CTO receives a notification.
Engineer inspects DLQ and sees event_id is missing.
Partner fixes the format.
DLQ stabilizes.
Platform continues operating without data loss.

Key points:

no component crashes
no data is lost
the platform signals the problem automatically
the issue is isolated to one layer

This is Zero‑Ops.

9.5. CTO Recommendations for Long‑Term Sustainability

Zero‑Ops is not only architecture — it is culture. To keep the platform stable:

Never allow service account keys “I need a JSON key for testing” is a red flag.
All changes must go through Git No manual edits in the GCP console.
All transformations must go through Dataform No ad‑hoc SQL scripts.
All containers must be built via CI/CD No local builds.
All alerts must be actionable If an alert doesn’t lead to action, it is noise.
All costs must be predictable If BigQuery becomes expensive, it is an architectural issue.
All layers must remain independent If one layer fails, others must continue working.

9.6. Why This Architecture Scales With Load

The platform scales horizontally:

Cloud Run scales with requests
Pub/Sub scales with messages
Storage Write API scales with streams
BigQuery scales with data volume
Dataform scales with DAG complexity

No component requires manual scaling.

9.7. Why This Architecture Is Resilient to Errors

Errors are isolated:

ingestion errors → DLQ
BigQuery errors → alerts
Dataform errors → assertions
CI/CD errors → fail‑fast
FinOps errors → budget alerts

Every layer has its own protection mechanism.

9.8. Why This Architecture Is Resilient to Change

The platform lives as code:

Terraform → infrastructure
Dataform → data
CI/CD → processes
IAM → access

Changes are:

versioned
tested
deployed automatically
safely rolled back

9.9. Final Conclusion

Zero‑Ops Data Platform is an architecture that:

does not require a DevOps team
does not lose data
does not fear traffic spikes
does not store secrets
does not allow financial incidents
does not break under errors
does not devolve into chaos
does not depend on manual control

It is a platform that operates autonomously. And evolves autonomously. And protects itself autonomously.