Build Your Own Analytics Pipeline: Stop Feeding Vendors and Start Collecting Raw Data in BigQuery

The era of carefree client-side analytics is over. If your data collection strategy still relies on just putting a Google Tag Manager script on your website and hoping for the best, I have bad news for you: you are making business decisions based on hallucinations.

AdBlock extensions cut up to 30–40% of client traffic. Browser restrictions (like ITP in Safari) are destroying third-party cookies. And when you finally realize you need raw data to build proper attribution or train ML models, SaaS vendors send you a bill for tens of thousands of dollars for an Enterprise plan, where the only killer feature is the “Export to BigQuery” button.

Let’s break down the architecture of a reliable Server-Side data collection pipeline that bypasses ad blockers and costs less than a cup of coffee per hundred gigabytes.

The Problem: Why Build Our Own Pipeline?

Businesses need raw logs in their Data Warehouse (DWH). It seems simple: why not just send an HTTP request from the frontend directly to the database?

Because it is architectural suicide. To write data directly to BigQuery from the client, you would have to hardcode your Service Account Key (GCP access key) into the website’s JavaScript. Within hours, scrapers would find this key on GitHub or in the page’s source code, and hackers would mine cryptocurrency using your Google Cloud budget or simply delete all your tables.

We need a server-side middleware. It must be incredibly cheap, scale from 0 to 1000 RPS (requests per second) in milliseconds, and guarantee that not a single JSON packet from a user is lost, even if the database itself temporarily goes offline.

Architecture: The Four Horsemen of Reliability

The solution is built on four technologies connected into a sequential pipeline.

1. The Client (JavaScript + navigator.sendBeacon)

We do not use standard fetch or XMLHttpRequest. Users are impatient: they click the “Buy” button and immediately close the tab. A fetch request is aborted at that moment.
We use the Web API navigator.sendBeacon(url, data). It places the network request in the browser’s system queue, guaranteeing the packet is sent even if the page is already dead. The script sends data to a subdomain of your website (e.g., track.your-domain.com), which allows it to bypass 99% of AdBlock filters since the request looks like First-Party traffic.

2. Face Control (Google Cloud Run + F#)

The request arrives at our microservice. Cloud Run is a Serverless environment for Docker containers. You only pay for the exact milliseconds your code processes the request.
Here we use F#. Why not Python or Node.js? Because we need mathematical strictness. F#, with its Strong Typing and Algebraic Data Types, validates the incoming JSON on the fly. If a bot sends garbage instead of the expected event structure, F# kills the request during deserialization, preventing bad data from leaking into the data warehouse. The service attaches a server timestamp and IP address to the event, and then passes it along.

3. The Shock Absorber (Google Cloud Pub/Sub)

The F# service does not write data to the database. Databases are slow; they lock, update, and crash. The service drops the validated JSON into Pub/Sub (a distributed message queue) and instantly returns an HTTP 200 OK to the frontend.
Pub/Sub acts as a shock absorber. If a million users visit your site in one hour (for example, during Black Friday), Cloud Run and Pub/Sub will swallow this traffic spike without a single error.

4. Direct Inject (BigQuery Pub/Sub Subscription)

Previously, you had to write another script to read the Pub/Sub queue and run INSERT statements in BigQuery. Now, GCP has a native integration: Pub/Sub Subscription to BigQuery.
You simply configure a rule in the interface (or define it in Terraform): “Take every message from this topic and silently put it into this BigQuery table.” Google manages the streaming write under the hood, with no code required on your part.

Math and Economics: Winning in Numbers

Let’s calculate the costs for a high-load project that generates 22 million events per day.

Input data:

  • Monthly event volume: 22,000,000 × 30 = 660,000,000

  • Average weight of one JSON event: 1 KB

  • Total data volume: 660 GB per month.

SaaS Costs (Amplitude / PostHog Enterprise):
At this volume, processing costs plus the fee to enable raw data export to the cloud will range from $8,000 to $12,000 per month, depending on the vendor’s sales manager.

Custom Architecture Costs (GCP):

  1. Cloud Run: A highly optimized F# container consumes minimal CPU and RAM. For 660 million requests, the cost will be around $30 – $50.

  2. Pub/Sub: Data transfer costs $40 per 1 TB. (0.66 TB × $40 = $26.40)

  3. BigQuery Ingestion: A direct subscription from Pub/Sub to BQ is billed at $50 per 1 TB. (0.66 TB × $50 = $33.00)

  4. BigQuery Storage: The first 10 GB are free, then $0.02 per GB. That’s about $13.

Total: The entire enterprise-grade server infrastructure will cost you approximately $100 – $120 per month.

Benefits in numbers:
You save over $100,000 a year on subscriptions, gain 100% control over raw data in real-time (latency from a browser click to a database row is less than 2 seconds), and recover up to 30% of traffic “lost” to AdBlockers.

The Harsh Reality: Trade-offs

This architecture is highly effective, but it comes with engineering risks:

  1. Schema Drift: If a web analyst adds a new tracking field on the frontend (e.g., user_discount), but this column does not exist in the BigQuery table, Pub/Sub will fail to write the row. You need to configure Schema Evolution mechanisms or store unrecognized fields in a JSON type column.

  2. Dead-Letter Queues (DLQ): Failed messages (e.g., data type mismatches) do not just disappear. They pile up in a special dead-letter queue. You will need to allocate data engineering time to periodically clear these backlogs and fix validation bugs in your F# code.

Similar Posts