Part 14. Why Most Data Pipelines Fail Before Processing Their First Record
Ask a group of engineers what a data pipeline is, and you will probably hear a list of technologies.
“Apache Beam.”
“Dataflow.”
“Kafka.”
“Pub/Sub.”
“Airflow.”
“Cloud Composer.”
Those answers are not wrong.
But they all skip the most important part.
A data pipeline is not a product.
It is not a service.
It is not a programming framework.
A data pipeline is simply a promise.
A promise that information created in one place will arrive somewhere else correctly.
That promise sounds surprisingly simple.
In practice, it is one of the hardest promises modern software systems try to keep.
Every day, millions of companies move customer registrations, payments, mobile events, invoices, website clicks, medical records and IoT measurements through data pipelines.
Most of those pipelines appear to work.
Until the day they do not.
And the frightening part is that many pipelines fail long before anyone notices.
Success Is Easy When Nothing Goes Wrong
Imagine a mobile application.
A customer opens it.
The application records a login event.
The event reaches BigQuery.
A dashboard updates.
Everyone is happy.
From the outside, the pipeline looks almost magical.
One event enters.
One event appears.
But real systems rarely behave so politely.
Networks become unavailable.
Servers restart.
APIs return temporary errors.
Files arrive late.
Cloud services retry requests.
Applications accidentally send the same event twice.
Users lose internet connectivity for several hours before reconnecting.
Suddenly the pipeline is no longer moving one clean stream of information.
It is negotiating with reality.
The architecture must survive that reality.
Pipelines Do Not Move Data
This sentence sounds strange at first.
After all, moving data is exactly what pipelines seem to do.
Yet experienced architects think differently.
Pipelines do not move information.
They move events.
That distinction matters.
Suppose an online store receives one thousand purchases.
The pipeline should not ask,
“How do I copy one thousand rows?”
It should ask,
“How do I preserve one thousand business events without changing their meaning?”
An order placed yesterday should remain yesterday’s order even if the upload happens today.
A payment processed twice should still represent only one payment.
A cancelled shipment should not disappear simply because another system overwrote its status.
Good pipelines preserve business history.
Bad pipelines merely copy databases.
Every Source Lies Differently
One of the biggest surprises for engineers building their first cloud platform is discovering that every data source behaves differently.
A CRM exports CSV files every night.
A payment provider exposes an API.
Google Analytics produces events.
AppsFlyer delivers attribution data.
IoT sensors stream measurements every second.
ERP systems sometimes rewrite historical records.
Nothing is consistent.
Nothing follows the same schedule.
Nothing guarantees the same quality.
Yet business users expect a single dashboard containing perfectly synchronized information.
This is why modern pipelines spend surprisingly little time transporting data.
Most of their work involves translating different realities into one consistent language.
Why Simple Scripts Eventually Collapse
Almost every successful pipeline begins as a small script.
Download a file.
Transform a few columns.
Load the result into BigQuery.
Done.
The script works perfectly.
For months.
Then someone asks for another source.
Another transformation.
Another destination.
Error handling appears.
Retry logic appears.
Logging appears.
Scheduling appears.
Notifications appear.
Soon the original script contains thousands of lines of code.
Nobody planned for this.
It simply happened because success created more work.
This is one reason Google Cloud offers so many orchestration services.
Not because Python stopped working.
Because successful businesses eventually outgrow simple automation.
Reliability Is More Important Than Speed
Many teams become obsessed with latency.
Can we reduce processing from five minutes to thirty seconds?
Can we process events instantly?
Sometimes those questions matter.
Often they do not.
Imagine a finance department receiving yesterday’s revenue report.
Would processing it in five seconds instead of five minutes change any decision?
Probably not.
Now imagine silently losing two percent of yesterday’s invoices.
That changes everything.
Business users almost always prefer complete information slightly later rather than incomplete information immediately.
Reliable pipelines build trust.
Fast pipelines build impressive demonstrations.
Architects know which one matters in production.
The Most Dangerous Pipeline Is the One That Looks Healthy
Modern cloud platforms provide beautiful dashboards.
Green indicators.
Successful jobs.
Healthy services.
Everything appears normal.
Meanwhile, thousands of records quietly disappear because an external API changed one field name.
Nobody notices.
The pipeline never technically failed.
It simply produced incorrect results.
This is one of the biggest differences between software engineering and data engineering.
Applications usually fail loudly.
Pipelines often fail silently.
Silent failures are far more expensive.
Because they corrupt decisions rather than interrupt systems.
Pipelines Are About Contracts
Experienced Google Cloud architects rarely describe pipelines as sequences of services.
Instead, they describe contracts.
A source promises to produce events.
A transformation promises not to change business meaning.
A destination promises durable storage.
Monitoring promises visibility.
Every stage has one responsibility.
When those responsibilities remain clear, replacing individual technologies becomes surprisingly easy.
Cloud Run may replace a virtual machine.
Dataflow may replace custom code.
Pub/Sub may replace direct HTTP communication.
The architecture survives because the contracts remain unchanged.
The services become implementation details.
Why Google Cloud Built an Entire Ecosystem
Many newcomers wonder why Google Cloud includes so many data services.
Pub/Sub.
Dataflow.
Cloud Run Jobs.
Composer.
Workflows.
Dataform.
Datastream.
Storage Transfer Service.
At first glance, the platform appears unnecessarily complicated.
The reality is different.
Each service solves a different failure mode.
Some guarantee delivery.
Some transform data.
Some coordinate execution.
Some recover after interruption.
Some maintain dependencies.
Google did not build many services because moving data is easy.
Google built many services because moving data reliably is incredibly difficult.
Architect’s Notebook
Pipelines do not exist to copy data.
They exist to preserve business events while systems constantly fail around them.
Every design decision should answer one question.
“If something breaks tomorrow, will we still trust yesterday’s data?”
If the answer is yes, the pipeline is probably well designed.
If the answer is no, adding more cloud services will not solve the underlying problem.
Closing Thought
The first question in pipeline architecture should never be,
“Should we use Pub/Sub or Dataflow?”
It should be,
“What promise are we making about this data?”
Because once that promise becomes clear, choosing the right Google Cloud services becomes much easier.