Backup Is Not Disaster Recovery: The Most Expensive Mistake CTOs Make
The Backup Worked. The Business Didn’t.
Most executives sleep better after hearing one sentence from their infrastructure team.
“Everything is backed up.”
It sounds reassuring.
Unfortunately, it answers the wrong question.
The question that actually matters is not whether the data exists somewhere else.
The real question is whether the business can continue operating after something important stops working.
This case began during what everyone expected to be a routine compliance exercise.
The company operated in twelve European countries, processed approximately 22 million financial transactions every month, and stored more than 240 TB of analytical data in BigQuery. Nearly every executive dashboard, fraud detection model, financial reconciliation report, and regulatory export depended on that platform.
The infrastructure looked mature.
Production datasets were protected.
Critical tables were replicated.
Daily exports were written to Cloud Storage.
Object Versioning was enabled.
Lifecycle policies archived historical snapshots for twelve months.
The platform had passed every external audit during the previous three years.
On paper, disaster recovery looked exemplary.
The CTO wasn’t convinced.
At the quarterly technology leadership meeting he proposed something nobody expected.
“We’re not going to review the documentation.”
He looked around the room.
“We’re going to pretend the platform disappeared.”
The room laughed.
Then he continued.
“I’m serious.”
The exercise was scheduled for the following Friday.
Only four people knew it was coming.
Nobody else—including operations engineers—would receive advance notice.
At exactly 09:00, the CTO announced the simulation.
“BigQuery is unavailable.”
No additional explanation followed.
No production systems were actually shut down.
Instead, every team had to explain, minute by minute, how they would continue operating if analytical workloads suddenly became inaccessible.
The first fifteen minutes were surprisingly calm.
Operations confirmed that customer-facing applications continued functioning because transactional systems were independent of BigQuery.
Engineering reported no service degradation.
Infrastructure monitoring remained green.
Several executives smiled.
Perhaps the exercise would prove that all previous investments had been worthwhile.
Twenty-three minutes later, the first phone call arrived.
Finance could not generate payment reconciliation reports.
Thirty-one minutes after the simulation started, the fraud investigation team reported that overnight anomaly analysis had stopped.
At the forty-minute mark, customer support lost access to account history dashboards.
Marketing could no longer monitor campaign performance.
Risk analysts couldn’t calculate daily exposure reports.
Regulatory reporting pipelines stopped completely.
Curiously, none of these failures represented technical outages.
Applications were healthy.
Servers were healthy.
Networks were healthy.
The business had become blind.
The CTO wrote two timestamps on the whiteboard.
Recovery Time Objective (RTO): 4 hours.
Reality after 1 hour: No department could estimate when critical reporting would return.
The backup strategy suddenly looked less impressive.
The infrastructure manager explained that restoring BigQuery data from Cloud Storage exports was technically possible.
The estimated process looked like this:
- Identify affected datasets.
- Retrieve export manifests.
- Restore compressed files.
- Reload tables into temporary datasets.
- Rebuild partition metadata.
- Validate row counts.
- Recreate authorized views.
- Reconnect reporting tools.
- Restart scheduled Dataform workflows.
- Validate downstream dependencies.
Estimated duration:
18 to 26 hours.
The room became very quiet.
One executive finally asked the obvious question.
“I thought our recovery objective was four hours.”
“It is,” the infrastructure manager answered.
“Then why does the recovery plan require almost a full day?”
Nobody had a satisfactory answer.
The company had invested heavily in backup technology.
It had invested almost nothing in recovery engineering.
The distinction would eventually change the entire platform architecture.
Because backups are created for systems.
Recovery is performed by people.
And people need something much more valuable than stored data.
They need a process that actually works.
The Company Had Backups. It Didn’t Have Recovery.
The CTO stopped the simulation after ninety minutes.
Not because the exercise had failed.
Because it had already revealed enough evidence.
Nobody questioned the quality of the backup infrastructure. Every scheduled export had completed successfully. Cloud Storage contained every required snapshot. Object lifecycle policies worked exactly as designed. Checksums matched. Retention policies satisfied regulatory requirements.
From an infrastructure perspective, the company could prove that its data existed.
From a business perspective, nobody could prove when that data would become useful again.
The difference was enormous.
The engineering team decided to measure recovery instead of assuming it.
A cross-functional task force documented every action required to rebuild the analytical platform from backup. They deliberately avoided theoretical estimates. Every step had to be executed in a test environment while another engineer measured the elapsed time.
The results were uncomfortable.
Restoring 243 TB of compressed exports into BigQuery required approximately 5 hours and 40 minutes.
Rebuilding partition expiration policies and clustering definitions consumed another 42 minutes because several datasets had been created manually over the years and were not managed through Infrastructure as Code.
Recreating authorized views required 51 minutes. The SQL definitions existed, but they were scattered across Git repositories, old deployment scripts, and manual documentation.
Dataform itself was operational within minutes, but rebuilding dependency chains and validating incremental models required almost 2 hours because many production tables depended on upstream datasets restored in a different order.
Looker presented another surprise.
Connections recovered immediately.
Semantic models did not.
Several dashboards referenced tables that no longer existed under exactly the same names because temporary restoration datasets had been used during testing. More than 180 explores required validation before business users could trust the numbers again.
By the end of the exercise, the engineering team had collected real measurements rather than assumptions.
| Recovery Activity | Planned | Actual |
|---|---|---|
| Restore BigQuery datasets | 4 h | 5 h 40 m |
| Restore Dataform pipelines | 30 m | 1 h 55 m |
| Validate BI dashboards | 20 m | 2 h 15 m |
| Reconfigure permissions | 15 m | 48 m |
| Business acceptance testing | Not estimated | 3 h 20 m |
The total recovery time exceeded 14 hours.
The official disaster recovery documentation still claimed 4 hours.
No one had intentionally misled the organization.
The documentation had simply never been validated under realistic conditions.
The postmortem produced another important observation.
Almost 70% of total recovery time had nothing to do with restoring data.
It was spent restoring relationships between systems.
Permissions.
Views.
Data lineage.
Transformation schedules.
Dashboard metadata.
Business validation.
That insight fundamentally changed the architecture strategy.
The objective was no longer to improve backups.
The objective became reducing recovery complexity.
During the following three months, the engineering team implemented several structural improvements.
Every production dataset received an Infrastructure-as-Code definition. Dataset creation, IAM policies, partitioning rules, labels, and expiration settings became fully reproducible.
Dataform repositories were reorganized so that a complete platform deployment could be recreated from Git without manual intervention.
Looker semantic models were version-controlled alongside transformation code instead of being managed independently.
Cloud Build pipelines were extended to provision analytical environments automatically, including datasets, service accounts, scheduled workflows, and monitoring policies.
Finally, the company introduced something it had never done before.
Quarterly recovery drills.
Not documentation reviews.
Not architecture presentations.
Actual recovery exercises measured with stopwatches.
The first repeat exercise took 8 hours and 17 minutes.
Three months later, another drill reduced that to 5 hours and 12 minutes.
After one year of continuous refinement, the complete analytical platform—including BigQuery, Dataform, Looker, scheduled workloads, IAM configuration, and validation testing—could be restored in 2 hours and 46 minutes.
Ironically, the company almost never needed to execute the full disaster recovery plan.
That wasn’t the point.
The repeated exercises transformed the engineering organization itself.
Architects began designing systems that were easier to rebuild.
Engineers stopped relying on undocumented manual procedures.
New projects were evaluated not only by scalability and cost, but also by recoverability.
The CTO later reflected on the exercise during a technology conference.
“We passed every compliance audit before we performed our first real recovery test.”
He paused.
“The audit verified that we had backups.”
“The exercise verified that we had a business.”
That distinction became part of the company’s engineering principles.
Backups protect information.
Recovery protects the organization.
Confusing those two concepts is one of the most expensive mistakes a technology leader can make.
Evidence Collected
- Cloud Storage backup inventory
- BigQuery dataset metadata
- Cloud Billing Export (recovery cost estimation)
- Dataform dependency graph
- IAM policy audit
- Looker semantic model inventory
- Cloud Build deployment logs
- Disaster recovery execution timeline
Executive Takeaways
- A backup strategy is only complete when recovery time has been measured, not estimated.
- Most recovery delays originate from dependencies, permissions, and business validation—not from data restoration.
- Disaster recovery should be engineered as a repeatable process, not documented as a theoretical procedure.
- Infrastructure as Code dramatically reduces recovery complexity.
- The most valuable disaster recovery metric is the one measured during practice, not the one written in policy.
One Question Every CTO Should Ask
“If our analytical platform disappeared at 9:00 a.m. tomorrow, what time would Finance, Risk, and the Board receive trusted numbers again—and do we know that from documentation or from experience?”