Everyone Blamed Google. Nobody Looked at the Queries.

At 8:12 on a Monday morning, the CFO forwarded a screenshot to the CTO with a short message.

“Can you explain this before today’s board meeting?”

The screenshot came from the monthly Cloud Billing report. There were no alarming spikes in Compute Engine, no unexpected networking charges, and no expensive AI experiments. The problem was concentrated almost entirely in one service.

BigQuery.

The previous month’s analytics bill had been $61,480.

The current estimate, with five days still remaining in the billing cycle, had already reached $123,970.

If the trend continued, the company would finish the month at approximately $142,000—an increase of more than 130%.

No major product had been launched.

Customer traffic had increased by only 11% compared to the previous month.

Engineering headcount was unchanged.

The data platform architecture had not been modified.

On paper, nothing justified a doubling of analytical costs.

The first executive meeting produced exactly the explanations one would expect.

“Google must have changed pricing.”

“Maybe someone forgot to reserve slots.”

“We’re storing too much data.”

“Marketing probably doubled event volume.”

Each explanation sounded plausible.

Every one of them was wrong.

The CTO resisted the temptation to start optimizing immediately.

Instead, he formed a small investigation team consisting of a FinOps engineer, a senior data engineer, a BigQuery specialist, and one analyst who had no involvement in building the existing platform. The last choice surprised several people.

“Fresh eyes,” the CTO explained. “The people who built the platform already have theories. I want someone who only trusts evidence.”

The investigation began exactly where every cloud cost investigation should begin—not in BigQuery itself, but in Cloud Billing Export.

The billing export confirmed that the increase originated almost entirely from analysis charges, not storage, networking, or streaming inserts.

That single observation immediately eliminated half of the initial theories.

Storage had increased by only 4.3%.

Streaming ingestion costs remained almost identical.

Long-term storage discounts were unchanged.

The additional spending came from one source:

Queries were reading dramatically more data than before.

The next step was to determine which queries.

Using INFORMATION_SCHEMA.JOBS_BY_PROJECT, the team extracted every completed query from the previous sixty days. More than 2.8 million jobs had been executed during that period. They grouped them by user, service account, scheduled job, bytes processed, execution frequency, and destination dataset.

The first chart immediately looked suspicious.

The number of queries had increased by only 9%.

The amount of data scanned had increased by 146%.

Those two numbers should normally move together.

Here, they did not.

Someone—or something—had started reading much larger volumes of data without increasing the number of executions.

The investigation shifted from quantity to efficiency.

The engineers ranked every query by total bytes processed during the month.

One scheduled transformation immediately stood out.

It executed every 15 minutes, exactly 96 times per day.

Each execution scanned approximately 9.4 TB.

Execution time averaged just under three minutes.

Nobody had previously considered it problematic because every run completed successfully.

The cost calculation changed the conversation.

BigQuery pricing for on-demand analysis is based primarily on bytes processed.

At approximately $5 per TB scanned, each execution cost roughly:

9.4 TB × $5 = $47.00

One execution looked harmless.

Ninety-six executions per day produced:

$47 × 96 = $4,512 per day

Across a 30-day month:

$4,512 × 30 = $135,360

The room became very quiet.

One SQL statement—running exactly as designed—had become more expensive than the salaries of several engineers combined.

The obvious question followed immediately.

Who wrote it?

Audit history showed that the query had existed for almost eighteen months.

It had never caused noticeable financial impact before.

Nobody had modified it recently.

The SQL itself wasn’t new.

The data it operated on was.

Eighteen months earlier, the source table contained approximately 180 million rows.

Today it contained more than 6.3 billion.

The transformation still performed a full table scan every fifteen minutes because, when it was originally written, scanning the entire table took less than twenty seconds and cost almost nothing.

The company had grown.

The query had never evolved with it.

At first glance, the solution looked almost embarrassingly simple.

Rewrite the SQL.

The senior BigQuery engineer wasn’t convinced.

“If this query escaped our attention,” he said, “it’s probably not the only one.”

Instead of fixing the obvious problem immediately, he proposed something more ambitious.

“Let’s find out whether this is a bad query…”

He paused while looking at the ranking of the top one hundred most expensive jobs.

“…or evidence of a bad engineering habit.”

That single question transformed what looked like a routine optimization into one of the most valuable FinOps investigations the company had ever conducted.

The Query Was Guilty. The Process Was Responsible.

The engineering team deliberately resisted the temptation to optimize the obvious SQL first.

It would have reduced costs quickly, but it would also have hidden the real problem. Experienced investigators know that the first visible failure is often just the symptom of a much larger system weakness.

Instead, they expanded the analysis.

Using INFORMATION_SCHEMA.JOBS_BY_PROJECT, they extracted every query executed during the previous 90 days and ranked them by total bytes processed, execution frequency, average slot time, execution duration, and total monthly cost.

The dataset contained approximately 4.3 million completed jobs.

Only 2.1% of those queries generated nearly 78% of the company’s entire BigQuery compute bill.

That statistic immediately changed priorities.

The objective was no longer to optimize thousands of queries.

It was to understand why fewer than one hundred SQL statements consumed almost all analytical resources.

The team selected the fifty most expensive jobs and started reviewing them one by one.

The patterns appeared surprisingly quickly.

Twenty-one queries performed full table scans against partitioned tables without filtering by partition date.

Fourteen queries joined massive fact tables before applying filters instead of reducing datasets first.

Nine scheduled reports refreshed every fifteen minutes despite being reviewed by executives once every morning.

Five materialized views had quietly been replaced months earlier by standard views during an unrelated migration, causing repeated recalculation of billions of rows.

Three exploratory notebooks created by analysts had accidentally become production jobs through Cloud Scheduler.

None of those mistakes looked dramatic in isolation.

Together, they explained almost the entire increase in spending.

The engineers decided to quantify the impact instead of discussing it abstractly.

One reporting query scanned 11.8 TB every execution.

Adding a simple partition filter reduced the scan to 340 GB.

Execution time decreased from 168 seconds to 11 seconds.

Cost dropped from approximately $59 to less than $2.

Another transformation joined two tables containing 5.9 billion and 1.8 billion rows before applying customer filters.

Reordering the execution plan allowed BigQuery to process only 4.6% of the original data volume.

Monthly cost for that single pipeline fell from roughly $18,900 to $1,270.

The largest surprise came from executive dashboards.

Looker usage statistics showed that one dashboard refreshed every fifteen minutes around the clock.

The board reviewed it once every Monday morning.

During the previous month it had executed 2,880 refreshes.

Only four had ever been viewed by executives.

The remaining 2,876 executions had consumed approximately $6,400 worth of BigQuery resources without informing a single decision.

Nobody had configured the refresh interval intentionally.

It was simply the platform default.

The investigation gradually stopped looking like cost optimization.

It started looking like organizational archaeology.

Every expensive query represented a perfectly reasonable engineering decision made years earlier.

Every scheduled refresh had once satisfied a genuine business requirement.

Every dataset had continued growing exactly as expected.

The business evolved.

The architecture remained frozen.

The CTO asked one final question before approving any technical changes.

“How many of these issues would have been detected automatically?”

The answer was uncomfortable.

None.

The platform monitored failures exceptionally well.

It monitored success almost not at all.

Every expensive query completed successfully.

Every scheduled job finished on time.

Every dashboard refreshed without errors.

From an operational perspective, the platform appeared perfectly healthy while quietly wasting hundreds of thousands of dollars.

The company decided not merely to optimize SQL but to redesign its engineering governance.

Every production query processing more than 1 TB automatically generated a weekly review report.

Cloud Billing Export became part of monthly architecture reviews rather than remaining a finance-only artifact.

New Dataform pipelines required estimated scan-cost analysis before deployment.

Scheduled dashboards needed an identified business owner and documented refresh requirements.

Queries without partition filters against eligible tables automatically failed code review.

Within eight weeks, measurable improvements appeared.

Monthly BigQuery analysis costs fell from an estimated $142,000 to $67,800.

Average query execution time decreased by 41%.

Daily scanned data volume dropped from approximately 1.1 petabytes to 420 terabytes, despite continued business growth.

Most importantly, no reports disappeared.

No analytical capability was lost.

Customers noticed nothing.

Executives noticed everything.

Annualized savings exceeded $890,000.

Ironically, less than 15% of those savings came from rewriting SQL.

The remaining savings came from changing engineering behavior.

Looking back, the senior data engineer summarized the investigation in a sentence that later became part of every architecture onboarding session.

“BigQuery never became expensive. We simply stopped asking whether our queries still deserved to exist.”

The lesson extended well beyond analytics.

Cloud platforms are remarkably efficient at executing exactly what engineers ask them to do.

They are remarkably indifferent to whether those instructions still make business sense.

Technology rarely wastes money by itself.

Processes do.

Evidence Collected

• Cloud Billing Export
• INFORMATION_SCHEMA.JOBS_BY_PROJECT
• INFORMATION_SCHEMA.PARTITIONS
• INFORMATION_SCHEMA.TABLE_STORAGE
• BigQuery Audit Logs
• Looker dashboard usage statistics
• Dataform dependency graph
• Cloud Scheduler execution history

Executive Takeaways

• Cost spikes are usually caused by a small percentage of queries, not by the platform itself.
• A successful query can still be an expensive query.
• Every production dashboard should justify both its existence and its refresh frequency.
• FinOps belongs inside engineering governance, not only in the finance department.
• Review query economics with the same discipline used for reviewing application code.

One Question Every CTO Should Ask

“If our BigQuery bill doubled tomorrow, how many hours would it take us to identify the ten queries responsible—and do we already have the visibility to answer that question?”