There is a fascinating and terrifying trend in modern data engineering. The moment a business hears the acronym “RAG” (Retrieval-Augmented Generation) or the phrase “semantic search,” architects immediately start drawing diagrams with a dozen third-party services. SaaS marketing has successfully convinced the industry that to do AI, you absolutely must buy a separate vector database…
I recently took over a new project to conduct a routine infrastructure audit. I decided to start with a standard check of the BigQuery storage and its costs. I brewed an espresso, opened the billing console, ran a few basic queries against INFORMATION_SCHEMA… and immediately choked on my coffee. The project was paying for “air”…
What BI Engine Is and How It Works If you read the marketing brochures, BI Engine is pure magic that makes your dashboards fly. If you look at the architecture, it is simply a distributed in-memory cache built on top of the standard BigQuery engine. Its sole purpose is to save your compute slots from…
Data warehousing in Google Cloud Platform offers massive scalability. It is incredibly fast and incredibly obedient. If you ask BigQuery to scan 100 GB of raw nested JSON five times a day, it will not stop you. It will not ask questions. It will just quietly execute the query and send a massive, painful invoice…
Chapter 1. Where It All Began: Pain and Optimization The fundamental reason behind launching any Cloud Data Engineering project almost always comes down to finances. Running a daily analytical script that does a basic SELECT * from raw logs and entirely overwrites the final data mart (a classic Full Refresh) works flawlessly. That is, until…
One of the first lessons every software engineer learns about databases is normalization. Split information into multiple tables. Remove duplication. Create primary keys. Create foreign keys. Join everything back together when you need the data. For transactional systems, this advice is excellent. For analytical systems, it can become surprisingly expensive. This is one of the…
1. The Wrong Mental Model: Sorting vs. Intelligent Organization When engineering teams design analytical data platforms, they often carry over habits from traditional relational database management systems (RDBMS) like PostgreSQL or MySQL. Ask an engineer what clustering does in BigQuery, and the most common answer is: “It sorts the data.” This is a dangerous oversimplification….
When a company launches its analytics platform on Google Cloud, everything usually looks impressive. The first datasets are small. Dashboards refresh in seconds. Marketing reports appear almost instantly. Data engineers proudly compare BigQuery with the old on-premises database and celebrate query times that dropped from minutes to seconds. For a while, everyone believes the architecture…
One sentence appears in almost every discussion about BigQuery. “BigQuery is expensive.” Sometimes it comes from developers. Sometimes from managers. Sometimes even from cloud architects. The sentence sounds convincing because everyone has seen screenshots of unexpectedly high cloud invoices. One poorly written query scans dozens of terabytes. Someone forgets a filter on a production table….