Mention Apache Airflow or Google Cloud Composer in a meeting, and someone will almost certainly say the same thing. “It schedules jobs.” Technically, that statement is true. Architecturally, it misses the point completely. Cron has been scheduling jobs since long before cloud computing existed. If scheduling were the only problem, Cloud Composer would be one…
Data warehousing in Google Cloud Platform offers massive scalability. It is incredibly fast and incredibly obedient. If you ask BigQuery to scan 100 GB of raw nested JSON five times a day, it will not stop you. It will not ask questions. It will just quietly execute the query and send a massive, painful invoice…
Let’s be honest: cloud bills can feel like a surprise party you never asked for. One day you run a few queries, the next day your finance team looks at you like you just bought a yacht. Predicting Google Cloud costs is not only smart, it’s survival. In this article we’ll walk through how to…
Ask someone what Google Cloud Dataflow is, and the answer is usually immediate. “It’s an ETL service.” Or perhaps: “It’s Google’s version of Apache Spark.” Neither answer is completely wrong. Neither explains why Google spent years building one of the most sophisticated distributed data processing engines in the cloud. Dataflow is not interesting because it…
Walk into almost any architecture meeting today, and sooner or later someone will say the same sentence. “We need real-time data.” Nobody asks why. Nobody asks what “real-time” actually means. The discussion immediately shifts toward technologies. Pub/Sub. Dataflow. Kafka. Streaming pipelines. Event-driven architecture. The assumption is simple. If data arrives instantly, the system must be…
Ask a group of engineers what a data pipeline is, and you will probably hear a list of technologies. “Apache Beam.” “Dataflow.” “Kafka.” “Pub/Sub.” “Airflow.” “Cloud Composer.” Those answers are not wrong. But they all skip the most important part. A data pipeline is not a product. It is not a service. It is not…
Chapter 1. Where It All Began: Pain and Optimization The fundamental reason behind launching any Cloud Data Engineering project almost always comes down to finances. Running a daily analytical script that does a basic SELECT * from raw logs and entirely overwrites the final data mart (a classic Full Refresh) works flawlessly. That is, until…
One of the first lessons every software engineer learns about databases is normalization. Split information into multiple tables. Remove duplication. Create primary keys. Create foreign keys. Join everything back together when you need the data. For transactional systems, this advice is excellent. For analytical systems, it can become surprisingly expensive. This is one of the…
1. The Wrong Mental Model: Sorting vs. Intelligent Organization When engineering teams design analytical data platforms, they often carry over habits from traditional relational database management systems (RDBMS) like PostgreSQL or MySQL. Ask an engineer what clustering does in BigQuery, and the most common answer is: “It sorts the data.” This is a dangerous oversimplification….