GCP

Ask someone what Google Cloud Storage is, and the answer is usually immediate. “It’s where we store files.” Technically, that answer is correct. Architecturally, it explains almost nothing. Cloud Storage is often introduced as the cloud equivalent of a hard drive. Upload a file. Download a file. Delete a file. Simple. Unfortunately, that way of…

Cloud Run Jobs: Stop Paying for Waiting Software engineering has spent the last decade obsessed with listening. We have conditioned ourselves to believe that every application must be a microservice, expose a REST API, bind to a network port, and sit in memory twenty-four hours a day, waiting for incoming traffic. This request-driven model is…

Mention Apache Airflow or Google Cloud Composer in a meeting, and someone will almost certainly say the same thing. “It schedules jobs.” Technically, that statement is true. Architecturally, it misses the point completely. Cron has been scheduling jobs since long before cloud computing existed. If scheduling were the only problem, Cloud Composer would be one…

Ask someone what Google Cloud Dataflow is, and the answer is usually immediate. “It’s an ETL service.” Or perhaps: “It’s Google’s version of Apache Spark.” Neither answer is completely wrong. Neither explains why Google spent years building one of the most sophisticated distributed data processing engines in the cloud. Dataflow is not interesting because it…

Ask a group of engineers what a data pipeline is, and you will probably hear a list of technologies. “Apache Beam.” “Dataflow.” “Kafka.” “Pub/Sub.” “Airflow.” “Cloud Composer.” Those answers are not wrong. But they all skip the most important part. A data pipeline is not a product. It is not a service. It is not…

If there is one decision that shapes the future of a system more than any other, it is the choice of the data layer. Ask most engineers where the architecture begins, and many will point to the application. They will talk about APIs, containers, deployment pipelines, or Kubernetes clusters. Those components are visible, easy to…

If you spend enough time reading cloud forums or watching conference talks, you will eventually encounter the same question in different forms. “Should I use Cloud Run or GKE?” “Is Kubernetes better than virtual machines?” “When should I move to Compute Engine?” At first glance, these seem like sensible technical questions. After all, each of…