Ask someone what Identity and Access Management, or IAM, does in Google Cloud, and the answer is almost always the same. “It manages permissions.” That answer is correct. It is also incomplete. If IAM existed only to decide who can open a bucket or query a table, it would be little more than a settings…
Ask someone what Google Cloud Storage is, and the answer is usually immediate. “It’s where we store files.” Technically, that answer is correct. Architecturally, it explains almost nothing. Cloud Storage is often introduced as the cloud equivalent of a hard drive. Upload a file. Download a file. Delete a file. Simple. Unfortunately, that way of…
When engineers discuss compute services in Google Cloud, the conversation usually revolves around three familiar names. Cloud Run. Google Kubernetes Engine. Compute Engine. Cloud Run Jobs rarely receives the same attention. Many architects assume it is simply Cloud Run with a different execution mode. Technically, that is true. Architecturally, it is a completely different way…
Mention Apache Airflow or Google Cloud Composer in a meeting, and someone will almost certainly say the same thing. “It schedules jobs.” Technically, that statement is true. Architecturally, it misses the point completely. Cron has been scheduling jobs since long before cloud computing existed. If scheduling were the only problem, Cloud Composer would be one…
Ask someone what Google Cloud Dataflow is, and the answer is usually immediate. “It’s an ETL service.” Or perhaps: “It’s Google’s version of Apache Spark.” Neither answer is completely wrong. Neither explains why Google spent years building one of the most sophisticated distributed data processing engines in the cloud. Dataflow is not interesting because it…
Walk into almost any architecture meeting today, and sooner or later someone will say the same sentence. “We need real-time data.” Nobody asks why. Nobody asks what “real-time” actually means. The discussion immediately shifts toward technologies. Pub/Sub. Dataflow. Kafka. Streaming pipelines. Event-driven architecture. The assumption is simple. If data arrives instantly, the system must be…
Ask a group of engineers what a data pipeline is, and you will probably hear a list of technologies. “Apache Beam.” “Dataflow.” “Kafka.” “Pub/Sub.” “Airflow.” “Cloud Composer.” Those answers are not wrong. But they all skip the most important part. A data pipeline is not a product. It is not a service. It is not…
If there is one decision that shapes the future of a system more than any other, it is the choice of the data layer. Ask most engineers where the architecture begins, and many will point to the application. They will talk about APIs, containers, deployment pipelines, or Kubernetes clusters. Those components are visible, easy to…
If you spend enough time reading cloud forums or watching conference talks, you will eventually encounter the same question in different forms. “Should I use Cloud Run or GKE?” “Is Kubernetes better than virtual machines?” “When should I move to Compute Engine?” At first glance, these seem like sensible technical questions. After all, each of…