A Map of AI Systems in Google Cloud: From Classical Machine Learning to Autonomous Agents

When engineers and business managers talk about Artificial Intelligence (AI) today, they often put every technology into one single bucket. Because Large Language Models (LLMs) are famous in the news, many beginners try to use chat models for every problem. They try to use a text model to forecast next month’s warehouse sales, or they try to train a custom neural network from zero just to read text from a scanned PDF invoice.

In real engineering, this approach wastes money and creates slow, unreliable software. Different problems require different tools. Some problems need simple statistics on a database table. Other problems need a system that can read company documents or run actions across multiple servers.

Google Cloud provides dozens of AI services, primarily unified under Vertex AI (which Google evolved into the Gemini Enterprise Agent Platform) and BigQuery. If you open the Google Cloud console without a map, the list of services looks confusing. However, once you see the engineering logic behind them, the map becomes very clear. Every technology in this system was created to solve a specific limitation of the technology that came before it.

To understand this map, we will look at AI in Google Cloud from two sides:

  1. How the system processes information: The evolution from Classical Machine Learning to Deep Learning, Foundation Models, Generative AI, RAG, and AI Agents.
  2. How much you build versus how much you rent: The choice between Ready-Made AI APIs, Foundation Models, and Custom Models.

Throughout this guide, we will use one realistic case study: a national retail chain that sells household goods and electronics through 200 physical stores and an online website. All of their sales data, product catalogs, and customer support records live in Google Cloud.

Part 1: How AI Systems Evolved (From Numbers to Actions)

Let us walk through the six core concepts of modern AI. Notice how each technology solves a real business problem, hits a hard technical wall, and forces engineers to invent the next layer.

1. Classical Machine Learning (Classical ML)

Long before computers could write essays or draw pictures, businesses needed computers to find patterns in numbers.

The Problem It Solves

Imagine you want to know how many bottles of laundry detergent Store #42 will sell next week. A programmer could try to write manual rules in code (if month == December, increase order by 10%). However, sales depend on 30 different factors at the same time: the day of the week, local weather, current discounts, past sales, and store size. A human cannot write manual if-else rules for 30 connected variables. We need a mathematical algorithm that looks at past data and finds the formula automatically.

How It Works

Classical Machine Learning works with structured (tabular) data—data that fits neatly into database tables with rows and columns, like an Excel sheet or a SQL table. You show the algorithm thousands of historical rows where the final answer is already known. The algorithm uses statistics (such as Linear Regression for predicting numbers, Logistic Regression or XGBoost for classifying items into groups, or ARIMA for time-series forecasting) to calculate how each column affects the final result.

How It Works in Google Cloud

In Google Cloud, the most practical place for Classical ML is BigQuery ML. Usually, training a model requires exporting gigabytes of data out of a database into Python files. BigQuery ML lets data engineers and analysts train and run Classical ML models directly inside the database using standard SQL queries (CREATE MODEL). For more complex tabular models, engineers use Vertex AI AutoML or custom XGBoost pipelines.

Realistic Retail Case

Our retail chain has 50 million rows of past customer purchases in BigQuery. The marketing team wants to know which customers are likely to stop buying from the store in the next 30 days (customer churn) so they can send them a discount coupon. An engineer writes a 15-line SQL query in BigQuery ML to train a Logistic Regression model using columns like days_since_last_order, average_basket_size, and return_count. Every Monday, the model scores all active customers and outputs a simple probability score between 0.0 and 1.0.

The Limitation (Why We Need the Next Step)

Classical ML is fast, cheap, and accurate, but it is blind to the messy real world. It only understands clean columns of numbers and fixed categories. What happens when our input is not a clean database row? What if the input is a photograph of a broken product, an audio recording of a phone call, or a free-form text message? Classical ML cannot look at raw pixels in a photo and understand what is inside.

2. Deep Learning (DL)

To help computers understand images, audio, and human language, engineers needed a model structure that could learn complex features on its own without a human creating database columns first.

The Problem It Solves

How do you teach a computer to recognize a damaged cardboard box in a photo? You cannot write a mathematical rule based on pixel number 452 being gray. The box might be photographed from the left, from the right, in bright light, or in shadow. We need a system that learns visual and audio concepts step by step, similar to how the human brain processes signals from the eyes and ears.

How It Works

Deep Learning uses Artificial Neural Networks with many layers (that is why it is called “deep”). When you feed a photo into a Deep Learning network:

  • The first layers only notice tiny lines, edges, and color changes.
  • The middle layers combine those lines into corners, textures, and shapes.
  • The final layers combine those shapes to recognize a complete object—such as a crushed corner on a shipping box—and output a classification label.

How It Works in Google Cloud

Engineers build and run Deep Learning models on Vertex AI Training and Vertex AI Endpoints using frameworks like TensorFlow or PyTorch running on graphics cards (GPUs) or Google’s Tensor Processing Units (TPUs). If a team has labeled photos or videos but does not want to write neural network code manually, they can use Vertex AI AutoML for image or video classification.

Realistic Retail Case

At the central retail warehouse, workers receive 3,000 returned parcels every day. Before opening a parcel, the warehouse wants to automatically record if the outer shipping box arrived damaged from the courier service. A camera above the conveyor belt takes a photo of every box. A Deep Learning vision model hosted on a Vertex AI Endpoint inspects each photo in 100 milliseconds and tags it as Intact or Damaged. If it is tagged as Damaged, the system saves the photo as proof for the insurance claim.

The Limitation (Why We Need the Next Step)

Traditional Deep Learning has two big problems:

  1. It is a narrow specialist. The model trained to recognize damaged boxes knows nothing else. If you now want to recognize product barcodes or classify customer emails, you have to collect thousands of new labeled examples and train a brand-new model from zero.
  2. It only selects from fixed labels. It can output 94% Damaged or 6% Intact, but it cannot generate new content. It cannot write a summary, explain why the box is damaged in a sentence, or answer a question.

3. Foundation Models

Around a decade ago, if a company wanted five different AI features, they had to collect five datasets and train five separate Deep Learning models from zero. This cost too much time and money for normal businesses.

The Problem It Solves

Most businesses do not have millions of labeled examples, dozens of PhD researchers, or millions of dollars to rent supercomputers every time they want to start a new project. Engineers needed a “general-purpose brain” that already understands human languages, grammar, computer code, and visual objects out of the box.

How It Works

A Foundation Model is a massive Deep Learning neural network trained once by a major research organization (like Google, Anthropic, or Meta) on an enormous volume of text, code, images, and audio. Because this base model has already learned how words, ideas, and images relate to each other across the world, you do not need to train it from scratch. You take this single foundation model and adapt it to hundreds of different company tasks simply by giving it instructions or fine-tuning it slightly.

How It Works in Google Cloud

Google Cloud provides a central library called Vertex AI Model Garden. Inside Model Garden, engineers can choose from over 200 foundation models:

  • Google’s own models: The Gemini family (multimodal models that understand text, images, audio, and video together).
  • Open-weight models: Models like Google’s Gemma (such as Gemma 4) or Meta’s Llama, where you can inspect and run the model weights yourself.
  • Partner models: Models like Anthropic’s Claude, available directly inside Google Cloud billing and security.

Realistic Retail Case

Our retail company sells products across Ukraine and Poland. Customers leave thousands of product reviews in Ukrainian, Polish, and English. In the past, the team would need to train three separate language models to analyze review topics. Instead, the team uses a single Gemini foundation model from Vertex AI Model Garden. Because the foundation model already knows all three languages and understands retail concepts, it can read any review immediately without the company training a custom language model.

The Limitation (Why We Need the Next Step)

A Foundation Model is the underlying engine. Now we need to understand how we actually use this engine to create new business outputs, and what goes wrong when we ask it to generate information on its own.

4. Generative AI (GenAI)

While Classical ML and traditional Deep Learning are discriminative (they separate data into existing categories or predict a number), Generative AI creates new content that did not exist before.

The Problem It Solves

In daily business operations, humans spend thousands of hours writing, summarizing, translating, and formatting text or code. Traditional software cannot read a messy 10-page supplier PDF and write a clean product description for an online shop. We need a system that can produce structured human language, code, or images on demand.

How It Works

Generative AI sits on top of Foundation Models. When you send a text instruction (called a prompt) to a Large Language Model (LLM), the model breaks your text into small pieces called tokens (words or parts of words). Based on everything it learned during its foundation training, it calculates the most likely next token, one by one, until it generates a complete answer, summary, or piece of code.

How It Works in Google Cloud

Engineers test prompts and call generative models through Vertex AI Studio and the Vertex AI Gemini API (part of the Gemini Enterprise Agent Platform). Engineers can pass text, product photos, PDF files, or audio recordings into a Gemini model and ask for output in either plain text or strict JSON format so that other software can read it.

Realistic Retail Case

Every month, our retail buyers add 1,500 new products to the online store. Suppliers send product data in messy spreadsheets with missing descriptions and ugly technical abbreviations (for example: "Kettle Stainless 1.7L 2200W auto-off blk"). An engineer sets up a batch pipeline that sends these raw lines plus a photo of the product to Gemini on Vertex AI. The model generates a clean, polite 80-word description for customers and extracts structured tags (Material: Stainless Steel, Volume: 1.7L, Color: Black) for the website filters.

The Limitation (Why We Need the Next Step)

Generative AI has a major engineering flaw when used alone: its knowledge is frozen in time and limited to public data. A base Gemini or Claude model does not know your company’s internal rules, private database records, or today’s prices. Even worse, because an LLM is designed to generate smooth, confident text, if you ask it about your company’s private warranty rules, it might guess and invent a rule that sounds completely real but is factually wrong. Engineers call this a hallucination. How do we force the model to use only our real company facts?

5. RAG (Retrieval-Augmented Generation)

If a Generative AI model invents false answers when asked about private company data, a beginner might think: “Let’s retrain the whole foundation model on our company documents!” In reality, retraining a giant model is expensive, slow, and still does not prevent hallucinations—plus, the moment a price or rule changes tomorrow, the model is outdated again. RAG is the engineering solution to this problem.

The Problem It Solves

We need a Generative AI model to answer customer or employee questions using only our private, up-to-date company documents and databases, without inventing facts and without retraining the model.

How It Works

Think of a standard LLM as a student taking an exam from memory. If the student does not know the answer, they might guess. RAG changes the exam into an open-book test. It separates the work into two strict stages:

  1. Retrieval (Finding the facts): Before the AI generates a single word, a search system takes the user’s question and searches your private company documents (PDFs, Word files, Confluence pages, or database tables). To search by meaning rather than just exact words, the system uses Vector Embeddings—it converts paragraphs of text into lists of numbers that represent their meaning, and finds the exact paragraphs that match the user’s question.
  2. Augmented Generation (Writing the answer): The system takes the user’s original question, attaches the 3 or 5 real paragraphs it just found, and sends both to the LLM with a strict instruction: “Answer the user’s question using ONLY the facts written in these attached paragraphs. Include a citation to the source document. If the answer is not in the paragraphs, say ‘I do not know.'”

How It Works in Google Cloud

Google Cloud provides two ways to build RAG:

  • Out-of-the-box managed RAG: Using Vertex AI Search (also referred to as Agent Search in the updated console) or Vertex AI RAG Engine. You point it to a Cloud Storage bucket full of PDFs, Google Drive, Jira, or BigQuery. Google automatically splits the documents into chunks, creates the vector embeddings, indexes them, and provides a single Grounded Generation API.
  • Custom DIY RAG: You use the Layout Parser in Document AI to split complex documents, generate embeddings with the Vertex AI Text Embeddings API, store them in Vertex AI Vector Search, and pass the results to Gemini.

Realistic Retail Case

Our retail chain has 200 stores and hires many new cashiers and support operators every month. Previously, when a customer brought back a complicated item (like a coffee machine bought on credit), the cashier had to call a supervisor to ask about the return procedure. The engineering team builds an internal RAG assistant using Vertex AI Search connected to 400 internal instruction PDFs and HR manuals. Now, the cashier types: “How do I process a return for an item bought with a bank installment plan?” In two seconds, the system retrieves the exact page from the Store Operations Manual, gives a 4-step checklist, and shows a clickable link to page 14 of the official PDF. If the company updates the PDF in Cloud Storage at 9:00 AM, the assistant gives the new answer at 9:01 AM.

The Limitation (Why We Need the Next Step)

RAG is an intelligent library. It reads, searches, and explains information accurately. However, RAG is passive—it has no hands. What if a customer writes to support: “My order #88310 is missing one item. Please check my invoice, verify if the item was scanned at the warehouse, and issue a refund to my card for that missing item”? A pure RAG system can explain the refund policy, but it cannot look up live order #88310 in the SQL database, check warehouse scan logs, or trigger a refund API in the payment system. To actually do work, we need the final layer of the evolution.

6. AI Agents

An AI Agent is what happens when you give a Foundation Model access to external software tools, memory, and the ability to plan multi-step actions.

The Problem It Solves

Real business processes rarely finish in one question and one answer. Completing a task usually requires checking System A, comparing the result with a rule in Document B, and sending a command to System C. Traditional automation scripts break if the user phrases a request differently, while simple chatbots can only talk without taking action.

How It Works

An AI Agent is built from four architectural building blocks:

  1. The Reasoning Brain (Foundation Model): A model like Gemini that reads the goal and decides what step to take first.
  2. Tools (Functions and APIs): Safe, strictly defined connections to external systems—for example, a function get_order_status(order_id), a SQL connection to BigQuery, or a ticket-creation API in Jira or Salesforce.
  3. Knowledge (RAG): Connection to company documents so the agent knows the business rules before acting.
  4. Memory (State Management): Short-term session memory to track the current multi-step task, and long-term memory across conversations.

When you give an AI Agent a task, it runs in a reasoning loop:

  • Step 1 (Observe & Plan): “The user wants to change the delivery address for order #88310. First, I need to check if the order has already left the warehouse.”
  • Step 2 (Call Tool 1): The agent calls check_shipment_status(order_id=88310).
  • Step 3 (Read Result & Reason): The API returns "Status: Packing in Warehouse". The agent sees the parcel has not left yet.
  • Step 4 (Call Tool 2): The agent calls update_delivery_address(order_id=88310, new_address="...").
  • Step 5 (Respond): The agent confirms to the user that the address has been updated in the system.

How It Works in Google Cloud

In Google Cloud, this entire layer lives in Vertex AI Agent Builder (now consolidated under the Gemini Enterprise Agent Platform). Engineers have two ways to build agents here:

  • Agent Development Kit (ADK): A code-first framework (available in Python, Go, Java, and TypeScript) where engineers write strict code to control how the agent uses tools, handles errors, and collaborates with other agents.
  • Agent Studio / Console: A visual, low-code interface for designing conversational flows and connecting data sources quickly.Once built, agents run on Agent Engine (Agent Runtime), a managed cloud environment that handles scaling, security permissions (scoped identity), Sessions, and Memory Bank. Google Cloud also provides managed Model Context Protocol (MCP) servers to connect agents safely to BigQuery.

Realistic Retail Case

Our retail chain receives 25,000 customer support requests per month. Around 40% of them are standard operational tasks: checking order status, changing a delivery store, canceling an unpaid order, or registering a bonus card problem. The engineering team builds a Customer Support Agent using ADK on Agent Engine. The agent has read-only tools to check orders in the ERP database, a RAG tool to read store rules, and restricted write tools to change delivery addresses for unshipped orders. If a customer asks for a cash refund over $100, the agent’s security rules forbid it from sending money automatically; instead, the agent collects the order details, attaches the invoice and database logs, and routes the ticket directly to a human operator with the full context prepared.

Part 2: Build vs. Rent in Google Cloud (Ready-Made APIs vs. Custom Models)

Now you understand how the six types of AI work. However, when you design a system in Google Cloud, you must also make an economic and architectural decision: Where do we get the model?

Should we rent a finished service from Google, use a Foundation Model, or train our own Custom Model from our own data? Let us compare Ready-Made AI APIs and Custom Models, and see how they fit with Foundation Models.

7. Ready-Made AI APIs (Pre-trained APIs)

Many beginners forget that Google has already solved the most common AI problems and packaged them as simple web services.

The Concept

With a Ready-Made AI API, you do not choose a neural network architecture, you do not collect training data, and you do not even need to write complex prompts. Google has already trained specialized models on massive datasets and exposed them as standard REST or gRPC APIs. Your code sends a file or text string to the API endpoint, and the API returns structured JSON data in milliseconds.

The Problem It Solves

Problems like converting human speech into text, translating Ukrainian to English, or reading tax numbers from a standard invoice are identical for a bank, a hospital, and a retail store. If your engineering team spends three months building an optical character recognition (OCR) model from scratch, you are wasting company money reinventing the wheel.

Key Ready-Made AI APIs in Google Cloud

  • Document AI: Specialized models that read unstructured PDFs and scans (such as invoices, receipts, utility bills, and contracts) and turn them into clean database fields.
  • Cloud Vision API: Detects objects, faces, product logos, explicit content, and printed or handwritten text inside images.
  • Cloud Speech-to-Text & Text-to-Speech API: Converts call center audio recordings into text transcripts, or turns text into natural voice responses.
  • Cloud Translation API: Translates text and documents between more than 100 languages rapidly and cheaply.

Realistic Retail Case

Our retail company receives 18,000 PDF invoices and delivery notes (накладные) every month from 400 different suppliers. Every supplier uses a slightly different table layout. Instead of asking C# or Python developers to write hundreds of regular expressions, and instead of calling a heavy conversational LLM for every page, the team sends the PDFs to Google Cloud Document AI. Document AI extracts the supplier tax code, invoice date, item table, and total sum into a structured JSON object, and loads it into BigQuery for accounting reconciliation.

8. Custom Models (Building Your Own Model)

At the opposite end of the spectrum from Ready-Made APIs are Custom Models.

The Concept

A Custom Model is a machine learning or deep learning model trained specifically on your company’s private historical data to solve a problem unique to your business. You control the input features, the training data, the evaluation metrics, and the serving infrastructure.

The Problem It Solves

Ready-Made APIs and general Foundation Models do not know the mathematical behavior of your specific business. Furthermore, Large Language Models like Gemini are relatively big and take hundreds of milliseconds (or several seconds) to generate text. What if you need a mathematical prediction in 15 milliseconds for 5,000 website visitors every second, at a cost of a few dollars a day? Or what if your task depends entirely on your private 5-year history of prices, warehouse stock, and regional store sales? For this, you must train a Custom Model.

Two Ways to Build Custom Models in Google Cloud

Google Cloud gives you two paths depending on how much manual code you want to write:

  1. Managed / Low-Code Custom Models (BigQuery ML & Vertex AI AutoML): You bring your own dataset in BigQuery, and Google Cloud handles the feature preprocessing, model selection, and hyperparameter tuning automatically. You get a custom model trained on your data without managing Python containers or GPU drivers.
  2. Code-First Custom Training (Vertex AI Custom Training): Your data science team writes custom training code in Python using Scikit-learn, XGBoost, PyTorch, or TensorFlow. They package the code in a Docker container, run the training job on Vertex AI Training, save the versioned artifact in Vertex AI Model Registry, automate retraining with Vertex AI Pipelines (MLOps), and deploy it to a Vertex AI Endpoint for fast real-time predictions.

Realistic Retail Case

When a customer opens the homepage of our online retail store, the website has 40 milliseconds to decide which 6 products to show in the “Recommended for You” banner. Calling a Generative AI model or an AI Agent on every page load would be far too slow and expensive. Instead, the data team uses BigQuery ML and Vertex AI to train a custom Matrix Factorization and XGBoost ranking model on 100 million historical clicks and purchases. The deployed custom model takes the user_id and current_category, calculates the top 6 product IDs in 12 milliseconds, and costs a fraction of a cent per thousand requests.

Comparing the Three Ownership Models

FeatureReady-Made AI APIsFoundation Models (via API / RAG)Custom Models (BQML / Vertex AI)
Who trained the base brain?Google trained it for one specific task.Google / Partners trained it for general tasks.You train it on your own company data.
Data you need to startZero training data needed.Zero training data (just prompts or RAG docs).Thousands to millions of historical rows.
Time to launch first version1 to 2 days (just an API call).1 to 2 weeks (prompting + RAG setup).2 weeks (BQML) to several months (Custom DL).
Speed per request (Latency)Fast (50ms – 500ms).Medium to Slow (500ms – 5+ seconds).Ultra-fast for tabular models (5ms – 50ms).
Best forOCR, invoices, speech, translation.Text generation, summarization, Q&A, agents.Forecasting numbers, high-speed ranking, churn.

Part 3: Architectural Map — How All Components Work Together

In a real mid-sized or large company, you never pick just one box from this article. A mature engineering architecture combines Classical ML, Ready-Made APIs, RAG, and Agents so that each component does only the job it was designed for.

Below is the architectural map of how our retail company processes data and requests inside Google Cloud:

Plaintext

====================================================================================
           REAL-WORLD GOOGLE CLOUD AI ARCHITECTURE (RETAIL PLATFORM)
====================================================================================

 [ INCOMING DATA & REQUESTS ]
   |
   +---> 1. Scanned Supplier Invoices (PDFs)
   |        |--> [ Ready-Made API: Document AI ]
   |               Extracts structured fields (Tax ID, Items, Total)
   |               Saves clean rows into ---> [ BigQuery Data Warehouse ]
   |
   +---> 2. Weekly Inventory & Churn Numbers (Structured Tables in BigQuery)
   |        |--> [ Classical ML / Custom Model: BigQuery ML (ARIMA_PLUS / XGBoost) ]
   |               Predicts next week's demand for 200 stores & customer churn scores
   |               Writes predictions back into ---> [ BigQuery Tables ]
   |
   +---> 3. Warehouse Return Conveyor Camera (Photos of Parcels)
   |        |--> [ Deep Learning Custom Model: Vertex AI Vision Endpoint ]
   |               Classifies box photo in 100ms -> "Intact" vs "Damaged"
   |
   +---> 4. New Raw Supplier Specs -> Need Website Descriptions
   |        |--> [ Generative AI: Gemini API on Vertex AI ]
   |               Generates clean Ukrainian/English product descriptions & tags
   |
   +---> 5. Employee Question: "What is our return rule for opened headphones?"
   |        |--> [ RAG: Vertex AI Search / RAG Engine ]
   |               Step A: Searches internal PDF manuals in Cloud Storage
   |               Step B: Sends top 3 paragraphs to Gemini to write a grounded answer
   |
   +---> 6. Customer Request: "Where is order #88310? Change delivery to Store #12."
            |--> [ AI Agent: Vertex AI Agent Builder (ADK on Agent Engine) ]
                   - Uses Brain (Gemini Foundation Model) to plan steps
                   - Calls Tool 1: SQL/API check on Order #88310 status
                   - Calls Tool 2: RAG check on store delivery rules
                   - Calls Tool 3: ERP API to update destination to Store #12
====================================================================================

Look closely at this diagram. Notice why this system is reliable and cost-effective:

  • We do not use an expensive LLM to read standard invoices when Document AI is built specifically for that job.
  • We do not ask a chat model to guess next week’s store sales when BigQuery ML can calculate statistical time-series forecasts accurately from historical rows.
  • We do not train a custom model from scratch to answer employee questions when RAG (Vertex AI Search + Gemini) can read updated PDF manuals immediately.

Part 4: Practice — The Technology Decision Matrix

When a business manager or client comes to you with a new task tomorrow, how do you choose the right technology in Google Cloud?

Let us examine the five most common business tasks—Classification, Forecasting, Search, Text Generation, and Process Automation—and build a practical decision matrix. Always follow the basic engineering rule: Start with the simplest, cheapest, most deterministic tool that solves the problem, and only move to more complex systems when the simple tool hits a limit.

Master Technology Selection Matrix

Business TaskWhat the Input Looks LikeFirst Choice (Simple / Fast to Launch)Advanced Choice (High Scale / Custom Need)Primary Google Cloud ServicesCommon Engineering Mistake to Avoid
1. Classification (Sorting items into known groups)Tables: Customer rows, transactions.

Text/Media: Support tickets, product photos.		For Text ( few examples): Foundation Model (Gemini Flash) via prompt.

For Standard Docs/Images: Cloud Vision API / Document AI.		For Tables or High-Volume Text/Images: Custom Classical ML (XGBoost) or Custom Deep Learning.BigQuery ML (LOGISTIC_REG, BOOSTED_TREE_CLASSIFIER), Vertex AI AutoML, Gemini Flash API.Calling a massive LLM 1,000 times a second to classify tabular numbers, or spending 3 months training a custom text classifier when you only have 200 tickets a day.
2. Forecasting / Prediction (Predicting future numbers)Historical tables with dates, quantities, prices, and promotions.Statistical Time-Series / Regression: Classical ML directly inside the data warehouse.Deep Learning Forecasting: Pre-trained time-series foundation models or custom multi-variable pipelines.BigQuery ML (ARIMA_PLUS, AI.FORECAST with TimesFM, Linear Regression), Vertex AI Forecast.Pasting a table of numbers into a chat LLM prompt and asking it to guess next month’s sales. LLMs predict words, not mathematical time-series trends.
3. Search (Finding relevant items or facts)Company PDFs, wiki pages, product catalogs, legal contracts.Managed Semantic & Hybrid Search: Pre-built enterprise search with automatic chunking and ranking.Custom Vector Pipeline: Custom document parsing + custom embedding models + custom vector index.Vertex AI Search (Agent Search), Vertex AI Vector Search, Vertex AI Text Embeddings API.Relying only on SQL LIKE '%word%' (which fails when users use synonyms), or dumping 500 pages into an LLM prompt on every single search query.
4. Text Generation (Writing, summarizing, translating)Prompts, raw notes, call transcripts, retrieved RAG paragraphs.For Translation: Cloud Translation API.

For General Writing: Prompting a Foundation Model.		For Factual Company Answers: RAG + Foundation Model.

For Specific Tone/Format: Supervised Fine-Tuning of a small model.		Gemini on Vertex AI, Vertex AI Model Garden (Gemma, Claude), Vertex AI RAG Engine, Check Grounding API.Asking a raw Foundation Model to write answers about private company rules without connecting a RAG retrieval layer first (causes hallucinations).
5. Process Automation (Executing multi-step tasks)Invoices, database events, user requests requiring API actions.Fixed Rules: Standard code/workflows + Ready-Made AI APIs (like Document AI).Dynamic Decisions: AI Agents with tool-calling, RAG memory, and human-in-the-loop approvals.Vertex AI Agent Builder (ADK, Agent Studio, Agent Engine), Document AI, Cloud Run / Workflows.Using an autonomous AI Agent for a strict, unchanging financial calculation where a normal 20-line Python or C# script is 100% predictable and free.

Detailed Engineering Rules for Each of the 5 Tasks

To use the matrix above in real projects, walk through these specific questions for each category:

Task 1: Classification (Sorting Items into Categories)

Classification means you have a fixed list of buckets (for example: Spam vs. Not Spam, Fraud vs. Normal Transaction, or Delivery Issue vs. Billing Issue), and the system must pick the right bucket.

  • When to use Classical ML (BigQuery ML): If your input is a database row of numbers and categories (like user age, order total, payment method, and past return rate), always choose Classical ML (Logistic Regression or Boosted Trees in BigQuery ML). It is fast, explainable, and shows you exactly which column influenced the decision.
  • When to use a Foundation Model (Gemini Flash): If your input is unstructured text (for example, classifying 500 customer emails a day into 10 department categories) and you do not have 10,000 labeled historical emails to train a model, use Gemini with a clear prompt. It works on day one with zero training data.
  • When to train a Custom Deep Learning Model: If that same text or image classification task grows to 5 million requests per day and needs to run in 20 milliseconds, calling a large foundation model becomes too expensive. At that point, use your historical data to train a small, specialized custom model on Vertex AI.

Task 2: Forecasting and Numerical Prediction

Forecasting means predicting a continuous number in the future: next month’s revenue, server CPU load tomorrow, or how many minutes a courier will need to deliver an order.

  • Never use a text LLM for raw numerical forecasting. Language models are trained to predict the next word in a sentence, not to calculate seasonality, holidays, or statistical confidence intervals.
  • When to use Statistical / Classical ML: If you have historical time-series tables in BigQuery, start with BigQuery ML ARIMA_PLUS or Google’s built-in AI.FORECAST (TimesFM) function. These algorithms automatically detect weekly cycles (for example, higher sales on Saturdays), yearly holidays, and sudden spikes without manual tuning.
  • When to use Custom XGBoost or Tabular Deep Learning: When your prediction depends on dozens of external changing variables at once (such as competitor prices, advertising spend, weather, and store square meters), train a custom XGBoost or Tabular model in BigQuery ML or Vertex AI.

Task 3: Search (Finding Information)

Modern search is no longer just matching exact letters in a database. If a customer searches an online electronics store for “cordless phone charger for traveling,” a traditional SQL search might fail because the product is named “Portable Power Bank 10000mAh.”

  • When to use Managed Enterprise Search: If you want employees to search across company PDFs, Google Drive, and Confluence, or customers to search your website catalog by meaning, use Vertex AI Search (Agent Search). It combines classic keyword search (important for exact product serial numbers like RK3304) with vector search (important for meaning and synonyms) out of the box.
  • When to build Custom Vector Search: If you have a specialized engineering requirement—for example, matching similar product photos visually or filtering millions of vectors in under 10 milliseconds inside a custom microservice—use the Vertex AI Embeddings API to turn items into vectors and index them in Vertex AI Vector Search.

Task 4: Text Generation (Creating, Summarizing, and Explaining)

Whenever the output must be newly written sentences—product descriptions, meeting summaries, translated catalogs, or polite answers to customer questions—you are in the Generative AI domain.

  • Rule 1 — Translation only: If the only goal is translating text from one language to another at scale, use the Cloud Translation API first. Only use an LLM like Gemini if you need the translation to simultaneously rewrite the tone or adapt marketing idioms.
  • Rule 2 — Generating from facts you already have in the prompt: If you pass the raw data directly inside the request (for example: “Summarize this 2-page call transcript into 3 bullet points”), call a Foundation Model (Gemini) directly on Vertex AI.
  • Rule 3 — Generating answers from a large knowledge base: If the model needs to find the facts first from thousands of company documents, you must put RAG (Vertex AI Search / RAG Engine) in front of the Foundation Model and verify the output with grounding checks.

Task 5: Process Automation (Turning Decisions into Actions)

Process automation is where software replaces repetitive human clicking and data entry across different systems.

  • Rule 1 — Fixed, predictable steps: If the process always follows the exact same steps (for example: 1. Receive PDF invoice by email -> 2. Extract fields -> 3. Compare sum with purchase order in SQL -> 4. If equal, mark as approved), do not use an autonomous AI Agent. Write a standard, deterministic program (in C#, Python, or F# on Cloud Run) and call a Ready-Made AI API (Document AI) only for Step 2 (reading the PDF). Normal code never hallucinates, is easy to test, and costs almost nothing to run.
  • Rule 2 — Dynamic, unpredictable requests: Use an AI Agent (Vertex AI Agent Builder / ADK) only when the entry point is unstructured human intent and the sequence of steps changes depending on what the system discovers along the way. Even then, give the agent narrow, safe tools with strict permission boundaries (scoped identity), and require human approval before executing high-risk actions like deleting data or sending money.

How to Apply This Map in Practice

When you look at Google Cloud’s AI ecosystem as an engineer, remember that complexity is a cost, not a badge of honor. Every time you move one step further along the chain—from a Ready-Made API or a SQL model in BigQuery ML up to a multi-step AI Agent—you gain flexibility, but you also pay more per request, wait longer for responses, and take on more monitoring work.

By keeping this map in mind, you can match every business requirement to its exact engineering tool: Classical ML for structured numbers and forecasts, Deep Learning and Ready-Made APIs for seeing and hearing standard documents and media, Foundation Models and Generative AI for writing and summarizing, RAG for grounding answers in real company documents, and AI Agents for orchestrating multi-step actions across your software systems.

Similar Posts