Vertex AI for RAG: the pieces worth using and the ones to build yourself

Google Cloud has more RAG building blocks than any other provider, and that is its main problem. Vertex AI Search, Vertex AI Vector Search, the grounding API, Agent Builder and the RAG Engine overlap enough that the first architecture decision is which of them you are actually using.

Here is how we choose, based on what survives contact with real users.

Vertex AI Search when the corpus is documents and the answer is a search result

If your use case is "search our documentation and answer with citations", Vertex AI Search (the search-and-conversation product) does the whole pipeline: crawl or ingest, chunk, embed, retrieve, rerank and generate with citations. It includes a reranker, which is a component teams usually skip and then rediscover when precision disappoints.

This is genuinely the right answer for internal knowledge bases, support portals and product documentation. Take it. The work you keep is the evaluation set and the permission model.

Assemble it yourself when retrieval is not document search

The moment the answer needs to combine a document with a row from a database, or retrieval depends on the user's permissions in a non-trivial way, or you need a retrieval step that filters by structured metadata before the vector search, you want the pieces:

• Vector Search (formerly Matching Engine) for the index. It is fast and scales well, but note the deployed index endpoint has a real fixed cost per hour — for small corpora, AlloyDB or Cloud SQL with pgvector is cheaper and simpler, and lets you join vectors against your relational data in one query, which is often the whole reason you were assembling it yourself.
• The embedding and generation models on Vertex, so the switch between model versions is a parameter.
• Your own orchestration in Cloud Run, because this is ordinary application code and it benefits from being testable.

The two managed pieces worth taking regardless

Grounding with Google Search or with your own data, exposed as a generation-time setting, returns support scores per claim. Use the score as a gate: below a threshold, the assistant says it does not know rather than guessing. This single behaviour does more for user trust than any amount of prompt engineering.

The Gen AI evaluation service runs model-graded metrics over a dataset. You still have to write the dataset — fifty to two hundred real questions with expected sources — but you do not have to build the harness. Wire it into CI so that a prompt change runs the evaluation before merge, the same way tests do.

Permissions, which is where these projects die

A vector index has no concept of your org chart. Two patterns work:

Metadata filtering at query time. Every chunk carries the ACL of its source document as a namespace or restrict field, and the query passes the user's groups. Fast, and it holds as long as your ingestion pipeline keeps ACLs current — which means re-syncing when permissions change, not only when content changes.

Per-tenant indexes. More expensive and much simpler to prove correct. For a handful of large tenants with strict isolation requirements, this is the right answer and the cost is worth the audit conversation it saves.

What does not work is filtering after retrieval in the application, because the model has already seen the chunks by then. This is the failure we are most often called in to fix, and the fix is re-ingestion.

Cost, and the number to track

Three lines: generation tokens, embedding tokens (mostly a one-off at ingestion, plus re-embedding when you change model), and the index or database. Add Cloud Run, which is usually noise by comparison.

Track cost per question from the first week, labelled by team, with billing exported to BigQuery. Batch embeddings rather than calling per document, cache the embeddings of unchanged chunks between ingestion runs, and use a small fast model for query rewriting and classification. Those three habits typically halve the bill without touching answer quality.

The general version of these decisions is in the four decisions that make or break a RAG project; the ordering here is specific to what Vertex hands you for free.

What to do this week

Write the evaluation set: fifty real questions from real users, each with the document that should answer it. Then run your current system against it and record the score. Everything after that is measurable, and almost nothing before it was.