RAG on your own cloud: the four decisions that matter more than the model

An assistant over your own documentation is the first AI project most companies run, and the one most likely to stall between pilot and production. The pilot impresses because someone asks it three questions they already know the answer to. Production disappoints because two hundred people ask it questions nobody anticipated.

The gap is rarely the model. Whatever you pick, the retrieval pipeline decides what it gets to read. These are the four decisions we settle in the first week of every applied AI project, in the order they should be made.

1. What a chunk is, and what travels with it

Splitting documents into fixed windows of five hundred tokens is what every tutorial does and what every production system regrets. A chunk has to be a unit that makes sense on its own: a section with its heading, a table with its caption, a procedure with all its steps.

Two things should travel with every chunk into the index:

• Provenance: source system, document, section, version and last modified date. Without it, the answer cannot cite anything and nobody will trust it.
• Context: the document title and the heading path. A paragraph that says "set it to 30 seconds" is useless without knowing it belongs to "Timeouts > Database connections" in the platform runbook.

Re-chunk when documents change, not on a schedule. A nightly full reindex of a wiki that changes ten pages a day is wasted money and a day of stale answers.

2. Who is allowed to read what

This is the decision that stops most pilots from ever reaching production, so make it first. Your documents already have permissions in Confluence, SharePoint, Google Drive or the ticketing system. The assistant must inherit them, per user, at query time.

The pattern that works: store the source's access control list alongside each chunk, resolve the user's groups when they ask, and filter the vector search by those groups before ranking. Filtering after ranking leaks the existence of documents through the ranking itself and breaks when the top results are all filtered out.

If your vector store cannot filter on metadata before search, that is a reason to change vector stores, not to skip the requirement.

3. How you will know it is good

"It seems to answer well" is not a test. Before writing the pipeline, write the evaluation set: fifty to a hundred real questions from real users, each with the document that should be cited and a short reference answer. Support tickets and search logs are the best source; they contain the questions people actually ask, including the badly phrased ones.

Then measure three things separately on every change:

• Retrieval: is the right chunk in the top five? This is where most failures are, and it costs nothing to evaluate because no model is involved.
• Answer: does the answer match the reference? An LLM judge with a rubric works well enough to catch regressions, as long as a human spot-checks a sample each week.
• Refusal: when the answer is not in the documents, does it say so? Count these. An assistant that never refuses is inventing.

Run this set in CI. A change to the chunking strategy or the prompt that drops retrieval from 88 to 71 percent should fail the build, not surface in a complaint three weeks later.

4. What every query costs, and where it runs

Cost per query is knowable before launch, and it is the number that decides whether the project survives its first budget review. Three levers, in order of impact:

• Context size. Sending ten chunks of a thousand tokens each is ten times the cost of sending the three that matter. Better retrieval is the cheapest cost optimisation there is.
• Model tiering. A small model classifies the question and answers the routine ones; a larger one handles what the small one is not confident about. In our projects, the small model handles 70 to 80 percent of traffic.
• Where it runs. Models available inside your cloud, such as through Amazon Bedrock, keep the data in your account and let you set budgets and alarms with the same tools you already use. Hosting your own open-weight model makes sense above a certain volume, and not before.

Treat this the way you would any other cloud cost problem: put a monthly budget alarm in place the day the pilot starts, with the same seriousness as a database alarm. Nobody notices an assistant that costs 400 euros a month; everybody notices the one that cost 9,000 in a month because a batch job started calling it.

The model decision, last

By the time these four are settled, the model choice is small: run the evaluation set against two or three candidates and pick the cheapest one that clears the bar. It will change within a year anyway, and a pipeline built around these decisions will let you swap it in an afternoon.