Chunking and embeddings decide RAG quality, not the generator

A client had an assistant answering 60 percent of questions acceptably. The plan on the table was to move to a larger, more expensive generation model. We ran their evaluation set with retrieval measured separately and found the correct chunk was in the top five results only 64 percent of the time. No generator can answer from a document it was never shown. The larger model would have produced more fluent wrong answers at three times the cost.

Retrieval sets the ceiling. The generator can only fail to reach it. Almost everything that determines retrieval quality happens before a single token is generated: how you split documents, what you attach to each piece, which embedding model you use, and whether you rerank.

Split on document structure, not character count

RecursiveCharacterTextSplitter at 1000 characters with 200 overlap is the default in every tutorial and it is wrong for most real corpora. It cuts tables in half, separates a heading from the paragraph it introduces, and splits a numbered procedure at step 4.

Split on structure instead. For Markdown and HTML, split on heading boundaries and keep sections whole up to a size limit, only falling back to sub-splitting for genuinely long sections. For contracts and policies, split on clause or article. For code, split on function or class. The unit should be something a human would answer with if you asked them to quote the relevant part.

Overlap still matters, but it is a patch for a bad boundary, not a strategy. When you split by structure, 10 to 15 percent overlap is enough. When you split by character count you need much more, which is a sign you are indexing the same text several times to compensate for cutting in the wrong places.

Size: aim for chunks that carry one idea. In practice 300 to 800 tokens works for prose. Smaller chunks retrieve more precisely and give the generator less to work with; larger chunks are the reverse. If you cannot decide, index small and use parent context.

Every chunk needs to know where it came from

A chunk that reads "set this to 30 seconds" is unretrievable and unusable. The fix is to prepend the document title and the full heading path to the chunk text before embedding it, so the embedding itself carries the topic. It is a two-line change and it is regularly worth several points of recall.

Beyond that, the parent document pattern: embed and search over small chunks, but return the larger surrounding section to the generator. You get the precision of small-chunk retrieval and the context of large-chunk generation. Store the parent ID as metadata and fetch it after ranking.

And keep provenance on every chunk: source system, document ID, section, version, last modified. Without it there is no citation, and without a citation nobody trusts the answer. This is the same requirement that drives permission filtering in the four decisions that matter more than the model.

Tables and PDFs are where pipelines quietly break

Most corpora that matter contain PDFs, and PDFs have no structure, only positioned glyphs. Naive text extraction turns a two-column layout into interleaved nonsense and a table into a stream of numbers with no headers.

What works, in order of effort:

• Use a layout-aware extractor that produces reading order and identifies tables as tables. Every major cloud has a document-understanding service; open-source layout parsers are viable too.
• Serialise each table as Markdown or HTML, keep it in one chunk, and prepend its caption and the surrounding heading. A table split across chunks is worse than no table.
• Scanned documents need OCR first, and OCR errors propagate silently into the index. Sample the extracted text before indexing 40,000 pages of it.

Spend your time here. In our experience document parsing quality moves end-to-end accuracy more than any prompt change.

Choosing the embedding model, knowing you are stuck with it

Changing embedding model means reindexing the entire corpus. Different models produce different dimensions and, more importantly, incompatible vector spaces, so you cannot mix old and new vectors. On a large corpus that is hours of compute and a migration plan with dual indexes and a cutover.

So choose deliberately, on your own data:

• Multilingual capability if your documents or your users are not all in English. Many strong English models degrade sharply on other languages, and a European corpus is rarely monolingual.
• Domain fit. Test on your evaluation set. A general model often beats a domain-specific one, and the only way to know is to measure retrieval recall for each candidate.
• Dimensionality. 1536 or 3072 dimensions cost more to store and search than 768. Some recent models support truncating the vector with modest quality loss, which is worth testing: halving dimensions roughly halves index memory, and index memory is what determines your vector database sizing and cost.
• Hosting. An API-hosted embedding model adds a network hop to every query and every document. A self-hosted small embedding model is cheap to run and removes that dependency.

Reranking is the best effort-to-result change available

Retrieve 30 to 50 candidates with hybrid search, then run a cross-encoder reranker over them and keep the top 5. Unlike the bi-encoder that produced your index, a reranker sees query and document together, so it judges relevance far better.

The gain is typically large: we routinely see top-5 accuracy move by 10 to 20 points on corpora that had no other changes. It costs one extra model call of 100 to 300ms and no reindexing, and you can add it to an existing pipeline in a day. Managed rerankers exist in each cloud's AI service, and small open-weight rerankers run fine on CPU at modest volume.

Things people forget

• Measure retrieval separately from answers. Recall@5 needs no generator, costs nothing, and tells you where the fault actually is.
• Deletions. When a source document is deleted or superseded, its chunks must leave the index. Stale chunks produce confidently outdated answers.
• Near-duplicates. Five versions of the same policy fill your top 5 with the same text. Deduplicate at ingest and prefer the current version.
• Embed the query the same way you embedded the documents. Some models expect an asymmetric prefix for queries versus passages. Getting this wrong silently costs recall.
• Chunk count is a cost driver. It sets index memory, embedding spend and reindex time. Halving chunk size doubles all three.

We treat parsing, chunking and reranking as the first week of any applied AI project, before the model conversation starts.

What to do this week

Take twenty questions your users actually asked, run them through your current retrieval, and record whether the correct chunk appears in the top five. If it is under 85 percent, stop tuning prompts. Add a reranker over 30 candidates and measure again.