Evaluating an LLM feature without building a research lab

The moment that forces this is always the same. A new model version comes out, or someone rewrites the system prompt, and the question is whether to ship it. The team tries ten questions by hand, the answers look nicer, it ships, and two weeks later support notices the assistant stopped citing sources.

Nobody could have caught that with ten hand-tried questions. You need a set of cases and a number, and the good news is that the version of this that works is far smaller than a research programme. A hundred well-chosen cases will catch most regressions that matter.

Build the set from real traffic and real failures

The worst evaluation set is the one someone invented at a desk. It tests what they imagined users would ask, which is systematically different from what users ask.

Three sources, in order of value:

Production traffic. Sample real queries, stratified so you are not just testing the common case. Take frequent questions, rare ones, long ones, ambiguous ones, and ones in each language you serve.

Reported failures. Every complaint, every thumbs-down, every support escalation becomes a case. This is the highest-value source because each one is a known failure with a known correct answer, and it means the set gets better every time something goes wrong.

Adversarial cases you write deliberately. Questions with no answer in the corpus, where the correct behaviour is to say so. Questions that mix two topics. Questions containing an instruction, to check injection handling. Questions about another tenant's data, to check the permission boundary.

A hundred to three hundred cases is the right size to start. Large enough to be stable, small enough that a human can review the whole set when something looks wrong. Store it in the repository as data, versioned alongside the prompts as described in versioning prompts and evaluating automations, and grow it deliberately rather than letting it sprawl.

Handle personal data properly. An evaluation set built from production traffic is production data, with the same retention and access rules.

Measure the components, not one overall score

A single "quality" number tells you something changed and nothing about what. Split it by the thing being measured, because the fixes are completely different.

Retrieval. Is the correct chunk in the top k results. Needs no model call, costs nothing, and is where most quality problems actually live. Measure it separately and first; the reasoning is in chunking and embeddings.

Groundedness. Is every claim in the answer supported by the retrieved context. This is the practical hallucination metric.

Answer correctness. Does it match the expected answer, for the cases where there is one.

Completeness. Did it answer the whole question or only the first half.

Refusal behaviour. On the cases where the correct response is "I do not know", did it say so. Teams forget to measure this and then wonder why the assistant is confidently wrong about things outside the corpus.

Format and policy compliance, which is usually a deterministic check.

Add cost and latency per case to the same report. A change that improves correctness by two points and doubles the bill is a decision, not an improvement, and you want both numbers in the same table.

The judge, and why it needs calibrating

Most of these metrics cannot be computed with string matching, so you use a model to grade. That works and it introduces a measurement instrument that has its own errors.

Judges have known biases. They prefer longer answers. They prefer answers in a style similar to their own output, which means a judge from the same family as your generator will flatter it. They are sensitive to option order in comparisons. And they are inconsistent at the margin, giving different scores to the same answer on different runs.

The fix is calibration, and it is one afternoon of work. Have a human label thirty to fifty cases. Run the judge on the same cases. Measure agreement. If the judge agrees with the human most of the time, you can trust its aggregate numbers. If it does not, fix the judging prompt, usually by making the question narrower and asking for a binary decision with a stated reason rather than a score out of ten.

Two practices that make judges much more reliable: ask for a binary or three-way verdict rather than a numeric score, since models are far better at "is this claim supported, yes or no" than at "rate this 1 to 10"; and give the judge the evidence rather than asking from memory, so groundedness is judged against the actual retrieved context.

Recalibrate when you change the judge model. The judge is a dependency with a version.

Offline gates the deploy, online tells the truth

Offline evaluation runs the set in CI on every prompt change, model change and retrieval change. It is fast, repeatable and cheap enough to run on a pull request. It is also artificial: a fixed set cannot cover what users will do tomorrow.

Online evaluation measures production. Thumbs up and down, though the signal is sparse and biased towards complaints. Implicit signals, which are better: did the user rephrase the question immediately, did they escalate to a human, did they stop. And an asynchronous judge running on a sample of real traffic, which is the single most useful ongoing measurement and costs a fraction of judging everything.

Run both. Offline catches regressions before release. Online catches the drift that a fixed set cannot see, including a provider silently updating a model behind a stable endpoint.

The gate that blocks a deploy

Make the rule explicit, or the evaluation becomes a report nobody acts on.

A workable default: no metric may drop by more than a small tolerance, no case that previously passed may now fail on the critical subset, and cost per query may not rise beyond a stated threshold without a decision. Maintain a critical subset of twenty or thirty cases that must pass absolutely, covering the failures that would be unacceptable: data leakage across tenants, an invented policy commitment, a refusal to answer something core.

Aggregate scores hide regressions. A change that improves eighty cases by a little and breaks three badly can show a net gain. Always report which specific cases changed direction, and look at them.

The things people forget

• Non-determinism. Set temperature to zero for evaluation where you can, and run multiple times where you cannot. A one-point difference on a single run is noise.
• The corpus changes under you. If documents are updated, an expected answer can become wrong. Version the corpus snapshot with the set.
• Language coverage. If you serve several languages, the set must too, and quality frequently differs a lot between them.
• Evaluation costs money. Three hundred cases with a judge on every pull request adds up. Run the full set on merge and a subset on each push.
• Nobody reads a dashboard. Put the diff in the pull request, next to the code, the same way cost belongs there.

What to do this week

Take the last twenty complaints or thumbs-down from your assistant, write the correct answer for each, and save them as a file in the repository. That file is your evaluation set. Run your current system against it and count how many it gets right. Whatever the number is, you now have a baseline, and every future change has something to be compared against. We build this in the first fortnight of an applied AI project.