Versioning prompts and evaluating automations without building a research lab

The prompt inside an automation is the only part of a production system that routinely gets edited by someone in a browser, with no diff, no review and no test. Teams that would never allow a config change to production without a pipeline will happily rewrite a classification prompt at four in the afternoon because a colleague complained about one output.

Then quality moves, in a direction nobody measures, and three weeks later somebody asks what changed.

Get the prompt out of the workflow

Step one is mechanical: prompts live in files in a repository, not inside a node's configuration. The workflow references a key; a small service or a table resolves the key to the current version.

This buys four things at once — a diff, a reviewer, a history, and the ability to roll back in seconds rather than by remembering what the text used to say. It also means the prompt hash you log on every run means something, which is what makes observability able to answer "what changed".

Version the model alongside it. A prompt and a model version are one unit; a prompt tuned against one model is not validated against another, and "latest" as a model setting means your production behaviour changes when a vendor ships.

The smallest useful evaluation

You do not need an evaluation framework. You need fifty examples and a script.

Build the set from production. Fifty real inputs with the correct output attached. The cheapest source is your human review queue: every correction a reviewer made is a labelled example, which is why the correction reason field pays for itself. Include the weird ones deliberately — the supplier whose invoices are photographs of a screen, the customer whose name breaks your parser. A set of only easy cases measures nothing.

Score what you actually care about. For classification and extraction this is exact match per field, and it is unambiguous. For generated text, pick three criteria you can state as yes-or-no questions — did it stay in the requested language, did it avoid inventing a number that was not in the source, did it follow the format — and check those. Vague quality scoring, whether by a human or by a model, produces numbers that move without meaning.

Run it on every change. A script that takes prompt version and model, runs fifty examples, prints a score and a diff of which examples changed verdict. Ten minutes to run, cents to execute. This is the whole thing.

The diff matters more than the score. A change that takes you from 86 to 88 percent by fixing four cases and breaking two is a different decision from one that fixes two and breaks none, and the aggregate hides it.

Regression, not perfection

The purpose of this set is not to prove the automation is good. It is to stop a change from making it worse.

That reframing lowers the bar to something a team will actually maintain. Fifty examples in a CSV, a script, and a rule that the prompt does not change without the script running. Over a year the set grows to two hundred, mostly from failures that reached production once and never will again, and at that point you have something genuinely valuable that nobody ever had to schedule a project to build.

Model upgrades are the moment this pays off

Every few months a better, cheaper model appears. Without an evaluation set, migrating is an act of faith followed by a month of anecdotes. With one, it is a twenty-minute test and a number, and you find out immediately about the specific case where the new model formats dates differently.

That single use justifies the whole practice, because model upgrades are not optional forever — providers retire versions, and the retirement notice is not when you want to start finding out what breaks.

What to do this week

Export the last thirty items your reviewers corrected, with the input and the corrected output, into a CSV. That file is your evaluation set and it already exists in your queue. The script that runs it is an hour of work and you will use it within the month.