Fine-tuning, RAG or a better prompt: they fix different problems
Fine-tuning teaches behaviour and format. Retrieval supplies knowledge. Prompting shapes the immediate task. Most teams reach for the expensive one to fix a problem it cannot fix.
A client wanted to fine-tune a model because the assistant did not know their product catalogue. It was the wrong tool, confidently chosen, and it would have cost a quarter to discover that.
Fine-tuning does not reliably teach facts. It teaches behaviour: how to respond, in what format, in what voice, following which implicit rules. Facts that change belong in retrieval, because a fine-tuned fact is frozen at training time and updating it means training again.
The diagnostic question is therefore not "should we fine-tune" but "what kind of thing is missing".
Match the failure to the technique
The model does not know something. It cannot answer questions about your documents, your customers, your catalogue, or anything that happened after its training data ended. This is retrieval. The four decisions that matter are in RAG on your own cloud, and no amount of fine-tuning fixes a knowledge gap that changes weekly.
The model knows but responds wrongly. It ignores your format, is too verbose, will not follow a nine-step procedure consistently, uses the wrong register, or refuses things it should handle. This is prompting first, and fine-tuning if prompting stops scaling.
The model cannot do the task at all. It fails at a specialised classification, a domain-specific extraction, a rare language or a notation it has barely seen. This is genuinely fine-tuning territory, and it is rarer than people assume.
The output is well-formed and wrong. Inconsistent quality, hallucinated specifics, occasional confident nonsense. This is an evaluation and grounding problem, not a training problem, and the approach is in evaluating an LLM feature.
Exhaust prompting first, because it is nearly free
The order is not a moral position, it is an economic one. A prompt change ships in minutes and costs nothing. A fine-tune is a data project.
Things that work and are routinely skipped: putting the instruction structure before the content, giving five to ten worked examples in the prompt, asking for a specified output schema and validating it, breaking a complex task into two calls instead of one, and giving the model an explicit way to say it does not know.
Few-shot prompting in particular covers a large share of what teams try to fine-tune for. If ten examples in the prompt produce the behaviour you want, you have your answer, and the only argument for fine-tuning is the cost of those tokens on every call.
Also try a better model before training a worse one. The gap between model tiers is often larger than the gap fine-tuning would close, and switching is a configuration change.
What fine-tuning actually buys
When prompting has genuinely plateaued, fine-tuning buys three things.
Consistency. A behaviour that the prompt produces eight times in ten becomes nearly reliable. For a workflow with an automated step downstream, that difference is the product.
Token cost and latency. Behaviour baked into the weights does not need a two-thousand-token instruction block on every call. At high volume this is the strongest commercial argument, and it compounds with the arithmetic in what an AI automation costs per run.
Small-model capability. The most valuable pattern available: use a large model to produce high-quality outputs, then fine-tune a much smaller one on them for the narrow task. The small model can match the large one on that task at a fraction of the cost and latency. This is distillation, and for a well-defined, high-volume task it is often the best decision available.
Parameter-efficient methods make this accessible. Rather than updating all weights, you train a small set of adapter weights, which cuts training cost and memory dramatically, produces artefacts of megabytes rather than gigabytes, and lets you serve several adapters on one base model. For nearly every business case this is the right form of fine-tuning.
The data is the project
Underestimating this is the standard failure. The compute is the cheap part.
You need examples of the behaviour you want, in the format you want, with the quality you expect. Typically hundreds to low thousands for a narrow task with parameter-efficient methods. They must be consistent, because the model learns your inconsistencies faithfully, and a set where two annotators formatted things differently teaches the model to be arbitrary.
Where they come from: existing production logs with human corrections, which is the best source; expert annotation, which is slow and expensive; or synthetic generation from a larger model, which is fast and requires human review or you are training on the larger model's mistakes.
Hold out a test set before you start. Check the provider's terms on using model outputs to train, since they vary. And if there is personal data in the examples, the training set is personal data with retention and deletion obligations attached, and it is far harder to delete from model weights than from a database, which is the point where GDPR as engineering controls becomes architectural.
The maintenance cost nobody quotes
A fine-tuned model is a fork you own forever.
When the base model is deprecated or a better one appears, your tuned version does not come along. You retrain, re-evaluate and re-deploy, and the improvement you were chasing may already be in the new base model for free.
When your requirements change, a prompt is an edit and a fine-tune is another training cycle.
And every fine-tune needs its own evaluation set and its own regression testing, because you can no longer rely on the provider's general quality signals.
Budget this as ongoing engineering, not a one-off cost. For a stable, narrow, high-volume task the trade is clearly worth it. For a feature still finding its shape, it is a commitment made too early.
A decision order that holds up
- Better prompt, with examples and a validated output schema.
- A stronger model, measured rather than assumed.
- Retrieval, if the gap is knowledge. Then improve the retrieval, since most RAG quality problems are chunking and reranking.
- Prompt caching and a smaller model for the easy subset, if the problem is cost.
- Fine-tuning, ideally as distillation into a small model on a narrow task.
- Training from scratch, which for almost every company is never.
Stop as soon as the numbers are good enough. The step you skip is free.
The things people forget
- Fine-tuning does not remove the need for retrieval. Tuned behaviour plus retrieved facts is the common production shape.
- It can make the model worse elsewhere. Narrow training degrades general capability. Evaluate beyond the target task.
- Provider hosting for custom models differs. Check availability, region and price before designing around it, especially if data residency matters.
- Version everything. Base model, adapter, training set and evaluation set are one release, and reproducing a result six months later needs all four.
- Measure before and after on the same set. Otherwise you cannot tell whether the improvement came from the tuning or from the prompt you also changed.
What to do this week
Take the failure you were going to fine-tune away, and write ten examples of the correct output into the prompt. Run your evaluation set. If it fixes it, you have saved a quarter. If it does not, you now have the first ten rows of the training set you actually needed. We work through this order in an applied AI project.