A client asked me last month whether they should fine-tune a model on their support docs. I asked one question back and the meeting ended nine minutes early. The question was: when someone edits one of those docs, does the answer need to change that day?
They said yes, obviously. So: RAG. We were done.
I have had some version of that conversation maybe fifteen times now, and it almost always resolves on that one question. Which is a little embarrassing, because I used to spend an hour on comparison tables.
The question that settles most of these
Fine-tuning bakes something into the weights. Retrieval looks something up at query time. That’s the whole distinction, and everything else follows from it.
If your knowledge has an edit history, you want retrieval. Support docs, pricing, policy pages, product specs, anything with a last_updated column. The moment a human changes that row, the system should say the new thing. A fine-tuned model will keep confidently saying the old thing until you retrain, and retraining isn’t a thing most teams do on a Tuesday afternoon because someone corrected a refund window.
If what you want is a consistent output shape, a house tone, a specific classification scheme, or a domain vocabulary the base model keeps mangling, that’s fine-tuning territory. OpenAI’s own fine-tuning guide frames it around teaching style and format rather than teaching facts, and that framing has held up in every project I have been near.
The trap is that both approaches will produce something demo-able in a week. The difference shows up in month four, when the docs have drifted and nobody can explain why the bot is quoting a policy that was deleted in March.
What the benchmark numbers actually tell you
There is a reasonably large empirical comparison of RAG, LoRA, and DoRA that ran 20,000 FAQ-style queries against a knowledge base of 400,000 entries. DoRA came out on top on their metrics: 90.1% accuracy, a relevance score of 0.88, and 110ms per query. You can read the paper yourself.
I want to be careful about what that does and doesn’t mean. It’s a FAQ benchmark. FAQ data is stable, it was fixed at evaluation time, and the fine-tuned variants never had to deal with the knowledge changing after training. That’s exactly the scenario where fine-tuning looks best, and it isn’t the scenario most business teams are actually in.
What I take from a result like that is narrower and more useful: if your corpus really is static, parameter-efficient fine-tuning is now cheap enough and good enough that dismissing it as “too expensive, just use RAG” is lazy. The 110ms figure matters too. A tuned model answers from weights. A RAG pipeline has to embed the query, hit a vector store, rerank, and stuff the result into a prompt, and that path is rarely under 300ms end to end unless you have worked at it.
So the tradeoff is not quality versus cost. It is freshness versus latency, mostly.
RAG fails in ways that look like model failures
This is the part I wish someone had told me three years ago. When a retrieval system gives a bad answer, the instinct in the room is always “the model isn’t smart enough, can we try a bigger one.” Usually it isn’t the model.
The paper on seven failure points in RAG systems is a good checklist here, drawn from actual deployed systems rather than a lab setup. The failures cluster around retrieval and formatting: the answer was never in the corpus at all, the right chunk existed but ranked below the cutoff, the chunk was retrieved but got trimmed out of the context window, the content was there but the model pulled the wrong specificity level.
Every one of those is an engineering problem with a boring fix. Better chunking. A reranker. Actually measuring retrieval recall separately from answer quality, which is the single highest-value thing most teams aren’t doing. If you can’t say what percentage of the time the correct document lands in your top-k, you don’t know whether you have a retrieval problem or a generation problem, and you will spend a lot of money guessing.
I also see people push the whole corpus into a long context and call it done. Sometimes that works. Often it degrades in ways that are hard to see from a demo, which I got into more in my piece on what happened when my anti-hallucination prompt backfired.
Fine-tuning is for shape, not facts
The clearest way I have found to explain this to a non-technical stakeholder: fine-tuning changes how the model talks, retrieval changes what it knows.
Cases where I have seen fine-tuning genuinely earn its place:
A classifier with a taxonomy of forty internal categories that no base model has ever heard of, where few-shot prompting kept collapsing similar categories together. Fine-tuning fixed it and cut the prompt from 1,800 tokens to about 200, which paid for the training run inside a month.
A summarisation job where the output had to follow a rigid internal report format every single time. Prompting got it to about 85% compliance. Fine-tuning got it to the high nineties, and the remaining failures were consistent enough to catch with a schema check.
A domain where the base model kept using the wrong word. Not wrong facts, wrong register. Fine-tuning is very good at register.
Notice that none of those are about knowledge. When people fine-tune to inject facts, the facts tend to come out fuzzy, mixed with adjacent training data, and impossible to update or audit. That’s a bad trade for anything a customer reads.
There is also a middle option people skip: a smaller model that’s fine-tuned for one narrow job often beats a large general model on that job, at a fraction of the cost. I went into when that’s worth it in my post on small models versus large ones.
The hybrid nobody finds exciting
Most of the systems I have actually shipped do both, and the split isn’t a compromise so much as a division of labour.
Retrieval handles the facts. It owns anything that lives in a database or a document and anything that has an owner who might edit it.
Fine-tuning, when it’s there at all, handles the output contract. Tone, structure, category names, the refusal behaviour when the retrieved context does not answer the question. That last one is underrated. Teaching a model to say “the documents I have don’t cover this” is a formatting behaviour, not a knowledge behaviour, and it’s very trainable.
Start with retrieval and good prompts. Measure. If your errors are wrong or missing facts, that’s a retrieval problem and no amount of tuning fixes it. If your errors are the model getting the right information and then presenting it badly, inconsistently, or in the wrong voice, that’s when tuning starts to make sense.
I don’t think this is a controversial position, but it’s one people arrive at slowly and expensively, usually after a fine-tuning run that didn’t do what they hoped.
What I would do this week
Take fifty real questions your users have asked. Not synthetic ones, real ones, from a support inbox or a chat log.
For each one, check by hand whether the correct source document appears in your retrieval results. Write down the percentage. That number, retrieval recall at your actual k, is the single most useful metric in this whole discussion, and almost nobody has it written down anywhere.
If it’s under 80%, you have a retrieval problem, and fine-tuning would have been a very expensive way to not fix it. If it is above 90% and the answers are still off, now the conversation about tuning gets interesting.
I do this kind of evaluation work as part of building these systems, and there is more about that on my portfolio. The fifty-question exercise takes an afternoon and it has changed the plan on more projects than any benchmark table I have ever read.
