RAG vs fine-tuning: which do you actually need

Most teams asking this question need neither.If your LLM feature returns wrong, vague, or stale answers, the defect is almost always in retrieval, chunking, or the fact that you have no evaluation set. Fine-tuning on top of a broken retriever doesn’t remove the wrongness. It makes the wrongness fluent, which is worse, because fluent errors survive review.

The rule that holds most of the time: RAG for knowledge, fine-tuning for behaviour, prompting for everything you haven’t seriously tried yet. The cost gap between the two isn’t the training run. It’s the maintenance, which recurs every time your source data changes.

Diagnose the failure before you pick the fix

Four failure classes cover nearly everything. Run this before you commit a quarter to either approach. It takes an afternoon.

Missing knowledge

Paste the correct source paragraph into the prompt and re-ask. If the answer becomes correct, the model was never the problem. This is a retrieval failure, the most common one by a wide margin, and fine-tuning is the expensive wrong answer.

Wrong tone, format, or structure

The facts are right and the shape is wrong. It won’t stop apologising. It ignores your JSON schema one time in twenty. If a stricter system prompt with three worked examples fixes that, ship it. If schema violations survive real prompt work at scale, this is where fine-tuning genuinely earns its cost.

Retrieval that looks fine and isn’t

The cited chunks are topically related but don’t contain the answer. For fifty real questions, check by hand whether the correct passage appears anywhere in the top ten retrieved chunks. If recall at ten is under roughly 0.85, stop everything else. Nothing downstream rescues a retriever that never surfaces the answer.

The culprits are dull and fixable: fixed-size chunking that splits tables mid-row, embedding-only search that can’t match identifiers like part numbers or clause references, and no reranking pass. If your corpus is PDFs and scans, the damage usually starts at parsing, which is its own subject. See how to automate document processing without a data team.

No evaluation

You’re arguing about whether last week’s prompt change helped. If you can’t answer that with a number, you have a measurement problem rather than a model problem, and every decision after it is a guess.

What RAG actually costs, input by input

Four cost lines, and only one of them is the model.

Ingestion is a one-off per document version. A 40-page PDF is roughly 20,000 to 30,000 tokens of text, so embedding it costs a fraction of a cent at prevailing rates. Parsing and chunking it well is where the hours go. Our own estimate: budget about a day of engineering per genuinely awkward source format, not per document.

Storageis nearly free until it isn’t. A million chunks at 1,024 dimensions in float32 is about 4 GB of vectors, which pgvector on a Postgres you already run handles. Past roughly ten million chunks you are buying a dedicated store and an operator for it.

Query timeis the recurring line, and it multiplies against every call. Six chunks of 500 tokens each adds 3,000 input tokens to every request. At a million calls a month that’s three billion input tokens of pure retrieval overhead. That number, not the training run, is what a fine-tune competes against.

Refresh is where RAG wins outright. Updating knowledge means re-embedding the documents that changed. Minutes, on a cron, with no redeploy and no suite to re-run.

What a fine-tune actually costs

Also four lines, and the training run is the smallest of them.

Data curationdominates. Our own estimate for LoRA on a task-specific behaviour is 200 to 500 curated examples as a starting point. That’s a working number from our reading of the parameter-efficient fine-tuning literature, not a benchmark we ran. Writing and reviewing one good example takes five to fifteen minutes, once you count the disagreements about what the right answer even is. Three hundred examples is 25 to 75 hours of skilled attention. That’s the real invoice.

The training run is cheap. A LoRA adapter on a 7B to 8B open model typically fits in one to four hours on a single A100 or H100, which at prevailing rental rates is tens of dollars, not thousands. Hosted fine-tuning APIs price per training token and land in a similar range at this size. Check live rates before quoting anyone.

Serving is the line people forget. A hosted fine-tune usually carries a per-token premium. Self-hosting an adapter means you own a GPU that must stay warm, or a cold start your users will feel.

Refresh is the killer. Every time the underlying knowledge changes you retrain, re-evaluate, and redeploy. Our own estimate puts a knowledge refresh at roughly ten to a hundred times the cost of the RAG equivalent. The width of that range is deliberate: it depends entirely on how much of your curated dataset each change invalidates.

Break-even, and why a single month figure is dishonest

Break-even is where recurring RAG token overhead exceeds the amortised cost of a fine-tune. Write it out with every input exposed, then put your own numbers in:

RAG per month  = calls x retrieved_tokens x input_rate + infra

Tune per month = (curation_hours x loaded_rate + gpu_cost)
                 / months_until_retrain
                 + calls x serving_premium

An illustration, not a case study: at 500,000 calls a month with 3,000 retrieved tokens each, RAG carries 1.5 billion input tokens a month of overhead. Against a fine-tune costing forty hours of curation plus a trivial GPU bill, amortised over six months, the fine-tune looks cheaper on paper.

Now change one input. If your documents change monthly rather than every six months, the amortisation window collapses and the fine-tune is roughly six times more expensive. If prompt caching removes most of the retrieval overhead, RAG wins immediately. If your call volume is 50,000 rather than 500,000, RAG wins by an order of magnitude.

This is why we distrust any article naming a break-even month. Break-even is dominated by two variables, refresh frequency and call volume, and both are yours. Sensitivity-test them before believing any number, including the one above.

RAG vs fine-tuning latency

A fine-tuned model is faster per request. The gap is real and usually smaller than people expect.

Added retrieval latency runs roughly 200 to 800 milliseconds per RAG query, on our own estimate. Where we think that budget goes: embedding the query is typically 20 to 60 ms, vector search is often under 50 ms on a warm index, and the rest is network hops, a reranker if you use one, and prefill on the extra 3,000 injected tokens. That last item is the one you can actually shrink.

The mitigations, in the order we’d try them. Cut the retrieved chunk count and add a reranker, so six good chunks beat twenty mediocre ones. Run retrieval concurrently with other pre-call work. Cache the system preamble. Then stream the response, so time to first token rather than total latency is what the user feels. That’s a front-end decision as much as a model one, and part of why we treat streaming states as product interface work.

If you have a hard sub-second budget and are already down to six reranked chunks, that’s a legitimate reason to look at a tuned model. It’s one of the few.

Build the evaluation harness first

This is the section that changes outcomes and the part almost everyone skips. Without a harness you can’t tell whether fine-tuning helped, or, more expensively, whether it hurt.

The golden set

Between 100 and 300 real questions, taken from user logs or from the people who answer them by hand today. Not questions you invented, because invented questions tend to be the ones your system already handles. Each row carries the question, the passage containing the answer, an acceptable answer, and a label for its part of the corpus.

Include the awkward cases deliberately. Questions with no answer in the corpus. Questions whose answer changed last quarter. Questions needing two documents combined. Roughly a fifth of the set should be cases where the correct response is a refusal, because a system that never says it doesn’t know will eventually say something expensive.

Measure retrieval separately from generation

This is the discipline most harnesses lack, and it’s exactly why teams misdiagnose. Score the retriever on its own: recall at k, meaning whether the gold passage appears at all, and mean reciprocal rank, meaning how high it sits. Then score the generator given perfect retrieval, by feeding it the gold passage directly.

Two numbers, two verdicts. If retrieval recall is 0.6, no amount of model work matters. If retrieval is 0.95 and answers are still wrong with the gold passage in hand, you have a genuine model or prompt problem, and only now is fine-tuning rational to discuss. Teams reporting one blended accuracy score can’t make this call at all.

Grading, and its limits

Using a model as judge is workable for scoring faithfulness and relevance at volume, and it’s what we use. It’s also biased toward verbose answers and agrees with itself far more readily than it agrees with you. So calibrate it: have a person grade fifty rows, measure agreement, and adjust the rubric until the judge matches on four out of five. Re-check whenever you change judge models. Anyone telling you a grader needs no human calibration is selling the grader.

Wire it into CI

A harness that runs when someone remembers is worthless. Every prompt change, model version bump, and chunking parameter change should run the suite and print a diff per metric. Ragas and promptfoo do this off the shelf and DeepEval covers similar ground, so it’s a day of work rather than a platform decision. Vendors ship new model versions on their own schedule, and a silent regression on a Tuesday is what this catches.

When we would fine-tune

Three cases, and they’re narrower than the market implies.

Rigid output format at high volume, where a one percent schema violation rate is operationally expensive and carrying few-shot examples in every prompt costs more in input tokens than a tuned model costs in total.

A specialist idiom the base model lacks. An internal taxonomy with fifty overlapping categories. A domain shorthand. A house voice you can demonstrate but cannot describe.

Distillation for cost or latency. A frontier model already does a narrow, stable, high-volume task well and you want a small open model doing it far cheaper. This is the most defensible fine-tuning case there is, because you already have a system generating the training data and a system to benchmark against.

What we wouldn’t do is fine-tune to teach a model facts. Facts change. Weights don’t, short of another training run. Making that call before anyone writes code is most of what our AI architecture and readiness consulting is for; the build that follows is ordinary retrieval and workflow automation engineering.

When you should not hire us, or anybody

If your corpus is under a few thousand documents, your users are internal, and your volume is a few thousand queries a month, buy something instead. The document-chat features inside the major assistant products, and the retrieval built into the model APIs themselves, cover that for a fraction of a custom build, and somebody else patches them and upgrades the model underneath.

Spend a week trying to hit your quality bar with an off-the-shelf tool and a carefully written prompt first. If it works, you’ve saved a quarter. If it doesn’t, you have a concrete list of what it couldn’t do, which is a better brief than you’d have written otherwise.

Custom retrieval starts earning its keep when you need hybrid keyword and vector search over identifiers, permissions that follow your existing access rules, paragraph-level citations for audit, or ingestion from systems no connector supports. Those are engineering problems. Everything short of them is a subscription.

The hybrid case, honestly

Hybrid means a tuned small model reading retrieved context: the model learns the format and the idiom, retrieval supplies the facts. That’s a real end state, and for some systems the right one. It’s also two systems to evaluate and two ways to regress. Build it second. Teams that start hybrid can’t tell which half is failing, which puts them back at the measurement problem they began with.

The compressed decision

Fix retrieval and evaluation first. Ship RAG for knowledge, because knowledge changes and re-embedding a document is cheaper than retraining a model. Fine-tune for behaviour that is stable, narrow, and high in volume, and only once a harness proves the generator rather than the retriever is what fails. Then measure again, because the only number worth trusting is the one from your own corpus.