It happens in week three of every AI project. The demo went fine, the pilot users are mildly impressed, and then someone, usually a smart someone, leans back and says it: shouldn't we fine-tune? The room nods. Nobody in the room, including the person who said it, can explain what fine-tuning would actually fix.
Welcome to the most reliably botched decision in applied AI: fine-tuning vs. RAG, when to use each, and when to use neither. Teams choose based on vibes, vendor blog posts, and whichever technique appeared in the last conference talk they half-watched. Then they spend six weeks discovering the model still does not know their return policy.
So let us settle it with the boring, correct order of operations, plus the handful of cases where each technique genuinely wins.
The short version: fix your prompts first, add retrieval when the model needs knowledge that changes, and fine-tune only when you need different behavior, meaning a format, a tone, or a classification pattern. RAG is for what the model knows; fine-tuning is for how the model acts. Most teams need the first two and never reach the third, which is fine, because the third is where the invoices live.
Fine-Tuning vs. RAG: When to Use Each — the Question in Every AI Meeting
Everyone nods because fine-tuning sounds like real AI. Training a model, how serious. RAG, by contrast, sounds like plumbing, which it is, and which is also why it works. Fine-tuning still carries a mystique left over from the era when base models were weaker and prompt engineering felt like wishing on a star.
Notice what the question skips. Asking fine-tune or RAG before asking what is actually failing is like arguing about engine swaps before checking the fuel gauge. The failure mode comes first. The technique is just the wrench that fits it.
The Short Answer: Prompts, Then RAG, Then Fine-Tuning
Each technique fixes a different layer of the problem. LLM prompt engineering, meaning clear instructions, a few worked examples, and a strict output schema, solves more than anyone expects, and it costs an afternoon rather than a sprint. Retrieval gives the model knowledge: current, citable, and updatable. Fine-tuning changes the model's habits: how it formats, how it classifies, how it sounds.
Knowledge versus behavior versus format. Keep those three buckets separate and the fine-tuning vs. RAG debate mostly resolves itself.
What RAG Is Good At
RAG wins whenever the answer lives in knowledge that changes: docs, tickets, contracts, prices, policies. Say you have a 40-page policy PDF and 3,000 historical support tickets. A basic retrieval augmented generation setup chunks the documents, embeds them, stores the vectors, and at question time fetches the most relevant few chunks into the prompt. When the policy changes, you update the document and re-index. No retraining, no waiting, no invoice.
The freshness is the entire point. If you are evaluating which vector database to use for retrieval, you are already further along than the team fine-tuning last quarter's prices into a model's weights.
Where RAG Falls Over
Retrieval fails in three predictable ways. Your source data is a swamp: three versions of the refund policy, all labeled current, all contradicting each other. Your chunking is crude: sentences sliced mid-clause, tables shredded into confetti. Or retrieval confidently fetches the wrong thing, so the model answers a question about international shipping using the 2019 holiday schedule, delivered with total, serene confidence.
None of these are model problems. They are data-hygiene problems wearing a model costume.
A typical mid-size firm discovers this the hard way. The retrieval demo works beautifully on the twelve documents someone hand-picked for the demo. Then it meets the real SharePoint, where the actual policy lives in a file called final_FINAL_v9.pdf next to six earlier finals, and the answers get weird in ways no benchmark warned you about. Budget a week for source-data cleanup before you budget anything for embeddings.
What Fine-Tuning Is Good At
Fine-tuning wins on behavior, format, and tone. Classification at scale is the classic case: routing 50,000 tickets a month into twelve categories, cheaper and more consistent than prompting a frontier model for each one. House style is another: a model that writes release notes the way your company writes release notes. Structured extraction against a fixed schema is a third, though before you fine-tune for format, try forcing structured outputs from the LLM with a schema, which often gets you there for free.
So when to fine-tune an LLM, practically: when you can write down a few hundred to a few thousand input-output examples of the exact behavior you want, and that behavior is stable. The compute is no longer ruinous; a modest fine-tune on a hosted model runs less than a team lunch. The discipline is what costs. You need example collection, a held-out test set, and the willingness to throw away a training run that made the model worse, which happens more often than the tutorial videos suggest.
Fine-Tuning Is Not a Memory
Do not fine-tune facts that change quarterly. The model will recite last year's prices with the calm confidence of a golden retriever, and updating them means retraining, re-evaluating, and redeploying. Facts belong in retrieval. Habits belong in fine-tuning. Teams that mix this up end up retraining a model every time marketing touches the pricing page, which is to say, constantly.
The Decision Table Nobody Gives You
Since actual tables are apparently too helpful for most blog posts, here is the fine-tuning vs. RAG, when-to-use-what logic in plain sentences:
- Knowledge changes more than once a quarter: RAG. No debate.
- Facts are fine but the output shape or tone is wrong: better prompts with examples first, fine-tune if that plateaus.
- Both problems at once: RAG for the knowledge, a light fine-tune for the format. This combo is more common than the purists admit.
- Neither retrieval nor examples fix it: your prompts are the problem. Go back to week one.
- You cannot name the failure mode: change nothing. Build an eval set first.
A Sane Rollout Order for a Small Team
Weeks one and two: prompts, plus an eval set of 30 to 50 test cases drawn from real user questions. Every change you make from now on runs against that set, so improvements are measured rather than felt. Weeks three and four: retrieval over your real documents, and measure which failures remain. Week five and beyond: fine-tune only if you can name the failure mode in one sentence that starts with the model keeps and ends with a behavior word, not a knowledge word.
This discipline is the same muscle that keeps bigger systems honest; running AI agents in production is evals and monitoring all the way down. And the eval set outlives every model swap, which matters, because there will be model swaps.
The room will ask about fine-tuning again in week nine. This time you will have an answer, an eval set, and the calm of a person who checked the fuel gauge first.