Prompting, RAG, or fine-tuning: pick in that order

The fine-tuning vs RAG question has a short answer: RAG is for knowledge, fine-tuning is for form, and before either comes prompting, which solves more than most teams expect. Treat the three as an escalation ladder and climb only when the rung below has demonstrably failed, because each step up costs roughly ten times more to build and maintain than the one before.

The ladder gets climbed in the wrong order constantly, and usually for a bad reason: fine-tuning sounds serious. "A model trained on our data" sounds like an asset to a board and like a project to a vendor. Meanwhile the actual problem, the model does not know your return policy, is solved with retrieval in a week.

Prompting solves more than you think

A prompt, written properly, is a specification: instructions, definitions, worked examples, the output format, the rules for edge cases. It runs to pages, and with current context windows it can include entire policy documents. Most of what gets diagnosed as "the model cannot do this" is really "we wrote two sentences and hoped".

The economics are the argument. Iterating on a prompt costs days, needs no infrastructure, and a change deploys in minutes. The limits are equally clear: prompting cannot give the model knowledge that changes daily, exceeds the context window, or must differ depending on who is asking. When you hit those walls, and only then, take the next rung.

RAG, when the model needs your knowledge

RAG retrieves the documents relevant to each question and hands them to the model along with the question, a mechanism worth understanding in business terms. It fixes exactly what prompting cannot: the knowledge stays current because updating a document updates the system, answers can cite their sources, and retrieval can respect permissions so people only get answers from documents they are allowed to open.

The cost is a pipeline you now operate: ingestion, indexing, retrieval quality. It fails in undramatic ways, mostly by retrieving the wrong things, and the fixes are well mapped. Budget for tuning after launch, not only for the build. Even so, RAG remains an order of magnitude lighter than training anything.

Fine-tuning vs RAG: form, not facts

Fine-tuning adjusts a model’s weights using your examples. What it is genuinely good at: consistent tone and format, a house style, narrow classification at high volume, and making a small cheap model perform a specific task that would otherwise need a large expensive one.

What it is bad at is the thing executives usually want it for: knowledge. Facts baked into weights go stale the day the source document changes, cannot be cited, cannot vary with the asker’s permissions, and cannot be deleted per user, which in Europe is a GDPR problem rather than a detail. Refreshing them means retraining. If the sentence "the model should know our" ends with something that changes monthly, fine-tuning is the wrong tool by construction.

The maintenance bill of each rung

Prompting costs almost nothing to carry: version the prompts like code and re-test them against an eval set when you change models.

RAG is a living pipeline: sync jobs, an index, periodic retrieval reviews. Plan for a slice of an engineer, ongoing.

Fine-tuning carries the steepest bill and the least visible one upfront: thousands of curated training examples that must be kept current, training and hosting costs, a heavier evaluation burden, and a treadmill nobody mentions in the pitch. Every new base model generation may beat your fine-tune out of the box, so the work gets re-justified, and often redone, each time the frontier moves. We have seen carefully built fine-tunes quietly abandoned one model generation later.

When to break the order

There are honest reasons to reach for fine-tuning: a narrow task at millions of executions where a fine-tuned small model cuts unit cost and latency, deployment on constrained or on-premise hardware, strict latency budgets, or products where voice and form are the product itself. Even then the sequence holds: prove the task with prompting and RAG first, then fine-tune as a measured optimization with the cost case written down rather than assumed. And on any rung, the choice of model matters less than teams fear, provided you keep the ability to change it.

Our default architecture is prompting plus RAG, revisited as models improve; fine-tuning appears in a minority of builds and almost never in the first month. That is the order we follow in our custom AI engagements: the cheapest thing that works, proven on your data, before anything that merely sounds impressive. If a proposal leads with fine-tuning, ask which rungs it skipped, and listen closely to the answer.