RAG vs Fine-Tuning vs Prompting

A team has a model that doesn't know their product. One engineer says "fine-tune it". Another says "just put the docs in the prompt". A third says "build RAG". All three have a point, and all three are answering different questions.

The choice isn't about which technique is more advanced. It is about what you're trying to change: what the model knows, or how it behaves.

Three levers, three jobs

Why fine-tuning is a poor way to add knowledge

It's tempting to "teach the model our docs". In practice fine-tuning stores facts unreliably, can't point to a source, gets stale the day a policy changes, and has no concept of who is allowed to see what. RAG handles all four naturally, because the facts stay outside the model where you can update, cite and permission them.

When fine-tuning does pay off

• Consistent behaviour. A house style or strict output format that prompting keeps drifting from.
• A narrow, high-volume task. Classification or extraction where a small tuned model matches a large prompted one.
• Cost and latency. A shorter prompt on a smaller model can cut both, as the numbers below show.

def monthly_cost(requests, in_tokens, out_tokens, price_in, price_out):
                          return requests * (in_tokens * price_in + out_tokens * price_out) / 1_000_000
                      
                      # Example numbers only (prices per million tokens): use your own measurements.
                      requests = 200_000
                      long_prompt = monthly_cost(requests, in_tokens=2_500, out_tokens=200, price_in=3.0, price_out=12.0)   # big model, long few-shot prompt
                      tuned_small = monthly_cost(requests, in_tokens=400, out_tokens=200, price_in=1.0, price_out=4.0)       # fine-tuned small model, short prompt
                      one_off = 20_000                                                                                         # data prep + training + evaluation effort
                      
                      monthly_saving = long_prompt - tuned_small
                      print(f"prompted: ${long_prompt:,.0f}/mo  tuned: ${tuned_small:,.0f}/mo  saving: ${monthly_saving:,.0f}/mo")
                      print(f"pays back in {one_off / monthly_saving:.1f} months, if quality holds on your eval set")
                      

The prices and volumes are examples only. The shape is the lesson: tuning pays back through volume. At low traffic the one-off effort dominates, and the break-even only holds if the tuned model passes your evaluation set. Most of the real cost is preparing good training data, not the training run.

They combine

Mature systems often use all three: a prompt that sets the task, RAG that supplies facts, and sometimes a tuned model that follows the format reliably. Add them one at a time and measure what each buys you.

The solution architect's view

This is a decision about cost, risk and change, so I'd run it like one.

Use a decision ladder

1. Can a clear prompt with a few examples meet the quality bar? Stop here if it can.
2. Is the gap missing or changing knowledge? Add retrieval.
3. Is the gap behaviour, format or unit cost at volume? Consider fine-tuning, with an evaluation set that proves the gain.

Count the ownership cost

A fine-tuned model is an asset you now maintain. It needs curated data, versioning, a retraining plan when the base model changes, and a regression suite. Hosted fine-tuning options and base-model lifecycles vary by provider, so check what is supported and for how long before committing.

Mind data governance

Training data can contain personal or confidential information that becomes hard to remove from a model afterwards. Knowledge you keep in a retrieval index can be deleted or restricted on demand. For regulated data, that difference often decides the question.

Keep the decision reversible

Put the model behind an interface and keep your evaluation set independent of the technique. Then you can swap a prompted large model for a tuned small one, or the reverse, without rebuilding the product.

Questions for the design review

1. Is the gap knowledge or behaviour, and what evidence says so?
2. Has a better prompt been tried and measured against the evaluation set?
3. Who owns the training data and the retraining schedule?
4. What happens to the tuned model when the base model is retired?
5. Does any of the training data need to be deletable?