The question, in plain words
You want an assistant that answers customer questions from your own knowledge: your products, your policies, your help articles. Two techniques come up in every conversation about it, and they are usually presented as rivals. They are not; they solve different problems.
RAG (retrieval-augmented generation) means the assistant first searches your documents for the relevant passages, then asks the model to answer using only what it found. The model's knowledge stays general; your knowledge stays in your documents.
Fine-tuning means training the model itself on your examples, so its behaviour changes: the way it writes, the format it follows, the decisions it makes on cases like the ones it was trained on.
The shortest useful summary: RAG changes what the assistant knows. Fine-tuning changes how it behaves.
When RAG wins
RAG is the right starting point for almost every support assistant, for four reasons:
- Your knowledge changes weekly. Prices, policies and features change; with RAG you update a document and the assistant is current. A fine-tuned model knows what it knew on training day.
- Answers can show their sources. A support answer that links the policy it came from earns trust and makes mistakes easy to spot. Fine-tuned knowledge is invisible; you cannot ask it where an answer came from.
- Permissions are enforceable. RAG searches only the documents this user is allowed to see. A fine-tuned model cannot un-know something for one audience.
- It is cheaper to start and safer to change. No training runs, no model versions to manage. You tune the search, the chunking and the prompt.
The catch: RAG is only as good as its retrieval. When a RAG assistant answers badly, the cause is usually that the right passage was never found, not that the model failed. That is an engineering problem with measurable fixes, which is good news.
When fine-tuning wins
Fine-tuning earns its cost in a narrower set of situations:
- A strict format or voice. If every reply must follow your exact structure, tone and phrasing, and prompting alone keeps drifting, tuning on a few thousand good examples fixes it durably.
- Repeated specialised judgements. Classifying tickets into your 40 categories, or triaging by your rules, is behaviour, and behaviour is what tuning changes.
- A smaller, cheaper model. A common pattern is to fine-tune a small model to match a big model's quality on your one narrow task, then run the small one and cut your running cost.
The catch: fine-tuning needs hundreds to thousands of clean examples, a way to measure quality before and after, and re-training whenever your needs change. Done without that discipline it makes things worse, confidently.
What they cost, roughly
- RAG costs engineering time up front (search, chunking, prompt, testing) and slightly more per request, because retrieved passages are extra input tokens. There is no training cost and updates are free.
- Fine-tuning costs preparation time (collecting and cleaning examples), a training run each time you update, and evaluation work. Per request it can be cheaper, especially if it lets you use a smaller model.
For the arithmetic behind per-request costs, see what an AI agent costs to run.
The order we recommend
- Start with RAG and a well-written prompt. Measure against a test set of real questions with known good answers.
- Fix retrieval until the right passage is found for at least nine questions in ten. This step alone resolves most quality complaints.
- If the assistant still fails on format, tone or a repeated judgement, and you have the examples to teach it, add a narrow fine-tune for that piece.
- Re-measure on the same test set. If the numbers did not improve, take the tuning out.
Most production support assistants we see end up as RAG plus a carefully tuned prompt, with fine-tuning added only for classification or formatting once the volume justifies it.