The full answer
Retrieval-augmented generation (RAG) looks up the passages relevant to a question in your own documents and puts them in the prompt, so the model answers from them. Use it when the answer depends on facts the model was not trained on, or that change: your policies, products, tickets, contracts. Fine-tuning changes how a model behaves (format, tone, a narrow task), not what it knows. And if all your documents fit comfortably in the prompt, you may need no retrieval at all.
How RAG works
- Index (ahead of time): split documents into chunks, turn each into an embedding, and store them in a search index, often a vector database, usually with a keyword index alongside.
- Retrieve (per question): embed the question, find the most similar chunks, and often rerank them.
- Generate: put the best chunks in the prompt with an instruction to answer only from them and cite them.
The name comes from Lewis et al. (2020), who paired a retriever with a generator so answers could draw on knowledge outside the model's weights.
Choosing between the three
| Situation | Best fit |
|---|---|
| Facts are private, large or change often | RAG |
| The whole knowledge base is small | Put it all in the prompt, with prompt caching |
| The model must follow a format, style or narrow task every time | Better prompts and examples first, fine-tuning if that fails |
| You need a small, cheap model to do one job well | Fine-tuning (where the provider offers it) |
The "just put it in the prompt" option
Anthropic's contextual retrieval write-up says a knowledge base under about 200,000 tokens can simply go in the prompt, and prompt caching makes repeating it cheap. That avoids every retrieval failure. It stops working when the documents outgrow the context window, when cost per question matters, or when different users may only see different documents.
Why not fine-tune facts in?
Facts trained into weights cannot be cited, are hard to update (every change means retraining), and can still be misremembered. RAG answers can quote the exact passage and change the moment a document changes. Fine-tuning is also less available than it was. OpenAI's model optimisation guide says it is winding down its fine-tuning platform for new users, so check what your provider offers before planning on it.
What an interviewer wants to hear
That you pick the simplest thing that meets the need, and that you know RAG has its own failure points: bad chunking, missed retrieval, and the model ignoring the context. Those are measured separately, which is the subject of the follow-up questions.
Last updated 16 October 2026
Sources