RAG vs fine-tuning vs long context: how to give a model your own knowledge

A model knows what it was trained on, up to its knowledge cutoffKnowledge CutoffThe date a model's training text ends. The model knows nothing after it unless it is given new text, for example by a web search., and nothing about your products, your policies or last week's changes. There are 3 ways to close that gap. You can look up what each question needs and add it to the prompt (RAGRAGA way to answer from your own documents. A search finds the passages that match the question, and they are added to the prompt before the model writes.), put everything in the prompt every time (long context), or train the model further on your own examples (Fine-TuningFine-TuningTraining an existing model a little further on your own examples so it picks up a task, a format or a style. Often a good prompt does the job first.). They are often presented as rivals. In practice they solve different problems, and the first question is whether you need the model to know something or to behave differently.

RAG: look it up for each question

RAG keeps your knowledge outside the model. Documents are cut into chunks by a ChunkingChunkingCutting documents into smaller passages before they are embedded for retrieval, so a search returns the part that answers a question rather than a whole file. rule, each chunk is turned into an EmbeddingEmbeddingA list of numbers that stands for the meaning of a piece of text, an image or other data. Things with similar meaning get similar numbers, so a computer can find them., and the embeddings go into a vector databaseVector DatabaseA database built to store embeddings and quickly find the ones closest to a query, which is how AI apps search by meaning.. When a question arrives, it is embedded the same way, the closest chunks are found, and only those are added to the prompt. The model answers from what it was just given.

That design has 3 strengths. The knowledge can be as large as you like, since only a few chunks are read per question. It stays current: change a document, re-embed it, and the next answer uses it. And the answer can point to its source, because the passages are in the prompt. Its weak spot is the retrieval itself. If the search misses the right chunk, the model answers without it, so most of the work in a RAG system goes into chunking, search quality and testing that the right passages come back.

Long context: put it all in the prompt

The simplest approach is easy to overlook. If your knowledge base is small, send all of it with every request. Anthropic's own guidance is that under about 200,000 tokens, roughly 500 pages, you can include the whole thing and skip retrieval entirely, and many current models accept far more than that.

The objection used to be cost: paying for the same 200,000 tokens on every question. Prompt caching answers most of it. You mark the end of the fixed part of the prompt, and requests that repeat it within the cache lifetime, 5 minutes by default, read it at a small fraction of the normal input price. Put the documents first and the question last, which Anthropic reports improves answers on long inputs.

Fine-tuning: change the model

Fine-tuning trains a model further on example conversations, so it learns to answer the way they do. Hosted fine-tuning is offered by some providers, such as Google for Gemini models and Amazon Bedrock for some of the models it hosts, while OpenAI is winding its platform down. For open models, a method such as LoRALoRALow‑Rank Adaptation, a parameter‑efficient fine‑tuning method trains a small adapter beside the frozen weights, cheap enough to run on a laptop for a small model.

What fine-tuning is good at is behaviour: a reply format followed every time, a tone, a classification with fixed labels, or a small, cheap model trained to do one narrow job as well as a large one. What it is poor at is facts. The model does not store your documents; it adjusts its weights towards the patterns in your examples, and a rule it saw a few times can come back wrong, said with the same confidence as a right one. It also cannot tell you which example an answer came from, and every change to the facts means training again.

Side by side

The same question 3 ways

import anthropic
import chromadb

# The handbook, cut into chunks. A real one has thousands, embedded once and kept in a vector database.
CHUNKS = [
    "Returns: laptops and tablets can be returned within 30 days of delivery, unopened or with all accessories.",
    "Returns: software licences and gift cards cannot be returned once activated.",
    "Shipping: standard delivery takes 3 to 5 working days; express delivery arrives the next working day.",
    "Warranty: every laptop has a 2-year warranty covering faults, not accidental damage.",
]
question = "How many days do I have to return a laptop?"

# Chroma embeds the chunks and the question with its default model, and finds the closest chunks.
collection = chromadb.Client().create_collection("handbook")
collection.add(ids=[str(i) for i in range(len(CHUNKS))], documents=CHUNKS)
found = collection.query(query_texts=[question], n_results=2)["documents"][0]

# Only the retrieved chunks go into the prompt.
context = "\n".join(f"<chunk>{chunk}</chunk>" for chunk in found)
reply = anthropic.Anthropic().messages.create(
    model="claude-sonnet-5-5",
    max_tokens=300,
    system="Answer from the handbook extracts only. If they do not say, reply that you do not know.",
    messages=[{"role": "user", "content": f"<handbook>\n{context}\n</handbook>\n\n{question}"}],
)
print(found)
print(reply.content[0].text)

import anthropic

# The whole handbook. Under about 200,000 tokens, it can go in every prompt with no retrieval step.
HANDBOOK = open("handbook.md").read()
question = "How many days do I have to return a laptop?"

reply = anthropic.Anthropic().messages.create(
    model="claude-sonnet-5-5",
    max_tokens=300,
    system=[
        {"type": "text", "text": "Answer from the handbook only. If it does not say, reply that you do not know."},
        # Cache the handbook: later requests within the cache lifetime read it at a fraction of the price.
        {"type": "text", "text": f"<handbook>\n{HANDBOOK}\n</handbook>", "cache_control": {"type": "ephemeral"}},
    ],
    messages=[{"role": "user", "content": question}],
)
print(reply.content[0].text)
print("written to cache:", reply.usage.cache_creation_input_tokens, "read from cache:", reply.usage.cache_read_input_tokens)

from datasets import load_dataset
from peft import AutoPeftModelForCausalLM, LoraConfig
from transformers import AutoTokenizer
from trl import SFTConfig, SFTTrainer

# Each line of replies.jsonl is 1 conversation in the reply style the model should learn:
# {"messages": [{"role": "user", "content": "Can I return a gift card?"},
#               {"role": "assistant", "content": "No, not once it has been activated.\nPolicy: Returns, gift cards.\nAcme Support"}]}
dataset = load_dataset("json", data_files="replies.jsonl", split="train")

# LoRA trains small adapter matrices beside the frozen model, not all of its weights.
trainer = SFTTrainer(
    model="Qwen/Qwen3-0.6B",
    train_dataset=dataset,
    peft_config=LoraConfig(r=16, lora_alpha=32, target_modules="all-linear"),
    args=SFTConfig(output_dir="acme-replies", num_train_epochs=10, per_device_train_batch_size=2, learning_rate=2e-4, report_to="none"),
)
trainer.train()
trainer.save_model("acme-replies")  # only the adapter, tens of megabytes, loaded on top of the base model

# Load the base model with the adapter, as you would to serve it, and ask a question it was not trained on.
model = AutoPeftModelForCausalLM.from_pretrained("acme-replies").eval()
tokenizer = AutoTokenizer.from_pretrained("acme-replies")
question = [{"role": "user", "content": "Can I return a laptop after 10 days?"}]
inputs = tokenizer.apply_chat_template(
    question, add_generation_prompt=True, enable_thinking=False, return_tensors="pt", return_dict=True
)
output = model.generate(**inputs, max_new_tokens=60)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

The RAG file sends the model 2 chunks: the returns rule and, because it also mentions laptops, the warranty rule. The long-context file sends the whole handbook and caches it. Both give the model the rule itself, so the answer can be checked against it.

The fine-tuning file is the instructive one. After training on 12 example replies, the model answered in exactly the trained format, with the answer, a policy line and the sign-off. But it said No to a return after 10 days, inside the 30-day window. It had learned how a reply looks, not the rule behind it. More and better examples would help; the point stands that facts belong in the prompt, where they can be read, checked and changed.

Which to use

Things to watch

• Start with the prompt. Try long context or RAG with good instructions first, and measure with an eval. Fine-tune only when evals show prompting cannot reach the behaviour you need.
• Test retrieval on its own. In a RAG system, check that the right passages come back for a set of real questions before judging the answers. Most wrong answers start as missed retrievals.
• Tell the model what to do when the answer is not there. Both prompt files say to reply that it does not know. Without that, a model fills gaps with plausible guesses, a form of HallucinationHallucinationWhen a model states something false as if it were true, such as a wrong date, a made-up quote or a source that does not exist..
• Price both ways. Long context sends more tokens per question; RAG adds embedding and a database. Compare them with the LLM cost calculator, the prompt caching calculator, the embedding cost calculator and the vector database size calculator.

To compare the places a RAG system keeps its vectors, read pgvector vs Qdrant vs Chroma. To see retrieval inside a whole agent, follow one request through How AI agents work.