Long context or RAG: when would you put whole documents in the prompt instead of retrieving chunks?

Put whole documents in the prompt when they fit comfortably, the same documents serve many questions (so caching makes them cheap), and the question needs the whole document (comparing sections, summarising, finding contradictions). Use RAG when the corpus is far larger than the context window, changes often, or different users may see different documents, or when cost and latency per question must stay low.

Why long context is attractive

• No retrieval step that can miss the answer.
• The model sees connections across the whole document.
• Simpler to build: no chunking, embeddings or index to maintain.

Anthropic's contextual retrieval write-up says that for a knowledge base under about 200,000 tokens you can skip RAG and put it all in the prompt, using prompt caching to keep it affordable.

Why it is not always enough

• Size. Many company knowledge bases run to millions or billions of tokens. No context window holds that.
• Cost and latency. Every question pays for every token in the prompt. Caching cuts the price of repeats, but an uncached 500,000-token prompt is slow and expensive.
• Attention. "Lost in the Middle" (Liu et al.) showed models use information at the start and end of a long context better than in the middle, even models built for long contexts. Newer models do better, but test with your own documents: put the answer at different positions and measure.
• Access control. If users may only see some documents, building a per-user prompt is retrieval anyway.
• Freshness. A changed document breaks the cache and needs re-sending. An index updates one chunk.

A middle way: retrieve documents, not chunks

Retrieve the few most relevant whole documents (or sections) and put them in full. This keeps context intact and leaves out the rest of the corpus. It often beats small chunks for long reports and contracts.

How to decide

The answer interviewers like: "I'd try the long-context version first because it's simpler, measure accuracy, cost and latency on real questions, and move to retrieval only where it fails one of those."