How does context length affect prompt design, and what strategies do you use to manage long prompts?

Context length limits how much text (instructions, conversation history, documents and the answer itself) a model can handle in one request, and even within that limit, quality tends to drop as the context grows. So prompt design for long inputs is about choosing what goes in, placing it well and splitting work that doesn't need to be done in one pass.

How context length affects design

• Hard limit: the context window is measured in tokens and covers input and output together. If a document doesn't fit, it must be shortened, split or retrieved in parts.
• Quality, not just capacity: as the number of tokens grows, models recall details less accurately, which Anthropic calls context rot. Research on long contexts also found models use information at the beginning and end better than in the middle.
• Cost and latency: every token is paid for and processed on every request, so a long prompt repeated across many calls is expensive and slow.

Strategies

1. Include only what changes the answer Cut boilerplate, duplicate documents and instructions the model follows anyway.

2. Retrieve instead of including everything Split documents into chunks, store their embeddings, and add only the most relevant chunks for each question (retrieval-augmented generation).

3. Place content deliberately Put long documents first and the question and instructions after them; this improves answers in long-context tests. Wrap each document in tags with its title and source.

4. Ask for quotes first For long documents, ask the model to extract the relevant quotes, then answer from those. It focuses attention and makes answers checkable.

5. Split the work

• Map-reduce: summarise or analyse each section separately, then combine the results.
• Chains: break a long task into steps, passing only the needed output to the next.

6. Manage conversation history Summarise older turns, drop irrelevant tool results, and keep important facts in a short running note. For agents working over many steps, compaction (summarising the context when it nears the limit) and saving notes outside the context window keep long tasks on track.

7. Reduce repeated cost Prompt caching reuses a long, unchanging prefix (system prompt, reference documents) across requests at lower cost and latency. Put the stable part first and the changing part last.

8. Test at realistic lengths A prompt that works with a two-page document may fail with a two-hundred-page one. Include long inputs in the test set and check whether details from the middle are used.

Larger context windows are useful, but more context isn't automatically better: curated, well-ordered context almost always beats everything pasted in.