What are "context windows" or "context length" in language models, and how do they impact prompt design?

The context window, or context length, is the maximum number of tokens a model can take into account in one request: the system prompt, the conversation so far, any documents or tool results, and the answer it writes. Anything beyond it can't be seen, and even inside it, more text doesn't automatically mean better answers.

Tokens, not words Models read text as tokens, pieces of words. In English a token is roughly three quarters of a word on average, but code, numbers and many other languages use more tokens per word. Limits, prices and speed are all counted in tokens.

Input and output share the window If a model's window is large and the prompt nearly fills it, there is little room left for the answer. APIs also set a separate maximum on output tokens.

What happens at the limit The API rejects a request that is too long, or a chat application drops or summarises older messages. The model has no memory of anything outside the window: a long conversation can "forget" early instructions once they are cut.

Bigger isn't automatically better Windows have grown from a few thousand tokens to hundreds of thousands or more. But accuracy and recall decline as the context fills, which Anthropic calls context rot, and research found models use information at the start and end of a long context better than information in the middle. Long prompts also cost more and take longer.

Impact on prompt design

1. Budget the window: decide how much goes to instructions, examples, retrieved content, history and the answer.
2. Retrieve, don't dump: with retrieval-augmented generation, only the relevant passages of a large knowledge base go into each request.
3. Order matters: put long documents first and the question and instructions after them, and label each document.
4. Keep instructions in the system prompt, so they stay in place however long the conversation gets.
5. Manage history: summarise older turns, remove tool output no longer needed, and for long-running agents compact the context and keep notes outside it.
6. Split large jobs: process documents in sections and combine the results.
7. Cache stable prefixes: prompt caching reduces the cost of repeating a long system prompt or document across requests.
8. Test with realistic lengths, checking that details from the middle of long inputs are used.

Why it matters in practice The context window decides whether a task can be done in one call or needs retrieval, chunking or several steps. Designing for it is less about squeezing everything in and more about choosing what the model needs to see for this request.