Skip to the page
Tecyfy

Understanding Context Windows in Generative AI

Discover the importance of context windows in Generative AI for daily use and development. Learn how tokenization works, explore real-world examples, and optimize AI usage effectively.

Tecyfy · Published · Updated · 10 min read

A context windowContext WindowEverything a model can see at once while it writes an answer, counted in tokens. When the window is full, something has to be left out. is the amount of text a language model can work with at once, measured in tokensTokenThe small piece of text a language model reads and writes, often part of a word. Prices, limits and the context window are all counted in tokens.. It holds everything the model sees for one answer: its instructions, the conversation so far, any documents or code you added, and the answer it is writing. Anything outside the window, the model cannot see. This guide explains what fills a window, how tokens are counted, what happens when it is full, and how to work well within it.

What fills the window

Think of the window as the model's working memory for one request, not its knowledge. What a model learned in training is stored in its weightsWeightA trainable parameter in a neural network that scales input values and is updated during training.; what it knows about your task is only what is in the window right now. Each request fills it with:

  • the system promptSystem PromptInstructions the builder of an AI app gives the model before you type anything. It sets the role, the tone and the rules, and it outranks what users say. the app or developer set,
  • every earlier message in the conversation, yours and the model's,
  • files, pages and tool results added along the way,
  • and the answer being written, which counts too.

Each new message is sent along with the conversation so far, so every turn adds to the window. That is why a long chat "costs" more of the window with each turn, and why a model can lose track of something said much earlier.

Tokens: how the window is counted

Models do not read words. A tokenizerTokenizerThe part that cuts text into the tokens a model reads, and turns the model's tokens back into text. Each model family has its own. splits text into tokens: whole common words, pieces of rarer words, punctuation and spaces. Here is a real split, made with OpenAI's o200k tokenizer:

The experienced developer quickly debugged the problematic code in the production environment.

The | experienced | developer | quickly | debug | ged | the | problematic | code | in | the | production | environment | .

12 words became 14 tokens. Common words are one token each, even long ones; "debugged" became 2. Rare words split more: "cryptocurrency" is 2 tokens, and a made-up word like "Tecyfy" is 3.

Two things use more tokens than plain English:

  • Code. Symbols, underscores and indentation are tokens too. The 2-line Python function def calculate_average(numbers_list): return sum(numbers_list) / len(numbers_list) # Returns float is 9 words but 21 tokens.
  • Other languages and scripts. "Hello, how are you?" is 6 tokens; the same question in Hindi, "नमस्ते, आप कैसे हैं?", is 9.

As a rough guide, one token is about three quarters of an English word, so 1,000 tokens is about 750 words. Every maker counts a little differently; our token counter shows the count for the same text on each model.

How big windows are

Windows have grown fast, from a few thousand tokens in early chat models to hundreds of thousands, and up to a million in some models today (Claude's largest). A million tokens is roughly 750,000 words, several long books. Sizes change with every model release, so we keep them on a chart rather than in this article: see context window against price for the current models.

When the window is full

What happens depends on where you use the model:

  • Through an API, a request that does not fit is rejected. Anthropic's API, for example, returns an error when the input alone is longer than the window, and, on Claude 4.5 models and newer, stops the answer early if writing it would run past the limit (earlier models reject such a request instead).
  • In chat apps and coding agents, the app can manage it for you, for example by dropping the oldest turns first; through Claude's API, compactionCompactionReplacing the older part of a long conversation with a summary, so an agent can keep working when its context fills up. (in beta) summarises earlier parts of the conversation so it can continue. Details from early in a long chat can be lost in the summary.

Bigger is not always better

A larger window lets you include more, but models do not use every part of a long window equally well. A 2023 study, Lost in the Middle, found that models answered best when the relevant information was at the beginning or end of a long input, and worse when it was in the middle. Anthropic's documentation calls the wider effect context rot: as the token count grows, accuracy and recall fall. Long inputs also cost more, because you pay for every input token on every request, and take longer to answer.

So the aim is not to fill the window but to put the right things in it, a practice called context engineeringContext EngineeringChoosing what goes into a model's context at each step of a task, what to fetch only when needed, and what to clear, note down or summarise as the context grows..

Working well within the window

For everyday use

  • Start a new chat for a new task. Old, unrelated messages take space and can distract the model.
  • Give only what the task needs. Paste the relevant section of a report, not the whole report, when you can.
  • Put the important part where it is read best. With a long document, put the document first and your question at the end.
  • Restate key facts in long chats. If something from early on matters, say it again.

For developers

  • Count before you send. Use the maker's token counting tool or API to check a request fits, and leave room for the answer.
  • Retrieve, do not stuff. For large document sets, use retrievalRAGA way to answer from your own documents. A search finds the passages that match the question, and they are added to the prompt before the model writes.: split documents into chunksChunkingCutting documents into smaller passages before they are embedded for retrieval, so a search returns the part that answers a question rather than a whole file., find the few that match the question, and send only those.
  • Trim what piles up. In agents, old tool results pile up in the window. Summarise or drop them once they have been used.
  • Measure on long inputs. If your app relies on long documents, test answers with the key fact at the start, middle and end.

Tecyfy Takeaway

The context window is everything a model can see for one answer, counted in tokens, and its answer counts too. Code and non-English text use more tokens than you might expect. A bigger window helps, but models read long inputs unevenly and every token costs money and time, so put in what the task needs, place the important parts first or last, and start fresh when the task changes.

Questions

What is a context window in AI?
A context window is the amount of text a language model can work with at once, measured in tokens. It holds the system prompt, the conversation so far, any files or tool results, and the answer being written. Anything outside it, the model cannot see.
How many words is 1,000 tokens?
About 750 English words: one token is roughly three quarters of a word. Code and other languages use more tokens for the same length, and every maker counts a little differently.
What happens when the context window is full?
In Claude's API, for example, a request whose input alone is longer than the window is rejected, and on Claude 4.5 models and newer an answer stops early if it would run past the limit. Chat apps can manage it for you, for example by dropping the oldest turns first, so early details can be lost.
Is a bigger context window always better?
No. Models read long inputs unevenly: the Lost in the Middle study found answers were best when the relevant part was at the start or end. Every input token also costs money and time, so put in what the task needs, not everything.
How do I make the most of a context window?
Start a new chat for a new task, give only what the task needs, put a long document first and the question at the end, and restate key facts in long chats. Developers should count tokens before sending, retrieve relevant chunks instead of whole documents and trim old tool results.