# What is a KV cache (and prompt caching)? Work a model saves from the text it has already read, so it is not redone for every new token. Prompt caching keeps that work between requests and charges less for it. - Page: https://tecyfy.com/concepts/kv-cache - Subject: How AI works (https://tecyfy.com/concepts) - Level: advanced - Also called: Key-Value Cache, Prompt Caching - The lesson: Same price per token, smaller bill: what a job costs (https://tecyfy.com/learn/what-ai-costs/same-price-smaller-bill) - Last checked: 2026-10-14 There are 2 caches with similar names here. One lives inside a model while it writes a single answer. The other lives at the provider, between your requests, and shows up on your bill. The second is built on the first. ## Inside one answer: the KV cache A model writes one [Token](https://tecyfy.com/concepts/token) at a time, and each new token is worked out from every token before it. To write the 1,000th token it needs information from the 999 before; for the 1,001st, the same 999 plus one more. Without help, it would redo almost all of that work for every single token. The KV cache is the help. As the model processes each token, its [attention](https://tecyfy.com/concepts#self-attention) layers produce 2 small bundles of numbers for it, called a key and a value. The cache keeps them. For the next token the model reuses what is stored and only computes the new token's share. In Hugging Face's words, the cost of each step stops growing with the square of the length and grows only in step with it. The price is memory. The cache grows with every token in the conversation, in every layer of the model. That is one reason long contexts are expensive to serve, and why people who run models on their own hardware watch the cache as closely as the model's size. ## Between requests: prompt caching A chat or an agent sends the whole conversation again on every turn. Anthropic's example is a task that grows from 20,000 to 120,000 tokens over 40 turns and processes about 2.8 million input tokens in total, because the start is re-sent every time. Prompt caching lets the provider keep the processed start of your prompt for a while and reuse it. With Claude, reading from the cache costs a tenth of the normal input price, or a twentieth on Opus 5.5 and Sonnet 5.5. Writing it the first time costs a little more than normal. The cache lasts 5 minutes by default, and each use resets the clock. Long documents also start answering sooner. ## What breaks it A cache only matches an identical beginning. The prompt is read in a fixed order (tools, then the [system prompt](https://tecyfy.com/concepts/system-prompt), then messages), and a change anywhere invalidates everything after it. Put a timestamp at the top of a system prompt and nothing after it can ever be reused. ## What to do with this If you build with a model: keep the fixed parts first and unchanged, and put whatever varies at the end. If you only use AI apps: this is part of why starting a fresh chat for a new subject is cheaper and faster than dragging a long one along. ## Related - [Inference](https://tecyfy.com/concepts/inference): Using a trained model to produce an answer. Training builds the model once; inference is every time someone asks it something, and it is what you pay for per token. - [Transformer](https://tecyfy.com/concepts/transformer): The design behind almost every modern language model. It uses attention to let each word in a passage look at every other word, all at once. - [Token](https://tecyfy.com/concepts/token): The small piece of text a language model reads and writes, often part of a word. Prices, limits and the context window are all counted in tokens. - [Context Window](https://tecyfy.com/concepts/context-window): Everything a model can see at once while it writes an answer, counted in tokens. When the window is full, something has to be left out. - [PagedAttention](https://tecyfy.com/concepts#pagedattention): Attention algorithm that manages KV cache in GPU pages. - [Self‑Attention](https://tecyfy.com/concepts#self-attention): A mechanism that lets a model weigh the importance of different positions in a single sequence ## Sources - [How caching works](https://huggingface.co/docs/transformers/main/en/cache_explanation), Hugging Face, Transformers documentation, 2026-10-14. Generation predicts one token at a time and each prediction depends on all previous context. A key-value cache stores the key and value pairs from the attention layers for tokens already processed and reuses them for later tokens instead of recomputing them. Without caching, attention cost per step is quadratic with sequence length; with caching it is linear, while memory grows linearly. Caching is done in each layer, and should only be used for inference. - [Prompt caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching), Anthropic, Claude API documentation, 2026-10-14. Prompt caching covers the prompt in the order tools, system, then messages, up to a marked point. The default cache lifetime is 5 minutes, refreshed for free each time it is used, and a 1-hour lifetime costs more. A 5-minute cache write costs 1.25 times the base input price and a cache read 0.1 times, or 0.05 times on Claude Opus 5.5 and Sonnet 5.5. Shorter prompts cannot be cached, with a minimum of 512 tokens on the newest models. It generally improves time to first token for long documents and does not change the response. The cache follows the order tools, system, messages, and a change at a level invalidates that level and all later ones. A block that changes, such as a timestamp, means the prefix never matches. - [What a task costs on Opus 5.5](https://claude.com/blog/what-a-task-costs-on-opus-5-5), Anthropic, 2026-09-25. A task that grows from 20,000 to 120,000 tokens over 40 turns processes about 2.8 million input tokens, though the conversation never grows past 120,000. Each turn sends the conversation so far.