What is a KV cache (and prompt caching)?
Work a model saves from the text it has already read, so it is not redone for every new token. Prompt caching keeps that work between requests and charges less for it.
advanced · Also called Key-Value Cache, Prompt Caching
There are 2 caches with similar names here. One lives inside a model while it writes a single answer. The other lives at the provider, between your requests, and shows up on your bill. The second is built on the first.
Inside one answer: the KV cache
A model writes one TokenTokenThe small piece of text a language model reads and writes, often part of a word. Prices, limits and the context window are all counted in tokens. at a time, and each new token is worked out from every token before it. To write the 1,000th token it needs information from the 999 before; for the 1,001st, the same 999 plus one more. Without help, it would redo almost all of that work for every single token.
The KV cache is the help. As the model processes each token, its attentionSelf‑AttentionA mechanism that lets a model weigh the importance of different positions in a single sequence layers produce 2 small bundles of numbers for it, called a key and a value. The cache keeps them. For the next token the model reuses what is stored and only computes the new token's share. In Hugging Face's words, the cost of each step stops growing with the square of the length and grows only in step with it.
The price is memory. The cache grows with every token in the conversation, in every layer of the model. That is one reason long contexts are expensive to serve, and why people who run models on their own hardware watch the cache as closely as the model's size.
Between requests: prompt caching
A chat or an agent sends the whole conversation again on every turn. Anthropic's example is a task that grows from 20,000 to 120,000 tokens over 40 turns and processes about 2.8 million input tokens in total, because the start is re-sent every time.
Prompt caching lets the provider keep the processed start of your prompt for a while and reuse it. With Claude, reading from the cache costs a tenth of the normal input price, or a twentieth on Opus 5.5 and Sonnet 5.5. Writing it the first time costs a little more than normal. The cache lasts 5 minutes by default, and each use resets the clock. Long documents also start answering sooner.
What breaks it
A cache only matches an identical beginning. The prompt is read in a fixed order (tools, then the system promptSystem PromptInstructions the builder of an AI app gives the model before you type anything. It sets the role, the tone and the rules, and it outranks what users say., then messages), and a change anywhere invalidates everything after it. Put a timestamp at the top of a system prompt and nothing after it can ever be reused.
What to do with this
If you build with a model: keep the fixed parts first and unchanged, and put whatever varies at the end. If you only use AI apps: this is part of why starting a fresh chat for a new subject is cheaper and faster than dragging a long one along.
Sources
Where each fact comes from
This page was last checked against these sources on 14 October 2026.
- 1How caching works
Hugging Face, Transformers documentation · 14 October 2026
Generation predicts one token at a time and each prediction depends on all previous context. A key-value cache stores the key and value pairs from the attention layers for tokens already processed and reuses them for later tokens instead of recomputing them. Without caching, attention cost per step is quadratic with sequence length; with caching it is linear, while memory grows linearly. Caching is done in each layer, and should only be used for inference.
- 2Prompt caching
Anthropic, Claude API documentation · 14 October 2026
Prompt caching covers the prompt in the order tools, system, then messages, up to a marked point. The default cache lifetime is 5 minutes, refreshed for free each time it is used, and a 1-hour lifetime costs more. A 5-minute cache write costs 1.25 times the base input price and a cache read 0.1 times, or 0.05 times on Claude Opus 5.5 and Sonnet 5.5. Shorter prompts cannot be cached, with a minimum of 512 tokens on the newest models. It generally improves time to first token for long documents and does not change the response. The cache follows the order tools, system, messages, and a change at a level invalidates that level and all later ones. A block that changes, such as a timestamp, means the prefix never matches.
- 3What a task costs on Opus 5.5
Anthropic · 25 September 2026
A task that grows from 20,000 to 120,000 tokens over 40 turns processes about 2.8 million input tokens, though the conversation never grows past 120,000. Each turn sends the conversation so far.