KV cache size: the formula, per token, with examples

The KV cache needs 2 × layers × KV heads × head size × bytes per number for every token, and that figure times the tokens in a conversation is its size. Llama 3.3 70B has 80 layers, 8 KV heads and a head size of 128, so at 16-bit it keeps 2 × 80 × 8 × 128 × 2 = 327,680 bytes a token: 10.7 GB for a 32,768-token conversation.

Multiply again by the conversations a server holds at once, because each has its own cache. On Llama 3.3 70B, 8 conversations of 32K tokens need 85.9 GB of cache, more than an 80 GB GPU holds before the weights are even loaded.

Where each number in the formula comes from

A model writes one TokenTokenThe small piece of text a language model reads and writes, often part of a word. Prices, limits and the context window are all counted in tokens. at a time. In every attention layer each earlier token has a key and a value, 2 lists of numbers the new token compares itself with. The KV cacheKV CacheWork a model saves from the text it has already read, so it is not redone for every new token. Prompt caching keeps that work between requests and charges less for it. keeps them so they are not worked out again for every new token. So for each token, in each layer, the cache holds:

• 2: 1 key and 1 value.
• KV heads: how many separate keys and values each layer keeps. This is num_key_value_heads in the model's config.json.
• Head size: how many numbers are in each one, head_dim in the config, or hidden_size ÷ num_attention_heads when it is missing.
• Layers: num_hidden_layers.
• Bytes per number: 2 at 16-bit, 1 at 8-bit.

The vLLM paper worked it out the same way for OPT-13B: 2 × 5,120 × 40 × 2 bytes, 800 KB a token. That model kept a key and a value for every attention headAttention HeadOne parallel attention mechanism within a multi‑head attention layer., so its 5,120 is all 40 heads of 128.

Use the KV heads, not the attention heads

Most models now share keys and values between heads. With grouped-query attention, several query heads read the same key and value, and only the shared ones are cached. Llama 3.3 70B has 64 attention heads and 8 KV heads. Put 64 into the formula and the cache comes out 8 times too large.

KV cache per token for popular models

At 16-bit, worked out from each model's config:

The models at the bottom of the table are larger than Llama 3.1 8B and keep less for each token. The formula above only covers layers that keep every token, and newer models have other kinds. Choose a model below to see its layers and the sum for each kind.

Try itWork out a KV cache, layer by layer

Model

·Granite 4.2 30B·Qwen3.8 27B·Nemotron 3.5 Lightning 30B A3B·GLM-5.3·Mistral Medium 3.5 128B·Gemma 4 31B·Llama 3.1 8B·Mistral Small 3.2 24B·Llama 3.3 70B·gpt-oss-120b·Kimi K2.6

Cache

16-bit8-bit

Tokens in each conversation

32Ktokens

prompt and answer together

4K32K256K

Conversations at once

1conversation

each has its own cache

1864

64 layers

• 64 × 2,048 × 2 B = 262 KB a tokenfull attention

Each token262 KB

1 conversation, 32K8.59 GB

1 at once8.59 GB

Layers that keep less

• Sliding windows. A window layer keeps only the last few tokens. In gpt-oss-120b, 18 of the 36 layers keep 128 tokens, and Gemma 4 31B keeps 1,024 in 50 of its 60. These add a fixed amount once the window is full, so for long conversations only the full layers count. vLLM frees the blocks that fall out of a window.
• Compressed (latent) attention. DeepSeek's multi-head latent attention stores 1 compressed vector a layer instead of a key and a value for every head. Kimi K2.6 uses it: 512 numbers plus 64 for position, 70.3 KB a token. Its 64 heads with full keys and values would need 2.5 MB a token, 36 times as much. DeepSeek reported a 93.3% smaller cache for DeepSeek-V2 than for its earlier 67B model.
• Linear attention. In Qwen3.8 27B, 48 of the 64 layers are Gated DeltaNet layers, which fold everything earlier into a state of fixed size. They add 154 MB per conversation however long it runs, and only the other 16 layers grow.
• Mamba layers. Nemotron 3.5 Lightning 30B A3B, from August 2026, has 52 layers: 23 Mamba layers, which also keep a fixed state, 23 mixture-of-experts layers, which keep nothing, and only 6 attention layers that keep every token. That is 6.14 KB a token and 49.1 MB of state, so a 128K-token conversation needs 854 MB. Granite 4.2 30B, released the same month at about the same size, keeps every token in all 64 layers: 34.4 GB.

For these models, add up each kind of layer separately. The GPU memory calculator does this for every model it lists.

Conversations at once, and what vLLM reserves

Each conversation has its own cache, so the total is per token × tokens × conversations. vLLM does not reserve the full context for each conversation up front. Its PagedAttentionPagedAttentionA way of storing each request's KV cache in small fixed-size blocks, like pages of virtual memory, so a serving engine wastes almost none of it and fits more requests in a batch. stores the cache in small blocks of tokens and hands them out as conversations grow. Its paper found that earlier systems used only 20.4% to 38.2% of the memory they set aside for the cache on real token states.

When vLLMvLLMAn open-source engine for serving language models fast, built on PagedAttention, that answers requests through the same kinds of API the big model makers offer. starts, it takes 92% of the GPU's memory, keeps what the weights and its own working memory need, and gives the rest to the cache. Then it prints a line like GPU KV cache size: 430,000 tokens, Maximum concurrency for 32,768 tokens per request: 13.12x. The second figure is how many conversations of the full length fit at once. If it is lower than you expected, the formula tells you which setting to change.

How to make the cache smaller

4 settings change it:

1. Shorten the context. --max-model-len sets the longest conversation. A model that reads 128K tokens does not need 128K reserved if your prompts are 8K.
2. Hold fewer conversations. --max-num-seqs caps the number running at once, and the rest wait their turn.
3. Store it in 8 bits. --kv-cache-dtype fp8 halves every figure in the table. vLLM recommends calibrating it, and window layers lose more from it, so check the answers you care about.
4. Choose a model that keeps less. For long documents, a model with windows, latent attention or linear layers can hold several times the context of an older model in the same memory.

QuantizingQuantizationStoring a model's numbers with less precision, such as 4 bits instead of 16, so it needs far less memory and can run on smaller hardware, at some cost in accuracy. the weights does not shrink the cache: it has its own setting, and a 4-bit model with a 16-bit cache keeps exactly the bytes in the table. Weights and cache together are worked out in how much VRAM you need to run an LLM locally, and the open models module of the free AI engineering course runs this sum on a real model with vLLM.