# What is PagedAttention, and why does it let vLLM serve more requests on the same GPU?

A way of storing each request's KV cache in small fixed-size blocks, like pages of virtual memory, so a serving engine wastes almost none of it and fits more requests in a batch.

- Page: https://tecyfy.com/concepts/pagedattention
- Subject: How AI works (https://tecyfy.com/concepts)
- Level: advanced
- Also called: Paged Attention

When a model writes an answer, it keeps the keys and values it computed for every earlier token so it does not compute them again. This [KV cache](https://tecyfy.com/concepts/kv-cache) is large: in the PagedAttention paper's example, a 13 billion parameter model on a 40 GB GPU spends about 65% of the memory on its weights and close to 30% on the cache. The cache, not the model, decides how many requests fit at once.

## The problem it solves

A server answers many requests together in a batch, because the GPU is far more efficient that way. Each request's cache grows as it writes, and no one knows in advance how long an answer will be. Earlier serving systems reserved a contiguous block of memory per request, big enough for the longest possible answer. Most of it sat empty: the paper's measurements found only 20.4% to 38.2% of the cache memory in those systems held tokens. Wasted cache memory means fewer requests in each batch, and fewer requests a second.

## How it works

The idea is borrowed from how an operating system manages memory. PagedAttention splits each request's cache into small blocks of a fixed number of tokens. Blocks are handed out as the answer grows and do not need to sit next to each other; a table records where each request's blocks are, as a page table does for a program. So:

- almost nothing is reserved and left unused, since a request holds only the blocks it has filled, plus part of the last one
- blocks are all the same size, so free memory never breaks into unusable gaps
- requests that share a start, such as several answers to the same prompt, can share those blocks instead of copying them

The attention computation itself is rewritten to read keys and values from blocks scattered in memory.

## What it changes

The same paper introduced [vLLM](https://tecyfy.com/concepts/vllm), the serving engine built on PagedAttention. With near-zero waste in the cache, it fits more requests per batch, and it measured 2 to 4 times the [Throughput](https://tecyfy.com/concepts/throughput) of the leading systems at the same latency, with the biggest gains on long sequences and large models.

For someone serving an open model, the practical lesson is that memory planning is mostly cache planning. The weights are fixed; the cache grows with the number of conversations and their length. Size the GPU for both, and expect the serving engine to make good use of whatever is left after the weights.

## Related

- [KV Cache](https://tecyfy.com/concepts/kv-cache): Work a model saves from the text it has already read, so it is not redone for every new token. Prompt caching keeps that work between requests and charges less for it.
- [vLLM](https://tecyfy.com/concepts/vllm): An open-source engine for serving language models fast, built on PagedAttention, that answers requests through the same kinds of API the big model makers offer.
- [Throughput](https://tecyfy.com/concepts/throughput): How much work a model system gets through in a given time, counted in requests or tokens per minute; on an API it is capped by rate limits, on your own hardware by batching and memory.
- [Inference Latency](https://tecyfy.com/concepts/inference-latency): The time a model takes to produce an answer, split into reading the prompt (prefill, which sets the time to first token) and writing the answer one token at a time (decode).
- [Inference](https://tecyfy.com/concepts/inference): Using a trained model to produce an answer. Training builds the model once; inference is every time someone asks it something, and it is what you pay for per token.

## Sources

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://arxiv.org/abs/2309.06180)