# What is inference latency, and why is a model slower to write than to read?

The time a model takes to produce an answer, split into reading the prompt (prefill, which sets the time to first token) and writing the answer one token at a time (decode).

- Page: https://tecyfy.com/concepts/inference-latency
- Subject: How AI works (https://tecyfy.com/concepts)
- Level: intermediate
- Also called: Time Per Output Token, TPOT, Prefill and Decode

Inference latency is the time a model spends producing an answer, leaving out the network around it. It has 2 phases with different costs, and knowing them explains most of what you see in a latency graph.

## Prefill: reading the prompt

First the model processes the whole prompt. All of its tokens can be handled at once, in parallel, so even a long prompt is read in one pass. At the end of it, the first output token is ready. That is why the prompt's length mainly affects the time to first token, and why caching the start of a prompt shortens it: the work for those tokens is already done and kept in the [KV cache](https://tecyfy.com/concepts/kv-cache).

## Decode: writing the answer

Then the model writes, one token at a time. Each new token needs another pass through the model, with everything before it as input. The time per output token (TPOT) is roughly steady, so the total time grows with the answer's length. A 1,000-token answer takes about 10 times as long to write as a 100-token one, whatever the prompt.

For an engineer the rule of thumb follows: to make an answer faster, make it shorter. Fewer output tokens, including thinking tokens, cut latency more reliably than anything you do to the prompt.

## Why providers trade it against throughput

A server running a model answers many users at once by batching their requests together, which raises [Throughput](https://tecyfy.com/concepts/throughput) but can slow each request. The DistServe paper describes the tension directly: when prefill and decode share the same hardware, one request's long prompt delays others' token-by-token writing, so systems have to favour one latency over the other or buy more hardware. Serving engines such as [vLLM](https://tecyfy.com/concepts/vllm) exist largely to manage this, keeping more requests in a batch without wasting memory. On a hosted API this is the provider's problem, visible to you as variation in speed; on a model you run yourself, it is yours.

## Related

- [Latency](https://tecyfy.com/concepts/latency): How long a user waits for a model's answer, measured as the time to the first token and the time to the last, and judged by the slowest requests, not the average.
- [Throughput](https://tecyfy.com/concepts/throughput): How much work a model system gets through in a given time, counted in requests or tokens per minute; on an API it is capped by rate limits, on your own hardware by batching and memory.
- [KV Cache](https://tecyfy.com/concepts/kv-cache): Work a model saves from the text it has already read, so it is not redone for every new token. Prompt caching keeps that work between requests and charges less for it.
- [PagedAttention](https://tecyfy.com/concepts/pagedattention): A way of storing each request's KV cache in small fixed-size blocks, like pages of virtual memory, so a serving engine wastes almost none of it and fits more requests in a batch.
- [vLLM](https://tecyfy.com/concepts/vllm): An open-source engine for serving language models fast, built on PagedAttention, that answers requests through the same kinds of API the big model makers offer.
- [Token](https://tecyfy.com/concepts/token): The small piece of text a language model reads and writes, often part of a word. Prices, limits and the context window are all counted in tokens.

## Sources

- [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://arxiv.org/abs/2401.09670)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://arxiv.org/abs/2309.06180)
- [Attention Is All You Need](https://arxiv.org/abs/1706.03762)