What is inference latency, and why is a model slower to write than to read?
The time a model takes to produce an answer, split into reading the prompt (prefill, which sets the time to first token) and writing the answer one token at a time (decode).
intermediate · Also called Time Per Output Token, TPOT, Prefill and Decode
Inference latency is the time a model spends producing an answer, leaving out the network around it. It has 2 phases with different costs, and knowing them explains most of what you see in a latency graph.
Prefill: reading the prompt
First the model processes the whole prompt. All of its tokens can be handled at once, in parallel, so even a long prompt is read in one pass. At the end of it, the first output token is ready. That is why the prompt's length mainly affects the time to first token, and why caching the start of a prompt shortens it: the work for those tokens is already done and kept in the KV cacheKV CacheWork a model saves from the text it has already read, so it is not redone for every new token. Prompt caching keeps that work between requests and charges less for it..
Decode: writing the answer
Then the model writes, one token at a time. Each new token needs another pass through the model, with everything before it as input. The time per output token (TPOT) is roughly steady, so the total time grows with the answer's length. A 1,000-token answer takes about 10 times as long to write as a 100-token one, whatever the prompt.
For an engineer the rule of thumb follows: to make an answer faster, make it shorter. Fewer output tokens, including thinking tokens, cut latency more reliably than anything you do to the prompt.
Why providers trade it against throughput
A server running a model answers many users at once by batching their requests together, which raises ThroughputThroughputHow much work a model system gets through in a given time, counted in requests or tokens per minute; on an API it is capped by rate limits, on your own hardware by batching and memory. but can slow each request. The DistServe paper describes the tension directly: when prefill and decode share the same hardware, one request's long prompt delays others' token-by-token writing, so systems have to favour one latency over the other or buy more hardware. Serving engines such as vLLMvLLMAn open-source engine for serving language models fast, built on PagedAttention, that answers requests through the same kinds of API the big model makers offer. exist largely to manage this, keeping more requests in a batch without wasting memory. On a hosted API this is the provider's problem, visible to you as variation in speed; on a model you run yourself, it is yours.
Sources