What is latency in an LLM app, and how do you reduce it?

Latency is how long someone waits. For a language model it has 2 parts that feel very different to a user: the time until the first word appears, called time to first token (TTFT), and the time until the answer is complete. A support chat that starts typing in under a second feels quick even if the whole reply takes 8 seconds; a form that shows a spinner for 8 seconds feels broken.

What makes it slow

• The model. Larger models are slower. Anthropic names Claude Haiku 5.5 as its fastest current model.
• The length of the answer. A model writes one TokenTokenThe small piece of text a language model reads and writes, often part of a word. Prices, limits and the context window are all counted in tokens. after another, so the answer's length sets most of the total time. Thinking counts here too: a model that reasons first writes those tokens before the answer, which is why a lower effort setting answers sooner.
• The length of the prompt. A long prompt takes longer to read before the first token comes out. Prompt cachingPrompt CachingLetting the API keep the processed start of a prompt for a few minutes, so later requests that begin the same way are cheaper and quicker. cuts this for a prompt that starts the same way each time.
• The network and the provider's load. Outside your control, which is why you measure rather than assume.

How to cut it, in order

1. Get the answer right first. Anthropic's advice is to build a prompt that works, then reduce latency. Speeding up a wrong answer saves nothing.
2. Ask for less. Shorter answers, a max_tokens limit, no preamble.
3. Stream. StreamingStreamingGetting a model's answer in small pieces as it is written, instead of all at once at the end, so people see text start sooner and long answers do not hit a timeout. does not make the answer faster, but the reader starts reading at the first token instead of the last.
4. Cache the fixed part of the prompt.
5. Choose a smaller model or lower effort for the jobs where your eval shows it does as well.

Measure the slow ones

An average hides the requests that annoy people. Log the time of every call and look at the median (p50) and the 95th percentile (p95), the time that 1 request in 20 is slower than. A p50 of 2 seconds and a p95 of 15 means 1 user in 20 waits 15 seconds or more, and an average of 3 seconds would never show it. The model-side detail of where the time goes is inference latencyInference LatencyThe time a model takes to produce an answer, split into reading the prompt (prefill, which sets the time to first token) and writing the answer one token at a time (decode)..