How do you reduce the latency of an LLM application?

First measure where the time goes: time to first token (TTFT, what the user feels) and total time (which grows with output length). Then stream the answer, so users see it start at once. Generate fewer tokens, since output is the slow part. Cache long repeated prompt prefixes, use a smaller or lower-effort model where it is good enough, and run independent steps in parallel.

Measure first

Anthropic's latency guide separates baseline latency (the model's processing time) from time to first token. Log both for each step of your pipeline (retrieval, reranking, each model call, tool calls). It is common to find that retrieval or a second model call, not generation, is the slow part. The guide also says to get quality right first, then cut latency, so you know what you are trading.

The levers, roughly by impact

1. Streaming. It does not shorten total time, but users see text within the first second instead of waiting for the whole answer. It is essential for chat.
2. Shorter output. Generation speed is roughly fixed per token, so 150 tokens finish far sooner than 800. Ask for concise answers, use structured output instead of prose where code consumes it, and set max_tokens.
3. Shorter or cached input. Trim the prompt to what the task needs. With prompt caching, a long system prompt or document that repeats across calls is not processed from scratch each time, which reduces both cost and time to first token.
4. Model choice and effort. A small fast model is often good enough for classification, routing and extraction. On models with an effort or thinking setting, lower it for routine requests.
5. Fewer sequential calls. Merge steps where one call can do both. Run independent calls (retrieval and a database lookup, or several subqueries) in parallel.
6. Faster retrieval. Tune the vector index for latency, cache embeddings of frequent queries, and rerank fewer candidates.
7. Answer ahead of time. Precompute answers for the most common questions, or summaries of documents that rarely change.

Perceived latency

Show progress ("Searching 3 sources…"), stream partial results, and return the first useful piece early. Users judge waiting by what they see.

Trade-offs to state

Smaller models and shorter prompts can lower quality, and aggressive caching can serve stale answers. Run your evaluation set after each change, and watch the 95th-percentile latency, not just the average. A slow tail is what users complain about.