The full answer
Break each request's time into its parts and see which one grew: your own steps before the model call (retrieval, tools, database), waiting and retries around the call, the time to the first token, and the time spent generating output. A sudden jump overnight usually has one cause, such as a deploy, a config change, a cache that stopped working, longer outputs, or retries against a busy API, and the breakdown shows which.
1. Measure the parts, not just the total
For each request you want a trace with timings for:
- Retrieval and other preprocessing.
- Each model call: when it was sent, time to first token (TTFT), time to the last token.
- Input tokens, cached input tokens and output tokens.
- Retries and their status codes.
- Tool calls, for an agent.
Compare the p95 of each part before and after the jump. If you have no breakdown yet, add it first: guessing at latency causes wastes days.
2. Read the pattern
| What grew | Likely cause |
|---|---|
| Output tokens per answer | A prompt or model change made answers longer, or max_tokens was raised |
| TTFT, with input tokens unchanged | Prompt cache hits fell, or a larger model or higher effort setting is in use |
cache_read_input_tokens near zero | Something now changes the cached prefix on every request |
| Input tokens per request | More retrieved chunks, longer histories, a bigger system prompt |
| Time outside the model call | Slower retrieval, a slow tool or database, cold starts |
| Retries per request | 429 rate limits or 529 overloads; each retry adds the wait before it |
3. The prompt cache is a common culprit
Prompt caching saves time and money on the repeated start of a prompt, but only when that start is identical. A small change can break it for every request: a timestamp or user name added to the system prompt, tools sent in a different order, or a new tool definition (changing tool definitions invalidates the whole cache). Check cache_read_input_tokens in the usage of recent responses. If it dropped to zero overnight, find what made the prefix differ.
4. Retries hide inside the average
The SDKs retry rate-limit and server errors with backoff automatically. A request that succeeds on its third try looks like one slow request in your logs unless you record retries. If 429s rose, look at traffic growth, a new batch job sharing the same key, or a lower rate limit.
5. Fix and protect
- Fix the cause: restore the stable prompt prefix, cap output length, move a batch job to its own key or to the Batch API, cache retrieval results.
- Stream responses so users see the answer start, even when the total time is long.
- Alert on p95 TTFT, output tokens per request and cache hit rate, not only on total latency.
What interviewers want: a per-request breakdown before any fix, TTFT versus generation time, the prompt cache and retries as named suspects, and monitoring on the parts.
Last updated 16 October 2026
Sources