The full answer
"Nobody changed the code" rarely means nothing changed. An LLM feature also depends on the model, the prompts and settings kept outside the code, the retrieved documents, the tools it calls and the questions users ask. First split the problem in two: replay last week's good requests through today's system. If the answers got worse, the system changed. If they did not, the inputs changed.
1. Pin down what "dropped" means
Which metric moved, and when? A thumbs-down rate, escalations to people, an online judge score, or complaints. Find the day it started and whether it hit every kind of request or one segment (a customer, a language, a document type, a feature). A change that began at a precise hour points at a deployment or config change; a slow slide points at data.
2. Replay to separate system from inputs
Take a sample of requests from before the drop, with the answers you logged then, and run them through today's pipeline.
- Answers changed for the worse: something in the system changed. Go to step 3.
- Answers still fine: the system is the same, and users are now asking different things. Go to step 4.
This only works if you log each request with what it used: prompt version, model ID, retrieved document IDs, tool results.
3. If the system changed, check everything that is not code
| Part | What to check |
|---|---|
| Model | Is the model ID a pinned snapshot? On the Claude API, every model ID from the 4.6 generation on is a pinned snapshot; for older models an alias is a pointer to a dated ID, so check which one your code resolves to. Other providers and cloud platforms have their own rules. Did a fallback model start serving traffic because of rate limits or errors? |
| Prompt and settings | Prompts kept in a config store, feature flags, temperature, max_tokens, effort |
| Retrieval | Index rebuilt? Documents added, removed or re-chunked? Embedding model changed for new documents but not old ones? A filter dropping more than it should? |
| Tools | An API the agent calls changed its output or started failing quietly |
| Context | Longer conversation histories or larger documents crowding out instructions |
Compare traces from before and after for the same kind of request. The difference is usually visible: other documents retrieved, a tool returning empty results, a truncated answer.
4. If the inputs changed
New users, a new product line, a new language, or a seasonal question the system was never tested on. Read a sample of the failing conversations. You will usually find a cluster of questions the evaluation set does not cover.
5. Fix it and close the gap
- Fix the cause: pin the model, roll back the config, rebuild the index properly, add the missing documents.
- Add the failing cases to the evaluation set, so this failure is tested from now on.
- Monitor the inputs too: retrieval scores, the share of "I don't know" answers, request topics and lengths. Production monitoring exists to catch exactly this kind of drift.
What interviewers want: the replay test that separates system changes from input changes, a list of the parts that change without a code deploy (model, prompts, index, tools, users), and logging rich enough to compare before and after.
Last updated 16 October 2026
Sources