The demo worked well, but accuracy in production is poor. How do you debug it?

The demo used clean, hand-picked inputs, and production gets the real distribution. So first collect and categorise the failures from production traces, then find which step fails for each category (input handling, retrieval, prompt, model, integration, or a mismatch with what users actually want), fix the biggest category first, and add those cases to the evaluation set so the gap between demo and reality closes for good.

1. Get real failures, not impressions

"Accuracy is poor" is a feeling. Ask for examples, pull traces of 50 to 100 recent requests, and label each: correct, wrong, partly right, refused, too slow. You now have an accuracy number and a list of failure examples.

2. Categorise

Common patterns behind demo-to-production drops:

• Inputs differ: scanned PDFs instead of clean text, other languages, very long emails with long reply chains, abbreviations and internal jargon.
• Content gaps: the answers are not in the documents the system can see, or the documents are out of date.
• Retrieval misses: the right document exists but is not found (chunking, no keyword search, filters, user vocabulary).
• Generation errors: retrieved correctly but the answer ignores or misreads it.
• Integration bugs: wrong customer id passed, a timeout returning an empty tool result, truncated context.
• Expectation mismatch: users ask questions the system was never scoped for, or "correct" means something different to them.

3. Trace each category to a step

For each failure, look at the input, what was retrieved, what the tools returned and the final prompt. Retrieval logs settle most arguments quickly: if the right passage was not in the prompt, no prompt change will fix it.

4. Fix in order of impact

• Parsing and data problems first. They affect everything downstream.
• Then retrieval (hybrid search, better chunking, query rewriting).
• Then prompts and examples for the cases that remain.
• Out-of-scope requests: detect them and hand off politely, and talk with the sponsor about scope.

5. Fix the measurement

The demo set did not represent production. Rebuild the evaluation set from real traffic in proportion to the categories, and report accuracy per category. Overall accuracy can hide a category that fails completely.

6. Communicate

Tell the customer what you found, with numbers: "40% of failures are scanned documents our parser misreads; fixing that is this week's work; expected gain: about 15 points." Customers forgive problems; they do not forgive silence.

What interviewers listen for: you debug with data rather than prompt tweaks, you separate retrieval from generation, and you keep the customer informed while you fix it.