The full answer
The evaluation is measuring something other than what users care about. Either its cases do not look like real traffic, its grader rewards answers people do not want, or its scores are saturated and no longer tell good from bad. Read the complaints, turn them into cases, check the grader against people, and keep adding production samples so the set follows the real traffic.
1. Read the complaints first
Pull 50 to 100 complaints with the full conversations and sort them by what went wrong: wrong facts, missing information, too long, wrong tone, refused something reasonable, ignored part of the question. That sorting is the most useful hour you can spend, because it tells you which gap to close.
2. Check the three ways an eval drifts from reality
| Gap | How to spot it | Fix |
|---|---|---|
| Coverage: the cases are not what users ask | Complaints cluster on topics, languages or lengths the set lacks | Sample real requests each month into the set, weighted to what fails |
| Grader: the scoring rewards the wrong thing | The judge passes answers that people rate badly | Calibrate the judge against human labels; rewrite the rubric around the complaint categories |
| Saturation: everything passes | Pass rates near 100%, and new versions all score the same | Add harder cases; an eval at 100% only catches regressions |
3. Calibrate the grader
Take 100 or so production answers, have people rate them with the same rubric the judge uses, and compare. Where they disagree, read both and fix the rubric or the judge prompt. Repeat until the agreement is high enough to trust. Common problems: the judge prefers longer answers, it cannot see the source documents so cannot check facts, or the rubric asks "is it helpful?" when users are complaining about accuracy.
4. Grade what users actually experience
Single-answer cases miss failures that only show up in a conversation: forgetting earlier context, contradicting a previous answer, looping. Add multi-turn cases. If the product calls tools or retrieves documents, grade the final outcome (did the user get the right refund, the right document) and not only the wording.
5. Close the loop permanently
- Every complaint that reveals a new failure becomes a test case.
- A fixed share of production traffic is sampled, graded by the calibrated judge and spot-checked by people each week.
- Online signals sit next to offline scores in one report: complaints, thumbs-down, escalations, re-asked questions.
What interviewers want: that a passing eval can still be the wrong eval, the three gaps (coverage, grader, saturation), calibrating the judge against human ratings, and a loop from production failures back into the test set.
Last updated 16 October 2026
Sources