The full answer
Decide first what "good" means in measurable terms. Then collect realistic inputs, including edge cases, with the expected result or grading rule for each, and pick the cheapest reliable grader: code where possible, an LLM judge with a rubric where not, people for a sample. Run it on every change and treat it as a product asset that grows with each failure you find.
1. Write success criteria
Anthropic's evaluation guide starts here: criteria that are specific and measurable. "Good summaries" is not a criterion. "Covers all decisions and owners from the meeting, under 120 words, with no facts absent from the transcript" is. Usually there are several: accuracy, format, tone, safety, latency, cost.
2. Collect inputs
- Real traffic first, anonymised: it has the messiness that made-up examples lack.
- Edge cases: empty or very long input, other languages, ambiguous requests, adversarial input, questions outside scope.
- Known failures: every bug report becomes a test case.
- Aim for the distribution of real use. The guide's advice is to prefer more cases with automated grading over a few hand-graded ones.
3. Define the expected result
| Task type | What to store | Grader |
|---|---|---|
| Classification, extraction | The correct label or fields | Exact match, in code |
| Questions with one answer | Reference answer or key facts | Code checks or LLM judge |
| Open-ended writing | Rubric of criteria | LLM judge, plus human sample |
| Must not happen | The forbidden behaviour | Code check or classifier |
4. Grade
- Code-based checks (exact match, regex, JSON schema, numeric tolerance) are fast, cheap and repeatable. Use them wherever possible.
- LLM-as-a-judge for quality: a clear rubric, one criterion at a time, and a different model from the one being tested, as Anthropic advises. Check the judge against human labels.
- Humans for a regular sample and for anything high-stakes.
5. Keep it trustworthy
- Split a development set you tune on from a held-out set you only report on, or you will overfit the prompt.
- Version the set, and record which version produced which score.
- Run each case more than once if outputs vary, and compare pass rates.
6. Use it
Run it in CI or before each release: new prompt, new model, new retrieval settings. Track scores over time, per criterion and per category of input, so a gain in one area cannot hide a loss in another.
How big?
Start with 20 to 50 cases the day you start building. That already catches most regressions. Grow to hundreds as the feature matures, mostly from real failures.
Last updated 16 October 2026
Sources