What is an eval set for an LLM app, and how do you build one?

The list of test cases an LLM app is graded on after every change. Each case is an input and what a good answer must do, so a score going up or down shows whether the change helped.

intermediate · Also called Evaluation Set, Eval Dataset, Golden Dataset, Test Cases for LLMs

An eval set is the fixed list of test cases you run an LLM app on every time something changes: the prompt, the model, the retrieval, a tool. Each case holds an input and what a good answer must do. You run all of them, grade each one, and read the score. If it drops, the change broke something, and the failing cases tell you what.

It plays the part unit tests play in normal code, with 2 differences. The app can answer the same input differently each run, so a case may be run several times. And a good answer can be worded in many ways, so a case says what the answer must do rather than the exact words.

What a case looks like

A case for a support-email classifier might be:

Input: "The bowl of my mixer arrived scratched. I want my money back. Order KX-50117."
Must do: category refund, order id KX-50117, not urgent, and the reply promises no refund.

Each "must do" is something a grader can decide: code for the fields, a model judge or a person for the reply. A case where 2 people who know the job would disagree about pass or fail is not testing the app, so rewrite it until they would agree.

Where the cases come from

The best cases are real failures. Read 100 or so outputs of the app, written down as traces, note each one that is wrong, and group the notes into kinds of failure. Each kind gets cases. Bug reports and the support queue add more once the app is live.

Cover both sides of every behaviour. If cases check that urgent emails are marked urgent, others must check that calm ones are not; otherwise an app that marks everything urgent scores perfectly. Add the edge cases real users send: an empty message, 2 order numbers, a different language, an instruction hidden in the text.

20 to 50 cases from real failures is a good start. To tell a small improvement from noise you need more, because a score on few cases has a wide margin of error. Synthetic cases, written by you or a model, fill gaps before launch and cover rare situations, but only real traffic shows how often a failure happens.

2 sets, not 1

Keep the cases the app must always get right in a regression set, which should pass close to 100% and runs before every change. Keep the cases it cannot do well yet in a capability set, which starts low and shows progress. When a capability case passes reliably, move it to the regression set. When a whole set passes almost everything, it has stopped telling you anything new: keep it as a guard and write harder cases.

An eval set is never finished. Every failure a user reports is a case you did not have.

Sources

Where each fact comes from

Asked in interviews

LLM as a Judge
Using a language model to grade another model's output against a rubric, for qualities code cannot check. It scales to thousands of answers, but must be checked against a person's verdicts before it is trusted.
Tracing
Recording every step an LLM app or agent takes for one request, each model call, tool call and retrieval with its input, output, tokens and time, so you can see why an answer went wrong.
Hallucination
When a model states something false as if it were true, such as a wrong date, a made-up quote or a source that does not exist.
System Prompt
Instructions the builder of an AI app gives the model before you type anything. It sets the role, the tone and the rules, and it outranks what users say.