How do you evaluate an agent: task success, traces and LLM-as-a-judge?

Judge the outcome first: did the agent finish the task correctly, at what cost and in how many steps, across a fixed set of realistic tasks? Then read the traces to see why runs fail. Grade with code wherever the result can be checked, with an LLM judge and a rubric where it cannot, and with people for what neither catches.

1. Build a task set

• Real tasks from your users, plus edge cases: missing data, ambiguous requests, tools that fail.
• Start small. Anthropic's research team began with about 20 queries, because early changes move results so much that a few cases show them. Grow the set as you find failures.
• For each task, write down what success means before you run it.

2. Grade the result

• Code-based where possible. Do the tests pass? Is the row in the database? Does the answer contain the right order id? This is cheap, exact and repeatable.
• LLM-as-a-judge for open answers. Give a separate model a rubric (factual accuracy, citation accuracy, completeness, tool efficiency) and ask for a score per criterion. Anthropic found a single judge call scoring 0.0 to 1.0, plus a pass or fail, matched human judgement well. Use a different model to grade than the one being graded, and check the judge against some human-labelled cases.
• Humans for what rubrics miss. In the same project, testers noticed the agent preferred SEO content farms to authoritative sources, which the automated grading had not caught.

3. Grade the path, not only the end

For agents that change things, check the end state (the files, the tickets, the database), since many different paths can be right. Also track:

• Turns and tokens per task, and cost per successful task.
• Wrong or wasted tool calls, repeated calls and errors.
• Safety: did it attempt anything outside its permissions?

4. Read traces

Log every turn: the prompt, the tool calls, the results and the reasoning summaries. Most fixes come from reading 10 failed runs and seeing the same mistake, often a confusing tool description or an unclear error message.

5. Run it on every change

A new model, prompt, tool or framework version can raise one score and lower another. Agent runs vary, so run each task several times and compare pass rates, not single runs.

What to say at the end

"I trust an agent change when the pass rate on the same task set goes up and the cost per success does not. Not when one demo looks better."