LLM as a judge: how to write a rubric you can trust

To write an LLM-as-a-judge rubric you can trust, start from answers a person has already graded, ask the judge for pass or fail on 1 clear question with a short reason first, and measure it on labelled answers it has never seen. Report 2 numbers, the share of bad answers it fails and the share of good answers it passes, never agreement alone.

A judge is a prompt like any other, and it fails like any other. The difference is that its failures hide inside your eval scores, where they look like your product getting better or worse.

1. Grade some answers yourself first

Before writing a word of the judge's prompt, have the person who knows the domain best grade real answers: pass or fail, with 1 or 2 sentences on why. Hamel Husain starts with about 30 and keeps going until no new kind of failure turns up.

This step writes the rubric for you. The reasons people give ("quoted a refund window we don't offer", "answered a different question") are the checks the judge needs. Researchers have found that people only settle their own criteria by grading outputs, so a rubric written before anyone has graded answers is a guess.

2. Ask for pass or fail, not 1 to 5

A 1 to 5 scale feels more precise and is less useful. What makes an answer a 3 and not a 4? Nobody can say, and 2 graders read the scale differently. A pass or fail verdict forces a clear line, and the written reason keeps the detail a score would lose.

Ask 1 question per judge. "Is it correct, polite and concise?" mixes 3 failures into 1 verdict, and a fail tells you nothing about which. Anthropic suggests a separate judge for each dimension.

3. Write the rubric as checks with examples

A rubric that holds up has 4 parts:

• The question, in 1 sentence: "Does the reply give the refund policy correctly for this order?"
• What fails, as concrete checks taken from the graded answers: a refund window other than 30 days, a promise of a refund for a used item, no answer to what was asked.
• What does not matter, so the judge stops penalising it: length, greeting, formatting.
• A few graded examples, pass and fail, each with its reason.

Then fix the output. Ask for the reason first and the verdict last, so the model reasons before it decides, and give it a way out:

<rubric>...the checks above...</rubric>
<order>{order}</order>
<reply>{reply}</reply>

Write 1 or 2 sentences on whether the reply breaks any check, then a verdict on its own line:
PASS, FAIL, or UNKNOWN if the order details are not enough to decide.

Give the judge the same facts the person had when grading (the order, the policy, the customer's plan). A judge asked "is this right?" without the facts can only guess.

4. Measure it the right way

Treat the judge as a test with 2 jobs: fail the bad answers and pass the good ones. Hamel Husain's split works well:

• Training examples (10 to 20% of the labels) may go in the prompt as examples.
• Development examples (40 to 45%) are where you run the judge, read where it disagrees with the person, and change the prompt.
• Test examples (the other 40 to 45%) stay hidden until the prompt is final. Run them once; that result is the judge's real accuracy.

Then report the true positive rate (the share of bad answers it fails) and the true negative rate (the share of good ones it passes). Not raw agreement, and here is why: if 5% of answers are bad, a judge that passes everything agrees with the person 95% of the time and catches none of them. Set the judge below to "Always says pass", then move the share of bad answers.

Try itWhy agreement flatters an LLM judge

Judge

Your judgeAlways says pass

Answers that are bad

5%of answers

as the person labelled them

5%20%50%

Bad answers it fails

40%true positive rate

a failure the person saw, the judge sees too

0%50%100%

Good answers it passes

98%true negative rate

no false alarms on good work

50%75%100%

100 answers

Bad, failed by the judgeBad, passed by the judgeGood, failed by the judgeGood, passed by the judge

Agreement95%

Bad answers caught40%

At the start, a judge that catches 40% of failures and passes 98% of good answers also scores 95% agreement. It looks as good as the useless one, and it lets 30 bad answers in every 1,000 through. The 2 rates show the difference; agreement hides it.

How many labels? About 100 per kind of failure to check a judge, with 30 to 50 of each label in the development and test sets where you can. Below 60, the range of doubt is usually too wide to say much.

5. Watch for the known biases

The MT-Bench study found strong judges agree with people about as often as people agree with each other, over 80% of the time, and listed 3 biases to design around:

• Position. In a comparison of 2 answers, judges favour 1 slot. Run each pair in both orders and count only verdicts that agree.
• Length. Judges favour longer answers. Say in the rubric that length does not matter, and include a short passing example.
• Their own writing. A judge can favour answers from its own model. Anthropic calls grading with a different model from the one that wrote the answer generally best practice, and Hamel Husain uses the strongest model his budget allows.

6. Keep checking it

A judge is right for the answers it was measured on. When you change the model under test, the judge's model or the product, grade a fresh sample yourself and rerun the test set. Once it holds, an occasional human check is enough.

The judge's verdicts then go into the rest of the eval: the outcome checks in code, the transcript limits and the repeated trials that how to evaluate an AI agent walks through. Before you trust a change in the judge's pass rate, the eval sample size calculator says whether it is larger than the noise. The concept page on LLM as a judgeLLM as a JudgeUsing a language model to grade another model's output against a rubric, for qualities code cannot check. It scales to thousands of answers, but must be checked against a person's verdicts before it is trusted. sums up when to use one at all.