What is LLM-as-a-judge, and when can you trust it?
Using a language model to grade another model's output against a rubric, for qualities code cannot check. It scales to thousands of answers, but must be checked against a person's verdicts before it is trusted.
intermediate · Also called LLM-as-a-Judge, LLM Judge, Model-Graded Eval, LLM Grader
An LLM judge is a model call whose only job is to grade. You give it an answer your app produced, a rule, and a few examples, and it returns pass or fail. It handles what code cannot: whether a reply promises a refund in some wording you did not foresee, whether an answer sticks to the source it was given, whether the tone is right for an angry customer.
Judges made large eval setsEval SetThe list of test cases an LLM app is graded on after every change. Each case is an input and what a good answer must do, so a score going up or down shows whether the change helped. practical, because a person cannot grade 5,000 answers after every change and a model can. Research on strong judges found them agreeing with human preferences over 80% of the time, about as often as people agree with each other. That is good enough to be useful and not good enough to trust blindly.
Where judges go wrong
The same research found 3 biases. A judge comparing 2 answers can prefer one for its position, first or second, rather than its content. It tends to prefer longer answers. And it can rate text written by its own model more highly. It also makes plain mistakes on questions that need careful reasoning.
Each has a counter. Ask about one answer at a time, or swap the order and ask again. Ask a specific question, so length does not look like quality. Grade with a different model from the one that wrote the answer. Let the judge reason before it gives the verdict, which helps on hard cases and tells you why it decided.
A judge that works
Good judges look alike:
- One question, pass or fail. "Does the reply promise a refund, a replacement or a delivery date?" A 1 to 5 score for "quality" is hard to act on, and people rarely agree on what a 3 means. Several questions need several judges.
- The rule and examples, taken from the notes of the person who knows the job best: a pass, a fail, and the borderline case that taught you the rule.
- A way out. Letting it answer "cannot tell" is better than a forced guess.
Check the judge before you trust it
Have that person label a set of answers pass or fail, ideally 30 or more of each, and run the judge on the same answers. Then report 2 numbers: the share of bad answers the judge fails, and the share of good answers it passes. Never just the agreement. If 6% of answers are bad, a judge that passes everything agrees 94% of the time and catches none of them.
When both numbers are high, the judge can grade every change. Keep a person reading a sample of its verdicts, because the app, its users and the judge's own model all change over time.
Sources