LLM-as-a-Judge: Evaluating AI Models with Amazon Nova on SageMaker AI
How a model can judge other models' answers, how to read a win rate and its confidence interval, and the biases to guard against, with Amazon Nova LLM-as-a-Judge as the example.
Tecyfy · Published · Updated · 4 min read
Written in January 2026, when this was news. Models and prices have moved on since: today's are on Model prices.
Why judging with a model
Scores like BLEU and ROUGE count how many words a model's answer shares with a reference answer. They cannot tell whether a summary is useful, whether the tone suits a customer, or whether an answer is right but worded differently. People can, but asking people every time you change a prompt or a model is slow and costly.
LLM-as-a-judge fills that gap: a language modelLarge Language ModelA model trained on a huge amount of text to predict the next token. Further training turns it into an assistant that follows instructions. reads a prompt and the answers to it, and decides which is better or gives a score. A 2023 study behind the MT-Bench benchmark found that strong judge models agreed with human preferences over 80% of the time, about as often as people agree with each other.
Amazon's Nova LLM-as-a-Judge, released on SageMaker AI in January 2026, is a good worked example of how such a judge is used and how its results should be read.
How a pairwise judge works
Nova's judge compares 2 answers side by side and picks the better one, or calls a tie. You give it a file with one comparison per line, in JSON Lines:
{"prompt": "Explain photosynthesis.", "response_A": "Answer A...", "response_B": "Answer B..."}
{"prompt": "Summarize the article.", "response_A": "Answer A...", "response_B": "Answer B..."}
response_A and response_B hold the 2 outputs you want to compare, for example from a baseline and from the version you want to try. SageMaker AI runs the evaluation as a training job and writes the results to S3. AWS trained the judge in stages of supervised training and reinforcement learning on human preference data, and reports that, over more than 10,000 human judgements across 75 other models, its own internal study, over more than 10,000 human-preference judgements against 75 third-party models, found a 3% aggregate bias relative to human annotations. AWS still recommends spot checks by people on comparisons that matter.
Reading the result: a win rate is not enough
The judge returns how often each side won, and a win rate for model B with a 95% confidence interval. AWS's own example:
| Model A preferred | 16 |
| Model B preferred | 10 |
| Win rate for B | 0.38 |
| 95% interval | 0.23 to 0.56 |
It is tempting to read 38% as "A is better". But the interval runs past 0.5, so with 26 comparisons this result cannot tell the 2 models apart. The rule:
- Interval entirely above 0.5: B is better.
- Interval entirely below 0.5: A is better.
- Interval crosses 0.5: inconclusive. Add more comparisons.
You can check this yourself. This function computes a win rate with a 95% Wilson interval, a standard interval for a proportion:
from math import sqrt
def win_rate(b_wins, a_wins, z=1.96):
"""Share of comparisons Model B won, with a 95% Wilson interval. Ties are left out."""
n = a_wins + b_wins
p = b_wins / n
centre = (p + z * z / (2 * n)) / (1 + z * z / n)
spread = z * sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / (1 + z * z / n)
return p, centre - spread, centre + spread
for a, b in [(16, 10), (160, 100)]:
rate, low, high = win_rate(b, a)
verdict = "inconclusive" if low < 0.5 < high else ("B is better" if low > 0.5 else "A is better")
print(f"A {a}, B {b}: B wins {rate:.0%} (95% interval {low:.0%} to {high:.0%}) -> {verdict}")
A 16, B 10: B wins 38% (95% interval 22% to 57%) -> inconclusive
A 160, B 100: B wins 38% (95% interval 33% to 45%) -> A is better
The same 38% is inconclusive over 26 comparisons and a clear result over 260. Decide how many comparisons you need before you run the evaluation, not after you see the number.
The biases to guard against
The MT-Bench study examined 3 biases that LLM judges can show:
- Position bias. Preferring the answer shown first, or second. One way to reduce it is to run each comparison twice with the answers swapped and reconcile the 2 results. AWS reports 4 of its 5 benchmark results this way, and PPE as a single pass.
- Verbosity bias. Preferring the longer answer, even when it is no better. Watch for a new prompt that "wins" only by writing more.
- Self-preference. A judge favouring answers from its own model family. Use a judge from a different family from the models you compare, or check its bias, as AWS did for Nova.
A judge is also limited by its own knowledge: on maths, code or facts it may not be able to check an answer that sounds right. For those, run the code, check the answer against known results, or groundGroundingMaking a model answer from sources you give it, such as retrieved documents, and only from them, so every statement can be traced to a passage and a missing answer is admitted, not invented. the judgement in a reference answer.
Where it fits
LLM-as-a-judge works best for comparisons you repeat: choosing between 2 prompts, checking that a new model version is no worse, or picking the best checkpoint during fine-tuningFine-TuningTraining an existing model a little further on your own examples so it picks up a task, a format or a style. Often a good prompt does the job first.. Keep a small set of comparisons judged by people too, and check now and then that the judge still agrees with them.
Tecyfy Takeaway
A judge model lets you compare 2 versions of a prompt or model without asking people to judge every answer. Read its result as a win rate with a confidence interval, never as a bare percentage: if the interval crosses 0.5, you do not yet have an answer. Swap the order of the answers, watch for long answers winning on length, use a judge from another model family, and keep people checking a sample.
Questions
- What is LLM-as-a-judge?
- LLM-as-a-judge uses a language model to read a prompt and its answers and decide which is better, or give a score. The MT-Bench study found strong judge models agreed with human preferences over 80% of the time, about as often as people agree with each other.
- What is Amazon Nova LLM-as-a-Judge?
- A pairwise judge model on SageMaker AI, released in January 2026. You give it a JSON Lines file of prompts with 2 answers each, it runs as a training job and writes how often each side won, with a win rate and a 95% confidence interval, to S3.
- How do I read a win rate from an LLM judge?
- Look at the 95% interval, not the bare percentage. If it lies entirely above 0.5, model B is better; entirely below, model A is better; if it crosses 0.5, the result is inconclusive and you need more comparisons.
- What biases do LLM judges have?
- Position bias (preferring the first or second answer), verbosity bias (preferring the longer answer) and self-preference (favouring its own model family). Run each comparison with the answer order swapped and reconcile the 2 results, watch for wins on length, and use a judge from another family.
Sources