Is your eval gain real, or just noise?
On 200 shared questions, a gain from 75 to 78 percent is give or take 5.9 points, so it could be noise; catching it reliably takes about 1,568 questions.
Put in the score now, the new score and the number of questions in your eval. See the margin of error, whether the gain is real at the confidence you choose, the smallest gain your eval can catch, and how many questions it would need. The sums follow Anthropic's statistical approach to model evaluations.
The calculator
The current prompt, model or agent on your eval.
The change you are testing, or the score you hope it reaches.
Run both on the same questions and the correlation of their question scores narrows the error. Anthropic found 0.3 to 0.7 between frontier models on popular evals.
The answer
On 200 shared questions, 75 to 78 percent is a gain of 3 points, give or take 5.9. That could be noise. To catch a 3-point gain 80 times in 100, run about 1,568 questions.
- One score's margin
- ±6points
- Smallest gain it catches
- 8.1points
- Questions needed
- 1,568
Questions in the eval
| Gain over 75% | Same questions, 0.5 | Separate questions |
|---|---|---|
| 1 point | 14,519 | 29,034 |
| 2 points | 3,579 | 7,155 |
| 3 points | 1,568 | 3,132 |
| 5 points | 548 | 1,092 |
| 10 points | 126 | 248 |
Each question scored right or wrong, the questions independent of each other, one answer a question. Separate questions means each version ran on its own set of this size. Questions in groups, such as several on one document, need more: their errors can be over 3 times as large. Several answers a question narrow the error.
How the calculator works it out
An eval score is a mean: the share of questions answered right. Like any mean taken from a sample, it has a standard error, the square root of p × (1 − p) ÷ n for a score p on n questions, and a 95 percent confidence interval of the score plus or minus 1.96 standard errors. For a score of 75 percent on 200 questions, that is ±6 points: run another 200 questions of the same kind and the score could land anywhere in that range.
Two versions are best compared on the same questions. Some questions are hard for both and some easy for both, so the difference on each question varies less than the scores do. Anthropic's paper shows the error of the mean difference shrinks as the correlation of the 2 versions' question scores grows, and found that correlation to be 0.3 to 0.7 between frontier models on popular evals. In the example, a 3-point gain is give or take 5.9 points on shared questions at 0.5, and 8.3 on 2 separate sets.
Power is the chance an eval detects a gain that is really there. The questions needed grow with the square of the confidence and power you want, and fall with the square of the gain: catching 3 points 80 times in 100 at 95 percent confidence takes about 1,568 shared questions, and catching 1 point about 14,519. The chart shows the smallest gain each eval size can catch.
Questions
- How many questions does an LLM eval need?
- It depends on the smallest change you need to see. To catch a 3-point gain over 75 percent 80 times in 100 at 95 percent confidence, you need about 1,568 questions when both versions run on the same questions and their question scores correlate at 0.5, and about 3,132 on separate sets. A 1-point gain takes about 14,519. A few hundred questions only catch large gains: 200 shared questions catch about 8.1 points.
- How do I know if an eval score difference is statistically significant?
- Work out the confidence interval of the difference. If it leaves out zero, the difference is unlikely to be noise. Going from 75 to 78 percent on 200 shared questions is a gain of 3 points, give or take 5.9, so the interval runs below zero and the gain could be noise. Compare the 2 versions question by question rather than their 2 overall scores.
- What is the margin of error of an eval score?
- For a score p on n questions scored right or wrong, the standard error is the square root of p × (1 − p) ÷ n, and the 95 percent margin of error is 1.96 times that. A score of 75 percent on 200 questions is ±6 points. Anthropic recommends reporting it with every eval score.
- Why compare two models on the same questions?
- Because it removes the difference in how hard each question is and leaves only the difference in the answers. The error of the mean difference shrinks as the 2 versions' question scores correlate more. In the example, the margin of a 3-point gain is 5.9 points on shared questions at a correlation of 0.5, against 8.3 on separate sets, and the questions needed fall from 3,132 to 1,568, or about 942 at 0.7.
- What is statistical power in an eval?
- The chance the eval detects a difference that is really there. At 80 percent power, a real gain of the size you set shows as significant 80 times in 100; the other 20 times it hides in the noise. Anthropic recommends power analysis to decide how many questions to run, or whether an eval is worth running at all for the difference you care about.
- What if my eval questions come in groups?
- Questions about the same document or passage are not independent, so the score varies more than the simple formula says. Anthropic recommends clustered standard errors on the unit of grouping, and found them to be over 3 times as large as naive ones on popular evals. The calculator assumes independent questions, so with grouped questions treat its margins as the least they can be.
- Should I sample several answers per question?
- Yes, when a model answers differently each run, as with chain-of-thought reasoning. Anthropic recommends resampling answers several times and using each question's average as its score, which lowers the variation from run to run. It does not remove the variation between questions, so more questions still matter most.
- Does this work for scores from an LLM judge?
- For pass or fail verdicts, yes: each question scores 1 or 0, which is what the calculator assumes. For scores on a scale, such as 1 to 5, the same method holds with the standard deviation of your question scores in place of the square root of p × (1 − p); work it out from your own run.
Put this calculator on your own site
Copy the code into any web page: a blog post, a course page or your team's wiki. The calculator runs in a frame and shows the same figures as this page, so when the prices change here they change on your page too. It is free and needs no account, and nothing a reader types is sent anywhere.
The code opens the calculator with the settings you have chosen above.
The short script at the end lets the frame grow and shrink with the calculator. Where a page cannot take scripts, choose a fixed height and the frame scrolls.
<iframe src="https://tecyfy.com/embed/eval-sample-size" title="Eval sample size calculator" width="100%" height="2100" style="border:0;display:block" loading="lazy" allow="clipboard-write"></iframe>
<p><a href="https://tecyfy.com/ai-agents/eval-sample-size">Eval sample size calculator</a> by Tecyfy</p>
<script>addEventListener("message",function(e){if(e.origin!=="https://tecyfy.com"||!e.data||e.data.type!=="tecyfy:height")return;var h=Number(e.data.height);if(!(h>0))return;document.querySelectorAll("iframe").forEach(function(f){if(f.contentWindow===e.source)f.style.height=Math.min(h,20000)+"px"})})</script>