Your new prompt scores 78% and the old one 75% on the eval set. How do you know the difference is real?

Treat the scores as estimates with error bars. Compute the standard error and a confidence interval for each score, compare the two versions on the same questions (a paired comparison), account for related questions and for run-to-run randomness, and check whether the eval set is big enough to detect a 3-point change at all. Often it is not.

Why 78 vs 75 may be noise

With 200 questions and a score near 75%, the standard error is about √(0.75 × 0.25 / 200), roughly 3 percentage points. The 95% interval (score ± 1.96 standard errors) is about ±6 points. A 3-point gap between two separate scores is well inside that.

What Anthropic recommends (in "A statistical approach to model evaluations")

1. Report the standard error of the mean with each score, and a 95% confidence interval of mean ± 1.96 × SEM.
2. Cluster standard errors when questions come in groups, such as several questions about the same document. Treating them as independent understates the error; the paper reports clustered errors on popular evals more than three times as large as naive ones.
3. Reduce variance within questions: sample several answers per question and average, so one lucky or unlucky generation matters less.
4. Compare pairs: both versions answer the same questions, so look at the per-question differences. This removes the variation due to question difficulty and usually gives a much tighter interval than comparing two separate scores.
5. Use power analysis to decide how many questions you need to detect the size of difference you care about.

In practice

• Run both versions on the same set, several samples each.
• For each question, compute new minus old. Report the mean difference and its confidence interval. If the interval excludes zero, the gain is likely real.
• Look at which questions changed. Gains on 10 questions and losses on 7 tell a different story from a clean improvement.
• If the set is too small for the differences you care about, grow it rather than over-reading results.

Other checks

• Was the new prompt tuned on this same set? Then the gain may be overfitting. Confirm on a held-out set.
• Did the grader change? Compare with the same judge model and prompt.
• Did cost or latency change? A real 3-point gain may not be worth double the tokens.

What interviewers want: error bars, paired comparisons and sample-size thinking, rather than declaring a winner from two numbers.