You must move from one LLM to its replacement. On your 200-question evaluation set the old model scores 84% and the new one 81%. Is the new model worse? How do you decide?

Not necessarily: on 200 questions a 3-point gap can be noise. Treat the eval as a sample and compare with statistics. Score both models on the same questions and analyse the paired differences per question, report a confidence interval for the gap, and if it still includes zero, either add questions (a power analysis tells you how many) or reduce noise by resampling answers. Alongside accuracy, compare what users would notice: format errors, refusals, latency and cost.

1. How big is the noise?

The standard error of a score near 84% on 200 questions is about √(0.84 × 0.16 / 200) ≈ 2.6 points, so each score alone is ±5 points at 95%. Two independent scores 3 points apart overlap heavily.

2. Use the pairing

Both models answered the same questions, so compare question by question:

Only the 30 disagreements carry information: 18 lost, 12 gained. Hard questions both fail and easy ones both pass no longer add noise. Anthropic's guidance on evals recommends exactly this paired analysis, since models' per-question results are correlated.

3. Other corrections

• Clusters: if several questions come from one document or one customer, compute errors by cluster; naive errors can be several times too small.
• Resample: run each question 3 to 5 times and average, so one unlucky sample does not decide a question.
• Power: decide the smallest drop that matters (say 3 points) and compute the questions needed to detect it before running.

4. Read the losses

Open the 18 questions the new model lost. A pattern, say every question needing a table, or one language, is a real regression even if the total is within noise, and usually a prompt change fixes it.

5. Decide

Switch when the interval for the gap excludes a drop you care about, the losses have no pattern you cannot fix, and cost and latency are acceptable. The eval sample size calculator does the interval and power arithmetic. See Eval Set and Model Deprecation.

What interviewers want: the score as a sample with a standard error, paired comparison on the same questions, clustered errors, resampling, power analysis, reading the lost cases, and a decision rule that includes cost and latency.