What is the difference between offline evaluation and online A/B testing of an LLM feature, and when do you use each?

Offline evaluation runs a fixed test set through the system before release and grades the outputs. It is fast, cheap and repeatable, and catches regressions. Online A/B testing shows the change to a share of real users and compares real outcomes (task completion, satisfaction, escalations, cost). Use offline evaluation on every change, and an A/B test before rolling out changes that could affect user behaviour or business results.

Offline evaluation

• A versioned set of inputs with expected results or rubrics.
• Graded by code, an LLM judge and some human review.
• Answers: "Is the new version at least as good on the cases we know about?"
• Weakness: it only covers what is in the set, and graders measure your idea of quality, not users' behaviour.

Online A/B testing

• Randomly assign users (or sessions) to the current version (control) or the new one (treatment).
• Compare product metrics: resolution rate, follow-up questions, thumbs down, hand-offs to people, time to complete, cost per session.
• Answers: "Does the change make things better for real users?"
• Weakness: slower, needs enough traffic, and exposes some users to a possibly worse version.

Using both

1. Offline evaluation as a gate: no change ships if it regresses the test set.
2. Shadow mode for risky changes: run the new version on live traffic without showing it, and grade the outputs.
3. A/B test on a small share, with guardrail metrics (error rate, complaints, cost) that stop the test if they worsen.
4. Roll out fully, and add surprising failures from the test to the offline set.

Statistics matter in both

Anthropic's paper on statistical evaluation recommends reporting standard errors and confidence intervals, comparing versions on the same questions (paired differences), and using power analysis to know how many cases you need to detect a difference. Without that, a 2-point gain on 100 cases is often noise. Online tests have the same problem: decide the sample size and the success metric before starting, and don't stop the test the first time it looks good.

LLM-specific pitfalls

• Outputs vary between runs; repeat offline runs or use several samples per case.
• Users change behaviour with a new version (longer chats may mean better engagement or more confusion); choose metrics that reflect success, not activity.
• Cost and latency belong in the comparison.

Summary: "Offline evaluation tells me it's not broken; an A/B test tells me it's better for users."