How do you test prompt changes before they reach production?

Treat prompts as code: keep them in version control, review changes, and run an automated evaluation suite on every change that compares the new version with the current one on the same cases. Release only if it passes the gates (no regressions on critical cases, overall quality not worse, cost and latency within budget), then roll out gradually with monitoring.

1. Prompts in version control

• Store prompts as files or templates next to the code, not in a database someone edits by hand.
• Each change gets a pull request with a reason and the evaluation results.
• Each production output records the prompt version that made it.

2. The test suite

• Unit-style checks for must-hold behaviour: refuses out-of-scope requests, never leaks the system prompt, always returns valid JSON, handles empty input.
• Quality set: representative cases graded by code, an LLM judge with a rubric, or both.
• Regression cases: every bug ever fixed.
• Safety set: injection and jailbreak attempts relevant to the feature.

3. Compare, don't just score

Run old and new prompts on the same cases, several samples each if outputs vary, and compare per case. Report the mean difference with a confidence interval (Anthropic's statistical evaluation guidance recommends this paired approach), and list the cases that got worse. A small overall gain with a regression on a critical case should not ship.

4. Gates

• Critical cases: all must pass.
• Overall quality: not significantly worse.
• Cost and latency per case: within budget (prompt changes often add tokens).
• Output length: no drift towards much longer or shorter answers.

5. Where to run it

On demand from the command line, and before every release. Full suites with LLM judges cost money and time, so keep a quick subset for every edit and the full set before release.

6. Release gradually

Behind a flag or to a small share of traffic, with live quality sampling and the ability to switch back in seconds.

Things that break silently

• Changing the order of instructions can change behaviour.
• Fixing one category often harms another; per-category scores catch this.
• A model version change with the same prompt is also a change: run the suite.

What interviewers want: version control, automated comparison on the same cases, explicit release gates and rollback. "I tried it on three examples and it looked better" is the answer they are screening out.