A new model version is released. How do you decide whether to switch?

Run your evaluation set on the new model with the current prompts, compare quality, cost and latency per category of input, fix any prompts that rely on the old model's quirks, then roll out gradually with monitoring and a way back. Switch when it is better on what your users need, or when the old model is being retired, not because it scored higher on public benchmarks.

1. Pin versions

Call a specific model version in production, not a moving alias, so behaviour changes only when you choose. Record the model id with every request and every evaluation result.

2. Know the timeline

Providers retire old models. Anthropic's lifecycle runs active → legacy → deprecated → retired, and requests to a retired model fail. It gives at least 60 days' notice before retiring a publicly released model and advises testing replacements well before that date. So model upgrades are not optional forever: plan for them as routine work.

3. Evaluate

• Run the full evaluation set on old and new models, with several runs each where output varies.
• Compare per criterion (accuracy, format, safety, tone) and per input category, not only the average. A model can improve overall and get worse on one important case.
• Measure cost per task (new models may use more or fewer tokens, including thinking tokens) and latency.
• Rerun the safety and red-team suite: refusal behaviour changes between versions.

4. Expect prompt and API changes

• Prompts often contain workarounds for the old model ("do NOT add a preamble"). The new one may not need them, or may follow them too literally.
• Parameters change too. Anthropic's newer models, for example, reject non-default temperature values, so code that sets one will fail.
• Read the provider's migration notes.

5. Roll out safely

• Shadow mode: send a copy of live traffic to the new model and compare outputs without showing them.
• Then a small share of real traffic, behind a flag, with quality, cost and latency compared live.
• Keep the old model ready to switch back until the new one has run cleanly for a while.

6. Decide

Switch if it is at least as good on the cases that matter, the cost and latency fit, and safety holds. If it is better on average but worse somewhere important, route that slice to the old model while you fix the prompt, until the retirement date forces the issue.

What interviewers want: evidence over hype. "Benchmarks tell me which models to try. My evaluation set tells me which to ship."