Skip to the page
Tecyfy

LLM apps and RAG interview question · medium

The model your production app calls has been deprecated and retires in 6 weeks. How do you move to its replacement without breaking anything?

The short answerTreat it as a release, not a rename.

The full answer

Treat it as a release, not a rename. Find every place the old model is called, fix the request parameters the new model rejects, measure the new model on your own evaluation set and on cost, then switch gradually with a way back, finishing well before the retirement date. After that date every request to the old model fails, so the deadline is hard.

1. Find every caller

  • Export usage by API key and model from the provider's console (Anthropic's Usage page has an Export to CSV).
  • Search code and config for the model ID, including scheduled jobs and scripts nobody owns.
  • Check each platform separately: the same Claude model retires on different dates on Amazon Bedrock and Google Cloud.

2. Fix what the new model rejects

A newer model can return 400 to requests the old one accepted. Moving from Claude Sonnet 4.5 to Sonnet 5.5, for example:

Old requestOn the new model
temperature, top_p, top_kRemove them
thinking with budget_tokensUse effort instead
Prefilled assistant turn400; ask for the format in the prompt or use structured output
tool_choice any or a named tool400; use auto with strict tools

3. Measure before switching

  • Run the same few hundred real requests, with known good answers, on both models. Compare accuracy, refusals, format errors and latency, not a demo.
  • Re-check cost: the same text can count as about 30% more tokens on the new tokenizer, even if the price per token fell.
  • Re-tune prompts that were tuned to the old model's quirks.

4. Switch with a way back

  • Keep the model ID in one config value.
  • Send a small share of traffic first, watch errors and quality, then raise it.
  • Keep the old model as the fallback only until shortly before retirement.

5. Prevent the next scramble

Subscribe to deprecation notices, keep the evaluation set current, and put the next "not sooner than" retirement date for each model in the team calendar. See Model Deprecation.

What interviewers want: an inventory of callers across keys and platforms, the request changes the new model rejects, an evaluation on real requests with cost re-checked, a gradual switch from one config value, and a habit that makes the next retirement routine.

Last updated 22 October 2026

Sources

Where each fact comes from

  1. 1Model deprecations
  2. 2Claude Sonnet 5.5 migration guide