What is model routing, and how do you route between a cheap and an expensive LLM without losing quality?

Sending each request to the cheapest model that handles it well, by rules, a classifier or a cascade, and checking with an eval that quality holds.

intermediate · Also called LLM Routing, Model Cascade, LLM Router

Most apps send every request to one model. Model routing sends each request to the model that suits it: a small, fast, cheap model for the easy ones and a large one for the hard ones. On Claude the gap is wide: Claude Haiku 5.5 starts at $0.10 per million input tokens and Claude Opus 5.5 costs $4, 40 times as much.

3 ways to route

By job. Each step of your app gets its own model, fixed in code. Anthropic names classification, extraction and routing as Haiku's jobs, so an email classifier on Haiku and a reply writer on Opus is the simplest routing there is.
By a classifier. A small model or a trained classifier looks at each request and decides which model should answer. The RouteLLM study trained such routers on human preference data and cut costs by over 2 times in some tests with no loss in answer quality.
By cascade. The cheap model answers first, and the request goes on to the expensive one when the cheap answer looks risky: it failed validation, it fell in a category where mistakes are costly, or the model said it was unsure.

The catch

A router makes 2 mistakes: sending a hard request to the weak model, and sending an easy one to the strong model. The second only costs money. The first costs quality, and it is invisible unless you measure. A cascade that escalates on the cheap model's own output inherits its errors: if the small model files a complaint under "other", the rule that escalates complaints never fires.

So treat the routing rule as part of the app and run your eval set through it, comparing cost and pass rate with sending everything to each model alone. Keep routing only where the numbers say it costs you nothing you care about, and re-run the comparison when a model changes.

Routing is not the same as a fallback. A fallback switches model because the first one is failing, overloaded or timing out; routing switches because the request does not need the bigger one. Both need the second model to pass your eval.

Sources

Where each fact comes from

Eval Set
The list of test cases an LLM app is graded on after every change. Each case is an input and what a good answer must do, so a score going up or down shows whether the change helped.
Latency
How long a user waits for a model's answer, measured as the time to the first token and the time to the last, and judged by the slowest requests, not the average.
Batch Processing
Sending many model requests together to be answered within hours instead of seconds, at a lower price; the right choice for any work no one is waiting on.
Prompt Caching
Letting the API keep the processed start of a prompt for a few minutes, so later requests that begin the same way are cheaper and quicker.
Large Language Model
A model trained on a huge amount of text to predict the next token. Further training turns it into an assistant that follows instructions.