Model routing: send each request to the cheapest model that can do it

Model routing sends each request to the cheapest model that can answer it well, and the simplest version to build is a cascade: a small model answers first, and only unsure or risky requests go on to a large one. On 1 million support tickets, sending all of them to Claude Sonnet 5.5 costs $7,000; a cascade where Claude Haiku 5.5 answers first and 30% move on to Sonnet costs $2,450.

The idea and the 3 ways to route (a fixed model per job, a classifier, a cascade) are in Model RoutingModel RoutingSending each request to the cheapest model that handles it well, by rules, a classifier or a cascade, and checking with an eval that quality holds.. Our Opus vs Sonnet vs Haiku article shows a monthly bill split 70/30. This article builds the cascade, prices it honestly (an escalated request pays twice) and shows how to test it.

A cascade in Python

The job: sort support tickets into categories. Haiku answers first with a structured outputStructured OutputA setting that makes a model's answer follow a JSON schema you give it, so your code can read fields such as a category or an order number without parsing free text., so the answer always parses, and says whether it is sure. The request moves up to Sonnet when Haiku is unsure or the category is one where a mistake costs real money, whatever Haiku says.

import json
import anthropic

client = anthropic.Anthropic()
CATEGORIES = ["billing", "delivery", "refund", "account", "legal", "other"]
RISKY = {"refund", "legal"}  # always checked by the larger model

schema = {
    "type": "object",
    "properties": {
        "category": {"type": "string", "enum": CATEGORIES},
        "sure": {"type": "boolean"},
    },
    "required": ["category", "sure"],
    "additionalProperties": False,
}

def ask(model, ticket, **extra):
    r = client.messages.create(
        model=model,
        max_tokens=1024,
        system="Sort the support ticket into one category. Set sure to false if it fits several or none.",
        messages=[{"role": "user", "content": ticket}],
        output_config={"format": {"type": "json_schema", "schema": schema}},
        **extra,
    )
    return json.loads(next(b.text for b in r.content if b.type == "text"))

def classify(ticket):
    first = ask("claude-haiku-5-5", ticket, thinking={"type": "disabled"})
    if first["sure"] and first["category"] not in RISKY:
        return first | {"model": "haiku"}
    return ask("claude-sonnet-5-5", ticket) | {"model": "sonnet"}

Haiku 5.5 thinks by default; for a one-word label we turn that off, which Haiku allows at the default effort. Log which model answered every request: the escalation rate is the number that decides the bill.

What it costs

Take 1 million tickets a month, 2,000 input and 300 output tokens each, at Claude API prices ($0.10 and $0.50 per million tokens for Haiku 5.5, $2 and $10 for Sonnet 5.5, $4 and $20 for Opus 5.5):

Every request pays for Haiku, and an escalated one pays for Sonnet too. That waste is small here because Haiku costs a twentieth of Sonnet. The rule:

A cascade costs less than the large model alone while the escalation rate is below 1 minus the price ratio. Haiku to Sonnet breaks even at 95% escalated, so it almost always saves. Sonnet to Opus is different: Sonnet costs half of Opus, so the cascade only saves while fewer than 50% move up. At 30% it costs $11,200 against $14,000 for all-Opus; at 60% it costs more than all-Opus and is slower for the requests that move.

Try itDoes the cascade pay?

Cascade

·Haiku → Sonnet·Sonnet → Opus·Haiku → Opus

Requests a month

1000000

Input tokens each

2000

Output tokens each

300

Moved up to the larger model

30%

All on Sonnet 5.5$7,000

Cascade$2,450

All on Haiku 5.5$350

Green: escalation rates where the cascade saves (0 to 95%). Line: yours.

Saves 65%Break-even 95%

Escalated requests also wait for 2 answers. For a chat reply that may matter more than the money; for a background job, it does not.

Test it before you trust it

A router makes 2 kinds of mistake. Sending an easy request up only costs money. Keeping a hard one on the small model costs quality, and nothing in the logs shows it. Haiku can be sure and wrong; the cascade then never asks Sonnet.

1. Build a test set of a few hundred real requests with the right answer for each. Our eval sample size calculator says how many you need to see a difference of the size you care about.
2. Run it 3 ways: all on the small model, all on the large one, and through the cascade. Record accuracy, cost and escalation rate for each.
3. Count the confident mistakes: requests Haiku marked sure, got wrong, and did not escalate. That is the cascade's real error rate, and the number to push down by adding risky categories or checks.
4. Keep the cascade only if its accuracy is within what you can accept of all-large. Re-run the test whenever you change a model or a prompt.

Research routers show the size of the prize: RouteLLM's trained routers cut costs by over 2 times in some tests with no loss in quality, and the FrugalGPT cascade matched GPT-4 at up to 98% lower cost on its tasks. Your own test set is what says how much of that you get.

Managed routers

If you would rather not build one, 2 clouds sell routing as a service:

Both say which model answered, and both tell you to evaluate the router against your current setup. Neither knows that a refund mistake costs more than a delivery one; a rule like RISKY above is still yours to add.

What to do

1. Start with a fixed model per job: the cheapest model that passes your test for each step. That alone is routing.
2. Where one step sees both easy and hard requests, add a cascade from a model at least 10 times cheaper, so the double payment stays small.
3. Escalate on checks you control (risky categories, failed validation, long inputs) as well as on the model's own doubt.
4. Watch the escalation rate in production; if it climbs past the break-even, send that step straight to the larger model.

To see which model is cheapest for a job at the quality you need, use which model should I use; more ways to cut the bill, from caching to batches, are in how to reduce LLM API costs.