GenAI in production: LLM cost, latency, fallbacks, batching and monitoring

The triage from modules 2 and 3 works on your laptop. In production it meets things a test run never shows: a bill that grows with every customer, a customer who waits, an API that is sometimes busy, and answers that slowly get worse with no error at all. This module adds what a GenAI feature needs to run for real: a budget, the cheapest way to meet it, a time limit, a fallback, and a log you can read.

Budget it before you build it

An LLM feature has a price per request you can work out before writing any code. Count the tokens each request sends and receives, multiply by the prices on the price page, and multiply by the traffic you expect.

For the shop's triage, using the token counts from the stand-in's log in the Build it part: a system prompt of about 600 tokens, cached after the first email, about 30 tokens of email and 160 tokens of output. On Claude Opus 5.5, at $4 per million input tokens and $20 per million output:

At 2,000 emails a day that is about $207 a month. The table also shows where the money goes: output is more than 90% of it. That is common for short tasks, because output costs 5 times as much per token as input on this model, and it tells you which change to try first. Write the estimate down with the traffic you assumed, and compare it with the real bill once the feature is live.

ToolCost calculatorWhat your job costs, a day, a month or a year

Cut the cost, in order of effort

Try these in order. The first 2 cannot change an answer; the last 2 can, so they need the eval from module 7.

1. Cache the part that repeats. Module 3 put the fixed instructions first and marked them for cachingPrompt CachingLetting the API keep the processed start of a prompt for a few minutes, so later requests that begin the same way are cheaper and quicker.. On Opus 5.5 and Sonnet 5.5 a cache read costs 5% of the input price. Check cache_read_input_tokens in the usage figures: 0 on every call means the cache is not working.
2. Send what can wait as a batch. BatchesBatch ProcessingSending many model requests together to be answered within hours instead of seconds, at a lower price; the right choice for any work no one is waiting on. cost half the normal price. Anything no one is waiting for qualifies; there is a section on them below.
3. Ask for less output. Set max_tokens to what a good answer needs, ask for a short reply in the prompt, and lower the effort setting on simple tasks: lower effort means less thinking, so fewer output tokens. In the table above this is the biggest line.
4. Use a smaller model where the eval says it holds. Claude Haiku 5.5 costs a small fraction of Opus 5.5 and is meant for high-volume tasks such as classification. Choosing a model per request is model routingModel RoutingSending each request to the cheapest model that handles it well, by rules, a classifier or a cascade, and checking with an eval that quality holds., and the next section but one shows how to decide it with numbers.

Sending less context, by retrieving only the passages a question needs (module 5), also cuts cost. Caching helps throughput too: for most Claude models, only uncached input tokens count toward the input tokens per minute limit, so with an 80% cache hit rate a limit of 2 million lets 10 million input tokens through each minute.

ToolPrompt caching calculatorYour repeated prompt and traffic in; the month with no cache, the 5-minute and the 1-hour cache out.

Cut the wait

2 times matter, and they have different fixes. Time to first tokenInference LatencyThe time a model takes to produce an answer, split into reading the prompt (prefill, which sets the time to first token) and writing the answer one token at a time (decode). is how long the model takes to start answering, mostly spent reading the prompt. The rest is the time it takes to write each token after that, multiplied by the number of tokens. Research on serving systems calls the first phase prefill and the second decode, and measures them separately for this reason.

Which one counts depends on who is waiting:

• A person reading the answer as it appears, as in a chat: stream it, so the first words show while the rest are written. Time to first token is what they feel, and a cached prompt shortens it.
• Code waiting for the whole answer, as with triage: total time is what counts, and it grows with the output. Fewer output tokens make it faster as well as cheaper.

Anthropic's guide to LatencyLatencyHow long a user waits for a model's answer, measured as the time to the first token and the time to the last, and judged by the slowest requests, not the average. lists the levers in its order: get the prompt working first, then choose the model (Haiku 5.5 is the fastest, with effort as its main control), send and ask for fewer tokens, set max_tokens, and stream. Measure the median and the 95th percentile, not the average. At the 95th percentile, 1 request in 20 waits at least that long, and that is where timeouts and complaints come from.

ConceptLatencyHow long a user waits for a model's answer, measured as the time to the first token and the time to the last, and judged by the slowest requests, not the…

When the API fails

Some calls will fail, and most failures pass if you wait. The Anthropic SDKs already retry connection errors, 429 and every status of 500 and above, twice by default, with exponential backoff, and follow the retry-after header when the API sends one. What is left for your code is to decide how long to wait and what happens after the retries.

4 decisions turn that into a design:

• Set your own timeout. The SDK's default is 10 minutes, far longer than a customer will wait. Choose what the feature can afford, and remember the retries come on top of it.
• Keep a fallback model that passes the same eval. A second model, such as Claude Sonnet 5.5 for Opus 5.5, keeps the feature running in an outage. Run module 7's eval on it before you rely on it, or the fallback ships answers you never tested.
• Make a person the last fallback. When both models fail, the email goes to the team's queue, the same place an email the model cannot sort goes.
• Catch errors by status code. In the Python SDK a 529 raises OverloadedError, and 503 and 504 have classes of their own too. None of them is a subclass of InternalServerError, so except anthropic.InternalServerError misses an overloaded API. Catch APIStatusError and test status_code, and then test the fallback by making the first model fail on purpose. The first version of production.py below had this bug, and only the outage test found it.

If your traffic grows sharply, the errors page warns that a sudden jump can meet acceleration limits and return 429s. Ramp up a launch gradually.

Route by eval

Routing sends each request to the cheapest model that will handle it. Research on learned routers, such as RouteLLM, cut costs by more than 2 times in some cases without losing quality. The rule you can start with is simpler: let a small model handle every request, and send the ones where a mistake costs the most to a larger one.

Do not decide it by reading a few answers. Run the eval from module 7 on each way of choosing the model and compare pass rate and cost side by side. route.py in the Build it part does that for 3 ways: all Opus 5.5, all Haiku 5.5, and a cascade where Haiku sorts every email and refunds, complaints or anything it cannot sort go to Opus as well.

The comparison usually shows something a rule hides. A cascade that decides by the small model's own answer cannot catch the cases where that answer is wrong: if Haiku calls a complaint "other", the complaint rule never sees it. Read each failure, then choose: send more to the large model, add cases like it to the eval, or accept it because a person reads that category anyway.

ConceptModel RoutingSending each request to the cheapest model that handles it well, by rules, a classifier or a cascade, and checking with an eval that quality holds.

Work no one is waiting for

Plenty of work has no one waiting: the emails that arrived overnight, a re-run of the eval, tags for last year's tickets. Send it with the Message Batches API. A batch costs 50% of the normal price for every request in it, holds up to 100,000 requests or 256 MB, and most batches end within an hour. Any request not finished within 24 hours expires, and the results stay available for 29 days.

3 things to get right:

• Give each request a custom_id, such as the email's id. Results can come back in any order, so the id is the only way to match them.
• Handle every result type. Each one is succeeded, errored, canceled or expired. Send the errored and expired ones again or to a person; a request rejected as invalid has to be fixed first.
• Reuse the same request code. The batch should send exactly what a live call sends, so it gets the same answers. Caching works in batches on a best-effort basis, and the 1-hour cache suits a batch whose requests share a long prompt.

ConceptBatch ProcessingSending many model requests together to be answered within hours instead of seconds, at a lower price; the right choice for any work no one is waiting on.

Watch it once it is live

A feature can fail loudly, with errors, or quietly, with answers that get worse, a cache that stopped working or a bill that doubled. You only see the quiet failures if every call leaves a record. Log 1 line per call, as JSON, with:

• the time, the item's id (here the email's), the prompt's name, version and hash from module 3, and the model
• the request id from the response, which Anthropic's support needs to look into a call
• the outcome: ok, the error's status, or no answer
• the time taken, and the tokens in, written to the cache, read from it, and out
• the cost, worked out from those tokens and the prices you pay

From those lines come the numbers to watch: failures by error, the median and 95th percentile time, the cost per request and the projected month against the budget, and the share of input read from the cache. report.py below prints them and stops a release if the projection is over budget.

Errors are the easy part. To catch answers getting worse, take a sample of live requests each week, grade them with module 7's eval, and turn each new kind of failure into a case. For money, set a spend limit on the feature's own workspace from module 0, so a bug cannot run up a bill without limit. An organization can also pull its usage and cost by model and workspace with Anthropic's Usage and Cost API, which needs an Admin API key and does not work for individual accounts. The Console's Usage and Cost pages show the same data.

ScenarioThe agent bill that grew 5 times in a monthA coding agent's monthly bill went from $1,200 to $6,700 while the number of tasks rose by a fifth. Find where the money went before you cut the wrong thing.

Build it: triage that runs in production

This wraps module 3's triage in what it needs to run unattended. The folder takes module 3's prompts and triage, module 7's eval and its cases, and 4 new files:

course/
├── prompts/ prompts.py triage.py    from module 3
├── evals.py cases.jsonl             from module 7
├── production.py                    a time limit, a fallback model and a log line per call
├── route.py                         3 ways to choose the model, compared on the eval
├── report.py                        the numbers to watch, from the log
└── backlog.py                       the night's emails as a batch

production.py is the version of triage you deploy. Every call has a 30-second limit and the SDK's 2 retries. If Opus 5.5 still fails with a server error or no connection, the email goes to Sonnet 5.5; if a request is refused, or both models fail, triage() returns None and the email goes to a person. Every call, working or not, writes 1 line to calls.jsonl:

"""Triage as it runs in production: a time limit, a fallback model, and one log line per call."""
import json
import logging
import time

import anthropic
from pydantic import ValidationError

from triage import CACHE_READ, PRICES, PROMPT, SCHEMA, Triage

PRIMARY, FALLBACK = "claude-opus-5-5", "claude-sonnet-5-5"   # both must pass the eval
PRICES = {**PRICES, "claude-sonnet-5-5": (2.00, 10.00)}       # dollars per million tokens, from the price page
CACHE_READ = {**CACHE_READ, "claude-sonnet-5-5": 0.05}

# A customer is waiting: give up on a call after 30 seconds, and retry a failure twice before falling back.
client = anthropic.Anthropic(timeout=30.0, max_retries=2)

log = logging.getLogger("calls")
log.setLevel(logging.INFO)
log.addHandler(logging.FileHandler("calls.jsonl"))
log.propagate = False


def cost_of(model: str, usage, batch: bool = False) -> float:
    input_price, output_price = PRICES[model]
    dollars = (usage.input_tokens * input_price
               + (usage.cache_creation_input_tokens or 0) * input_price * 1.25
               + (usage.cache_read_input_tokens or 0) * input_price * CACHE_READ[model]
               + usage.output_tokens * output_price) / 1_000_000
    return dollars / 2 if batch else dollars  # batches are billed at half price


def params(email: str, model: str) -> dict:
    """The request for one email, the same whether it is sent now or in a batch."""
    return {
        "model": model,
        "max_tokens": 4000,
        "system": [{"type": "text", "text": PROMPT.system, "cache_control": {"type": "ephemeral"}}],
        "messages": [{"role": "user", "content": PROMPT.render(email=email)}],
        "output_config": {"effort": "low", "format": {"type": "json_schema", "schema": SCHEMA}},
    }


def parse(response) -> Triage | None:
    if response.stop_reason != "end_turn":
        return None
    text = next(block.text for block in response.content if block.type == "text")
    try:
        return Triage.model_validate_json(text)
    except ValidationError:
        return None


def call(email: str, model: str, email_id: str) -> Triage | None:
    """One model call, logged whether it works or not."""
    record = {"at": time.strftime("%Y-%m-%dT%H:%M:%S"), "email": email_id, "prompt": PROMPT.id, "model": model}
    started = time.perf_counter()
    try:
        response = client.messages.create(**params(email, model))
    except anthropic.APIStatusError as error:
        log.info(json.dumps({**record, "outcome": f"error {error.status_code}", "request": error.request_id,
                             "seconds": round(time.perf_counter() - started, 3)}))
        raise
    except anthropic.APIConnectionError:  # includes timeouts
        log.info(json.dumps({**record, "outcome": "no connection", "seconds": round(time.perf_counter() - started, 3)}))
        raise
    result = parse(response)
    usage = response.usage
    log.info(json.dumps({
        **record, "request": response._request_id, "seconds": round(time.perf_counter() - started, 3),
        "outcome": "ok" if result else f"no triage ({response.stop_reason})",
        "input": usage.input_tokens, "cache_write": usage.cache_creation_input_tokens or 0,
        "cache_read": usage.cache_read_input_tokens or 0, "output": usage.output_tokens,
        "cost": round(cost_of(model, usage), 6),
    }))
    return result


def triage(email: str, email_id: str) -> Triage | None:
    """The triage of one email, from the primary model or, if it is failing, the fallback. None means a person sorts it."""
    for model in (PRIMARY, FALLBACK):
        try:
            return call(email, model, email_id)
        except anthropic.APIConnectionError:
            continue  # no answer in time, after the SDK's retries: try the next model
        except anthropic.APIStatusError as error:
            if error.status_code >= 500:
                continue  # a server error or 529 overloaded, after the retries: try the next model
            return None  # a request the API refuses, such as a bad key or a spend cap, fails on any model
    return None


if __name__ == "__main__":
    cases = [json.loads(line) for line in open("cases.jsonl")]
    for case in cases:
        result = triage(case["email"], case["id"])
        print(f"{case['id']:<18} {result.category if result else 'to a person'}")

Then compare 3 ways of choosing the model on the eval. route.py runs every case 3 times, like module 7, and counts a case as passed only if every run passes:

"""Compare 3 ways to choose the model on module 7's eval cases: all Opus, all Haiku, or Haiku first with Opus for the hard ones."""
import contextlib
import io
import json

from evals import TRIALS, grade
from triage import triage

ESCALATE = {"refund", "complaint"}  # the categories where a wrong reply costs the most


def cascade(email: str):
    """Haiku sorts every email; refunds, complaints and anything it cannot sort go to Opus as well."""
    result, cost = triage(email, "claude-haiku-5-5")
    if result is None or result.category in ESCALATE:
        result, more = triage(email, "claude-opus-5-5")
        cost += more
    return result, cost


STRATEGIES = {
    "all Opus": lambda email: triage(email, "claude-opus-5-5"),
    "all Haiku": lambda email: triage(email, "claude-haiku-5-5"),
    "cascade": cascade,
}

if __name__ == "__main__":
    cases = [json.loads(line) for line in open("cases.jsonl")]
    for name, choose in STRATEGIES.items():
        passed, spent, failed = 0, 0.0, []
        for case in cases:
            ok = True
            for _ in range(TRIALS):
                with contextlib.redirect_stdout(io.StringIO()):  # triage prints a line per call; not needed here
                    result, cost = choose(case["email"])
                    ok &= all(grade(case, result).values())
                spent += cost
            passed += ok
            failed += [] if ok else [case["id"]]
        print(f"{name:<10} {passed} of {len(cases)} cases pass every run, ${spent / (len(cases) * TRIALS):.4f} an email"
              f"   fails: {', '.join(failed) or 'none'}")

report.py reads the log and prints what a dashboard would show, by model, then the cost per email and the month's projection. Set the 2 numbers at the top to your traffic and budget. It exits with an error when the projection is over budget, so you can run it before each release:

"""Read calls.jsonl and print what a dashboard would show: volume, failures, speed, cost and caching, by model."""
import json
import statistics
import sys
from collections import defaultdict

EMAILS_A_DAY, BUDGET_A_MONTH = 2_000, 400.00  # the shop's numbers; set your own

calls = [json.loads(line) for line in open("calls.jsonl")]
by_model = defaultdict(list)
for record in calls:
    by_model[record["model"]].append(record)

print(f"{'model':<18} {'calls':>5} {'failed':>6} {'p50 s':>6} {'p95 s':>6} {'cost':>8} {'cached':>7}")
for model, records in by_model.items():
    ok = [r for r in records if r["outcome"] == "ok"]
    seconds = sorted(r["seconds"] for r in ok) or [0.0]
    p95 = statistics.quantiles(seconds, n=20)[-1] if len(seconds) > 1 else seconds[0]
    cached = sum(r["cache_read"] for r in ok) / max(1, sum(r["input"] + r["cache_write"] + r["cache_read"] for r in ok))
    print(f"{model:<18} {len(records):>5} {len(records) - len(ok):>6} {statistics.median(seconds):>6.2f} {p95:>6.2f} "
          f"${sum(r['cost'] for r in ok):>7.4f} {cached:>7.0%}")

handled = sum(r["outcome"] == "ok" for r in calls)
spent = sum(r.get("cost", 0) for r in calls)
month = spent / handled * EMAILS_A_DAY * 30
print(f"\n{handled} emails sorted, ${spent / handled:.4f} each; at {EMAILS_A_DAY:,} a day that is "
      f"${month:,.2f} a month against a budget of ${BUDGET_A_MONTH:,.2f}")
if month > BUDGET_A_MONTH:
    sys.exit("Over budget: look at the cost column before the next release.")

backlog.py sends the night's emails as 1 batch, using the same params() as a live call, waits for it to end, and reads the results back by id:

"""Triage the night's emails as a batch, at half price, and read the results back by id."""
import json
import time

from production import PRIMARY, client, cost_of, params, parse

emails = [json.loads(line) for line in open("cases.jsonl")]
batch = client.messages.batches.create(
    requests=[{"custom_id": email["id"], "params": params(email["email"], PRIMARY)} for email in emails]
)
print(f"{batch.id}: {batch.processing_status}")

while batch.processing_status != "ended":
    time.sleep(60)  # most batches end within an hour; results wait up to 24 hours
    batch = client.messages.batches.retrieve(batch.id)
print(f"{batch.id}: {batch.request_counts.succeeded} succeeded, {batch.request_counts.errored} errored, "
      f"{batch.request_counts.expired} expired")

spent = 0.0
for entry in client.messages.batches.results(batch.id):  # in any order: match on custom_id
    if entry.result.type != "succeeded":
        print(f"{entry.custom_id:<18} {entry.result.type}: send it again, or to a person")
        continue
    message = entry.result.message
    spent += cost_of(PRIMARY, message.usage, batch=True)
    result = parse(message)
    print(f"{entry.custom_id:<18} {result.category if result else 'to a person'}")
print(f"${spent:.4f} for {len(emails)} emails at batch prices")

Run them with your key set:

python route.py
python production.py
python report.py
python backlog.py

The output below comes from a stand-in for the API, not from Claude. It fails some eval cases on purpose so the comparison has something to show, it can make Opus 5.5 return 529 to test the fallback, and its times and token counts are its own. Your numbers will differ; the shape of each run is what to compare. First route.py:

all Opus   5 of 6 cases pass every run, $0.0036 an email   fails: happy-customer
all Haiku  4 of 6 cases pass every run, $0.0001 an email   fails: rude-courier, refund-injection
cascade    5 of 6 cases pass every run, $0.0012 an email   fails: rude-courier

Then production.py twice, the second time with every Opus call returning 529, then report.py:

model              calls failed  p50 s  p95 s     cost  cached
claude-opus-5-5       12      6   0.19   0.22 $ 0.0206     95%
claude-sonnet-5-5      6      0   0.19   0.22 $ 0.0117     79%

12 emails sorted, $0.0027 each; at 2,000 a day that is $161.84 a month against a budget of $400.00

2 lines from calls.jsonl during the outage show what happened to 1 email: Opus 5.5 failed after the SDK's retries, and Sonnet 5.5 sorted it, writing its own copy of the prompt to the cache for the emails after it:

{"at": "2026-10-17T15:20:07", "email": "late-mixer", "prompt": "triage@v2 (a46a857d)", "model": "claude-opus-5-5", "outcome": "error 529", "request": "req_test0124", "seconds": 1.309}
{"at": "2026-10-17T15:20:08", "email": "late-mixer", "prompt": "triage@v2 (a46a857d)", "model": "claude-sonnet-5-5", "request": "req_test0125", "seconds": 0.188, "outcome": "ok", "input": 33, "cache_write": 592, "cache_read": 0, "output": 160, "cost": 0.003146}

And backlog.py, with its results in a different order from the cases:

msgbatch_test001: in_progress
msgbatch_test001: 6 succeeded, 0 errored, 0 expired
refund-injection   refund
rude-courier       complaint
happy-customer     other
induction-pan      product_question
scratched-bowl     refund
late-mixer         delivery
$0.0103 for 6 emails at batch prices

What to read in it:

• The cascade costs a third as much as all Opus and passes as many cases, but not the same ones. It fails rude-courier, the case the small model gets wrong, which the rule cannot catch. That is the decision to make with the failed case in front of you.
• The outage cost time, not emails. Each 529 was retried twice before the fallback ran, which is why the failed Opus calls took over a second each in the log, and every email was still sorted. Sonnet 5.5's line shows it served the 6 emails at half Opus's price, with its own cache.
• The projection is the number to watch. At this traffic it sits well under the $400 budget. If a prompt change doubled the output, it would show here before the bill did.
• The batch costs about half as much as the same emails sent live, and its results come back unordered, matched by custom_id.

Now break it on purpose. Change the except in triage() back to anthropic.InternalServerError and run the outage again: every email goes to a person. Set timeout to 0.05 seconds and watch the calls fail with no connection. Then add a case to cases.jsonl that the cascade gets wrong and run route.py again. Module 11 puts this triage, the agent and the guard together, and runs the whole eval and the budget check before the assistant ships.