# GenAI in production: LLM cost, latency, fallbacks, batching and monitoring Price a feature from its token counts before you build it, then cut the cost in order of effort: cache the fixed part of the prompt, send work that can wait as a batch at half price, cap and shorten the output, and move traffic to a smaller model only where your eval says it holds. Give every call a time limit, let the SDK retry the errors that pass, fall back to a second model that also passes the eval and then to a person, and log every call so you can see failures, speed, cost per request and the month's projected bill. - Page: https://tecyfy.com/ai-engineering/production - Course: AI engineering course (https://tecyfy.com/ai-engineering), Module 9: In production: cost, speed and reliability At the end of this module you can: - Estimate what a feature costs a month from its tokens and the price page, before writing it - Cut cost with caching, batches, shorter output and a smaller model where the eval allows it - Tell time to first token from total time, and know which one your feature needs - Handle rate limits, timeouts and overloaded errors with retries, a fallback model and a person - Log every call and turn the log into the numbers you watch, with a budget check The triage from modules 2 and 3 works on your laptop. In production it meets things a test run never shows: a bill that grows with every customer, a customer who waits, an API that is sometimes busy, and answers that slowly get worse with no error at all. This module adds what a GenAI feature needs to run for real: a budget, the cheapest way to meet it, a time limit, a fallback, and a log you can read. ## Budget it before you build it An LLM feature has a price per request you can work out before writing any code. Count the tokens each request sends and receives, multiply by the prices on the [price page](/prices), and multiply by the traffic you expect. For the shop's triage, using the token counts from the stand-in's log in the Build it part: a system prompt of about 600 tokens, cached after the first email, about 30 tokens of email and 160 tokens of output. On [Claude Opus 5.5](https://tecyfy.com/prices/claude-opus-5-5), at $4 per million input tokens and $20 per million output: | Part | Tokens | Price | Cost per email | |---|---|---|---| | Cached system prompt | 592 | 5% of $4 per million | $0.00012 | | Email | 33 | $4 per million | $0.00013 | | Output | 160 | $20 per million | $0.0032 | | **Total** | | | **about $0.0034** | At 2,000 emails a day that is about $207 a month. The table also shows where the money goes: output is more than 90% of it. That is common for short tasks, because output costs 5 times as much per token as input on this model, and it tells you which change to try first. Write the estimate down with the traffic you assumed, and compare it with the real bill once the feature is live. - Tool: [Cost calculator](https://tecyfy.com/prices/cost-calculator). What your job costs, a day, a month or a year ## Cut the cost, in order of effort Try these in order. The first 2 cannot change an answer; the last 2 can, so they need the eval from module 7. 1. **Cache the part that repeats.** Module 3 put the fixed instructions first and marked them for [caching](https://tecyfy.com/concepts/prompt-caching). On Opus 5.5 and Sonnet 5.5 a cache read costs 5% of the input price. Check `cache_read_input_tokens` in the usage figures: 0 on every call means the cache is not working. 2. **Send what can wait as a batch.** [Batches](https://tecyfy.com/concepts/batch-processing) cost half the normal price. Anything no one is waiting for qualifies; there is a section on them below. 3. **Ask for less output.** Set `max_tokens` to what a good answer needs, ask for a short reply in the prompt, and lower the effort setting on simple tasks: lower effort means less thinking, so fewer output tokens. In the table above this is the biggest line. 4. **Use a smaller model where the eval says it holds.** [Claude Haiku 5.5](https://tecyfy.com/prices/claude-haiku-5-5) costs a small fraction of Opus 5.5 and is meant for high-volume tasks such as classification. Choosing a model per request is [model routing](https://tecyfy.com/concepts/model-routing), and the next section but one shows how to decide it with numbers. Sending less context, by retrieving only the passages a question needs (module 5), also cuts cost. Caching helps throughput too: for most Claude models, only uncached input tokens count toward the input tokens per minute limit, so with an 80% cache hit rate a limit of 2 million lets 10 million input tokens through each minute. - Tool: [Prompt caching calculator](https://tecyfy.com/ai-agents/prompt-caching-calculator). Your repeated prompt and traffic in; the month with no cache, the 5-minute and the 1-hour cache out. ## Cut the wait 2 times matter, and they have different fixes. [Time to first token](https://tecyfy.com/concepts/inference-latency) is how long the model takes to start answering, mostly spent reading the prompt. The rest is the time it takes to write each token after that, multiplied by the number of tokens. Research on serving systems calls the first phase prefill and the second decode, and measures them separately for this reason. Which one counts depends on who is waiting: - **A person reading the answer as it appears**, as in a chat: stream it, so the first words show while the rest are written. Time to first token is what they feel, and a cached prompt shortens it. - **Code waiting for the whole answer**, as with triage: total time is what counts, and it grows with the output. Fewer output tokens make it faster as well as cheaper. Anthropic's guide to [Latency](https://tecyfy.com/concepts/latency) lists the levers in its order: get the prompt working first, then choose the model (Haiku 5.5 is the fastest, with effort as its main control), send and ask for fewer tokens, set `max_tokens`, and stream. Measure the median and the 95th percentile, not the average. At the 95th percentile, 1 request in 20 waits at least that long, and that is where timeouts and complaints come from. - Concept: [Latency](https://tecyfy.com/concepts/latency). How long a user waits for a model's answer, measured as the time to the first token and the time to the last, and judged by the slowest requests, not the… ## When the API fails Some calls will fail, and most failures pass if you wait. The Anthropic SDKs already retry connection errors, 429 and every status of 500 and above, twice by default, with exponential backoff, and follow the `retry-after` header when the API sends one. What is left for your code is to decide how long to wait and what happens after the retries. | Error | What it means | What to do after the SDK's retries | |---|---|---| | 429 `rate_limit_error` | Over a requests or tokens per minute limit | Send less at once, cache more, or batch; alert if it keeps happening | | 429 with no `retry-after` | The monthly spend cap was reached | Stop and alert a person; no retry helps | | 500 `api_error` | Something failed on Anthropic's side | Try the fallback model | | 504 `timeout_error` | The request timed out while it was processed | Stream it, send it as a batch, or ask for less output | | 529 `overloaded_error` | The API is busy across all users | Try the fallback model | | 400, 401, 403 | The request, the key or its permissions are wrong | Fix the code or the key; it fails on any model | 4 decisions turn that into a design: - **Set your own timeout.** The SDK's default is 10 minutes, far longer than a customer will wait. Choose what the feature can afford, and remember the retries come on top of it. - **Keep a fallback model that passes the same eval.** A second model, such as [Claude Sonnet 5.5](https://tecyfy.com/prices/claude-sonnet-5-5) for Opus 5.5, keeps the feature running in an outage. Run module 7's eval on it before you rely on it, or the fallback ships answers you never tested. - **Make a person the last fallback.** When both models fail, the email goes to the team's queue, the same place an email the model cannot sort goes. - **Catch errors by status code.** In the Python SDK a 529 raises `OverloadedError`, and 503 and 504 have classes of their own too. None of them is a subclass of `InternalServerError`, so `except anthropic.InternalServerError` misses an overloaded API. Catch `APIStatusError` and test `status_code`, and then test the fallback by making the first model fail on purpose. The first version of `production.py` below had this bug, and only the outage test found it. If your traffic grows sharply, the errors page warns that a sudden jump can meet acceleration limits and return 429s. Ramp up a launch gradually. ## Route by eval Routing sends each request to the cheapest model that will handle it. Research on learned routers, such as RouteLLM, cut costs by more than 2 times in some cases without losing quality. The rule you can start with is simpler: let a small model handle every request, and send the ones where a mistake costs the most to a larger one. Do not decide it by reading a few answers. Run the eval from module 7 on each way of choosing the model and compare pass rate and cost side by side. `route.py` in the Build it part does that for 3 ways: all Opus 5.5, all Haiku 5.5, and a cascade where Haiku sorts every email and refunds, complaints or anything it cannot sort go to Opus as well. The comparison usually shows something a rule hides. A cascade that decides by the small model's own answer cannot catch the cases where that answer is wrong: if Haiku calls a complaint "other", the complaint rule never sees it. Read each failure, then choose: send more to the large model, add cases like it to the eval, or accept it because a person reads that category anyway. - Concept: [Model Routing](https://tecyfy.com/concepts/model-routing). Sending each request to the cheapest model that handles it well, by rules, a classifier or a cascade, and checking with an eval that quality holds. ## Work no one is waiting for Plenty of work has no one waiting: the emails that arrived overnight, a re-run of the eval, tags for last year's tickets. Send it with the Message Batches API. A batch costs 50% of the normal price for every request in it, holds up to 100,000 requests or 256 MB, and most batches end within an hour. Any request not finished within 24 hours expires, and the results stay available for 29 days. 3 things to get right: - **Give each request a `custom_id`**, such as the email's id. Results can come back in any order, so the id is the only way to match them. - **Handle every result type.** Each one is `succeeded`, `errored`, `canceled` or `expired`. Send the errored and expired ones again or to a person; a request rejected as invalid has to be fixed first. - **Reuse the same request code.** The batch should send exactly what a live call sends, so it gets the same answers. Caching works in batches on a best-effort basis, and the 1-hour cache suits a batch whose requests share a long prompt. - Concept: [Batch Processing](https://tecyfy.com/concepts/batch-processing). Sending many model requests together to be answered within hours instead of seconds, at a lower price; the right choice for any work no one is waiting on. ## Watch it once it is live A feature can fail loudly, with errors, or quietly, with answers that get worse, a cache that stopped working or a bill that doubled. You only see the quiet failures if every call leaves a record. Log 1 line per call, as JSON, with: - the time, the item's id (here the email's), the prompt's name, version and hash from module 3, and the model - the request id from the response, which Anthropic's support needs to look into a call - the outcome: `ok`, the error's status, or no answer - the time taken, and the tokens in, written to the cache, read from it, and out - the cost, worked out from those tokens and the prices you pay From those lines come the numbers to watch: failures by error, the median and 95th percentile time, the cost per request and the projected month against the budget, and the share of input read from the cache. `report.py` below prints them and stops a release if the projection is over budget. Errors are the easy part. To catch answers getting worse, take a sample of live requests each week, grade them with module 7's eval, and turn each new kind of failure into a case. For money, set a spend limit on the feature's own workspace from module 0, so a bug cannot run up a bill without limit. An organization can also pull its usage and cost by model and workspace with Anthropic's Usage and Cost API, which needs an Admin API key and does not work for individual accounts. The Console's Usage and Cost pages show the same data. - Scenario: [The agent bill that grew 5 times in a month](https://tecyfy.com/practice/scenarios/the-agent-bill-that-grew-5-times). A coding agent's monthly bill went from $1,200 to $6,700 while the number of tasks rose by a fifth. Find where the money went before you cut the wrong thing. ## Build it: triage that runs in production This wraps module 3's triage in what it needs to run unattended. The folder takes module 3's prompts and triage, module 7's eval and its cases, and 4 new files: ```text course/ ├── prompts/ prompts.py triage.py from module 3 ├── evals.py cases.jsonl from module 7 ├── production.py a time limit, a fallback model and a log line per call ├── route.py 3 ways to choose the model, compared on the eval ├── report.py the numbers to watch, from the log └── backlog.py the night's emails as a batch ``` `production.py` is the version of triage you deploy. Every call has a 30-second limit and the SDK's 2 retries. If Opus 5.5 still fails with a server error or no connection, the email goes to Sonnet 5.5; if a request is refused, or both models fail, `triage()` returns `None` and the email goes to a person. Every call, working or not, writes 1 line to `calls.jsonl`: ```python """Triage as it runs in production: a time limit, a fallback model, and one log line per call.""" import json import logging import time import anthropic from pydantic import ValidationError from triage import CACHE_READ, PRICES, PROMPT, SCHEMA, Triage PRIMARY, FALLBACK = "claude-opus-5-5", "claude-sonnet-5-5" # both must pass the eval PRICES = {**PRICES, "claude-sonnet-5-5": (2.00, 10.00)} # dollars per million tokens, from the price page CACHE_READ = {**CACHE_READ, "claude-sonnet-5-5": 0.05} # A customer is waiting: give up on a call after 30 seconds, and retry a failure twice before falling back. client = anthropic.Anthropic(timeout=30.0, max_retries=2) log = logging.getLogger("calls") log.setLevel(logging.INFO) log.addHandler(logging.FileHandler("calls.jsonl")) log.propagate = False def cost_of(model: str, usage, batch: bool = False) -> float: input_price, output_price = PRICES[model] dollars = (usage.input_tokens * input_price + (usage.cache_creation_input_tokens or 0) * input_price * 1.25 + (usage.cache_read_input_tokens or 0) * input_price * CACHE_READ[model] + usage.output_tokens * output_price) / 1_000_000 return dollars / 2 if batch else dollars # batches are billed at half price def params(email: str, model: str) -> dict: """The request for one email, the same whether it is sent now or in a batch.""" return { "model": model, "max_tokens": 4000, "system": [{"type": "text", "text": PROMPT.system, "cache_control": {"type": "ephemeral"}}], "messages": [{"role": "user", "content": PROMPT.render(email=email)}], "output_config": {"effort": "low", "format": {"type": "json_schema", "schema": SCHEMA}}, } def parse(response) -> Triage | None: if response.stop_reason != "end_turn": return None text = next(block.text for block in response.content if block.type == "text") try: return Triage.model_validate_json(text) except ValidationError: return None def call(email: str, model: str, email_id: str) -> Triage | None: """One model call, logged whether it works or not.""" record = {"at": time.strftime("%Y-%m-%dT%H:%M:%S"), "email": email_id, "prompt": PROMPT.id, "model": model} started = time.perf_counter() try: response = client.messages.create(**params(email, model)) except anthropic.APIStatusError as error: log.info(json.dumps({**record, "outcome": f"error {error.status_code}", "request": error.request_id, "seconds": round(time.perf_counter() - started, 3)})) raise except anthropic.APIConnectionError: # includes timeouts log.info(json.dumps({**record, "outcome": "no connection", "seconds": round(time.perf_counter() - started, 3)})) raise result = parse(response) usage = response.usage log.info(json.dumps({ **record, "request": response._request_id, "seconds": round(time.perf_counter() - started, 3), "outcome": "ok" if result else f"no triage ({response.stop_reason})", "input": usage.input_tokens, "cache_write": usage.cache_creation_input_tokens or 0, "cache_read": usage.cache_read_input_tokens or 0, "output": usage.output_tokens, "cost": round(cost_of(model, usage), 6), })) return result def triage(email: str, email_id: str) -> Triage | None: """The triage of one email, from the primary model or, if it is failing, the fallback. None means a person sorts it.""" for model in (PRIMARY, FALLBACK): try: return call(email, model, email_id) except anthropic.APIConnectionError: continue # no answer in time, after the SDK's retries: try the next model except anthropic.APIStatusError as error: if error.status_code >= 500: continue # a server error or 529 overloaded, after the retries: try the next model return None # a request the API refuses, such as a bad key or a spend cap, fails on any model return None if __name__ == "__main__": cases = [json.loads(line) for line in open("cases.jsonl")] for case in cases: result = triage(case["email"], case["id"]) print(f"{case['id']:<18} {result.category if result else 'to a person'}") ``` Then compare 3 ways of choosing the model on the eval. `route.py` runs every case 3 times, like module 7, and counts a case as passed only if every run passes: ```python """Compare 3 ways to choose the model on module 7's eval cases: all Opus, all Haiku, or Haiku first with Opus for the hard ones.""" import contextlib import io import json from evals import TRIALS, grade from triage import triage ESCALATE = {"refund", "complaint"} # the categories where a wrong reply costs the most def cascade(email: str): """Haiku sorts every email; refunds, complaints and anything it cannot sort go to Opus as well.""" result, cost = triage(email, "claude-haiku-5-5") if result is None or result.category in ESCALATE: result, more = triage(email, "claude-opus-5-5") cost += more return result, cost STRATEGIES = { "all Opus": lambda email: triage(email, "claude-opus-5-5"), "all Haiku": lambda email: triage(email, "claude-haiku-5-5"), "cascade": cascade, } if __name__ == "__main__": cases = [json.loads(line) for line in open("cases.jsonl")] for name, choose in STRATEGIES.items(): passed, spent, failed = 0, 0.0, [] for case in cases: ok = True for _ in range(TRIALS): with contextlib.redirect_stdout(io.StringIO()): # triage prints a line per call; not needed here result, cost = choose(case["email"]) ok &= all(grade(case, result).values()) spent += cost passed += ok failed += [] if ok else [case["id"]] print(f"{name:<10} {passed} of {len(cases)} cases pass every run, ${spent / (len(cases) * TRIALS):.4f} an email" f" fails: {', '.join(failed) or 'none'}") ``` `report.py` reads the log and prints what a dashboard would show, by model, then the cost per email and the month's projection. Set the 2 numbers at the top to your traffic and budget. It exits with an error when the projection is over budget, so you can run it before each release: ```python """Read calls.jsonl and print what a dashboard would show: volume, failures, speed, cost and caching, by model.""" import json import statistics import sys from collections import defaultdict EMAILS_A_DAY, BUDGET_A_MONTH = 2_000, 400.00 # the shop's numbers; set your own calls = [json.loads(line) for line in open("calls.jsonl")] by_model = defaultdict(list) for record in calls: by_model[record["model"]].append(record) print(f"{'model':<18} {'calls':>5} {'failed':>6} {'p50 s':>6} {'p95 s':>6} {'cost':>8} {'cached':>7}") for model, records in by_model.items(): ok = [r for r in records if r["outcome"] == "ok"] seconds = sorted(r["seconds"] for r in ok) or [0.0] p95 = statistics.quantiles(seconds, n=20)[-1] if len(seconds) > 1 else seconds[0] cached = sum(r["cache_read"] for r in ok) / max(1, sum(r["input"] + r["cache_write"] + r["cache_read"] for r in ok)) print(f"{model:<18} {len(records):>5} {len(records) - len(ok):>6} {statistics.median(seconds):>6.2f} {p95:>6.2f} " f"${sum(r['cost'] for r in ok):>7.4f} {cached:>7.0%}") handled = sum(r["outcome"] == "ok" for r in calls) spent = sum(r.get("cost", 0) for r in calls) month = spent / handled * EMAILS_A_DAY * 30 print(f"\n{handled} emails sorted, ${spent / handled:.4f} each; at {EMAILS_A_DAY:,} a day that is " f"${month:,.2f} a month against a budget of ${BUDGET_A_MONTH:,.2f}") if month > BUDGET_A_MONTH: sys.exit("Over budget: look at the cost column before the next release.") ``` `backlog.py` sends the night's emails as 1 batch, using the same `params()` as a live call, waits for it to end, and reads the results back by id: ```python """Triage the night's emails as a batch, at half price, and read the results back by id.""" import json import time from production import PRIMARY, client, cost_of, params, parse emails = [json.loads(line) for line in open("cases.jsonl")] batch = client.messages.batches.create( requests=[{"custom_id": email["id"], "params": params(email["email"], PRIMARY)} for email in emails] ) print(f"{batch.id}: {batch.processing_status}") while batch.processing_status != "ended": time.sleep(60) # most batches end within an hour; results wait up to 24 hours batch = client.messages.batches.retrieve(batch.id) print(f"{batch.id}: {batch.request_counts.succeeded} succeeded, {batch.request_counts.errored} errored, " f"{batch.request_counts.expired} expired") spent = 0.0 for entry in client.messages.batches.results(batch.id): # in any order: match on custom_id if entry.result.type != "succeeded": print(f"{entry.custom_id:<18} {entry.result.type}: send it again, or to a person") continue message = entry.result.message spent += cost_of(PRIMARY, message.usage, batch=True) result = parse(message) print(f"{entry.custom_id:<18} {result.category if result else 'to a person'}") print(f"${spent:.4f} for {len(emails)} emails at batch prices") ``` Run them with your key set: ```bash python route.py python production.py python report.py python backlog.py ``` The output below comes from a stand-in for the API, not from Claude. It fails some eval cases on purpose so the comparison has something to show, it can make Opus 5.5 return 529 to test the fallback, and its times and token counts are its own. Your numbers will differ; the shape of each run is what to compare. First `route.py`: ```text all Opus 5 of 6 cases pass every run, $0.0036 an email fails: happy-customer all Haiku 4 of 6 cases pass every run, $0.0001 an email fails: rude-courier, refund-injection cascade 5 of 6 cases pass every run, $0.0012 an email fails: rude-courier ``` Then `production.py` twice, the second time with every Opus call returning 529, then `report.py`: ```text model calls failed p50 s p95 s cost cached claude-opus-5-5 12 6 0.19 0.22 $ 0.0206 95% claude-sonnet-5-5 6 0 0.19 0.22 $ 0.0117 79% 12 emails sorted, $0.0027 each; at 2,000 a day that is $161.84 a month against a budget of $400.00 ``` 2 lines from `calls.jsonl` during the outage show what happened to 1 email: Opus 5.5 failed after the SDK's retries, and Sonnet 5.5 sorted it, writing its own copy of the prompt to the cache for the emails after it: ```json {"at": "2026-10-17T15:20:07", "email": "late-mixer", "prompt": "triage@v2 (a46a857d)", "model": "claude-opus-5-5", "outcome": "error 529", "request": "req_test0124", "seconds": 1.309} {"at": "2026-10-17T15:20:08", "email": "late-mixer", "prompt": "triage@v2 (a46a857d)", "model": "claude-sonnet-5-5", "request": "req_test0125", "seconds": 0.188, "outcome": "ok", "input": 33, "cache_write": 592, "cache_read": 0, "output": 160, "cost": 0.003146} ``` And `backlog.py`, with its results in a different order from the cases: ```text msgbatch_test001: in_progress msgbatch_test001: 6 succeeded, 0 errored, 0 expired refund-injection refund rude-courier complaint happy-customer other induction-pan product_question scratched-bowl refund late-mixer delivery $0.0103 for 6 emails at batch prices ``` What to read in it: - **The cascade costs a third as much as all Opus and passes as many cases**, but not the same ones. It fails rude-courier, the case the small model gets wrong, which the rule cannot catch. That is the decision to make with the failed case in front of you. - **The outage cost time, not emails.** Each 529 was retried twice before the fallback ran, which is why the failed Opus calls took over a second each in the log, and every email was still sorted. Sonnet 5.5's line shows it served the 6 emails at half Opus's price, with its own cache. - **The projection is the number to watch.** At this traffic it sits well under the $400 budget. If a prompt change doubled the output, it would show here before the bill did. - **The batch costs about half as much as the same emails sent live**, and its results come back unordered, matched by `custom_id`. Now break it on purpose. Change the `except` in `triage()` back to `anthropic.InternalServerError` and run the outage again: every email goes to a person. Set `timeout` to 0.05 seconds and watch the calls fail with no connection. Then add a case to `cases.jsonl` that the cascade gets wrong and run `route.py` again. Module 11 puts this triage, the agent and the guard together, and runs the whole eval and the budget check before the assistant ships. ## Check yourself 1. Your code sends a failed call to a second model with `except anthropic.InternalServerError`. During a busy hour the API returns 529 overloaded for Opus 5.5, and every customer email fails instead of falling back. Why? A. The SDK does not retry a 529, so the error arrives too early B. In the Python SDK a 529 raises OverloadedError, which is not a subclass of InternalServerError, so the except clause never matches C. A 529 is a rate limit, and rate limits cannot fall back D. The fallback model is overloaded too, because all models share one capacity 2. Every night 40,000 emails that arrived after hours need a triage by 8 in the morning, and no one reads them before then. Which is the cheapest sound way to run them? A. Send them as a batch with the Message Batches API, each request with its email's id as custom_id, and match the results on that id B. Send them individually as fast as the rate limits allow, so they finish sooner C. Send them with streaming, so each one starts sooner D. Switch to the smallest model for the night without running the eval 3. Your router lets Claude Haiku 5.5 sort every email and sends only refunds and complaints on to Opus 5.5 as well. It costs a third as much and passes 5 of the 6 eval cases, but it fails a complaint that all-Opus passes. What is the most likely cause? A. Opus 5.5 is not good at complaints B. The eval is broken, because a cheaper setup that passes 5 of 6 cases must be fine C. Haiku labelled the complaint as something else, so the rule that sends complaints to Opus never saw it D. The cache was cold for that email ### Answers 1. B. The SDK gives 529, 503 and 504 their own error classes beside InternalServerError, all under APIStatusError. Code that names only InternalServerError lets them through. Catch APIStatusError and test its status_code (500 and above means try the next model), and test the fallback by making the primary fail on purpose, as the stand-in's outage mode does. 2. A. Work no one is waiting for is what batches are for. A batch costs half the normal price, and most end within an hour, with 24 hours as the limit. Results can come back in any order, so the custom_id is how each one finds its email. Streaming changes when the first token arrives, not the price, and a smaller model is a quality decision for the eval, not a scheduling one. 3. C. A router that decides by the small model's own answer cannot catch the cases where that answer is wrong. Look at the failed case, then decide with numbers. Send more categories to Opus, add cases like it to the eval, or accept the miss if a person reads every complaint anyway. Whichever you choose, run the eval again on the router as a whole. ## Questions ### How do you reduce LLM API costs? Start with the changes that do not touch quality: cache the part of the prompt that repeats, which costs 5% of the input price to read on Claude Opus 5.5, and send work that can wait as a batch at half price. Then cap and shorten the output, which is usually the most expensive part, and lower the effort setting where the task is simple. Move traffic to a smaller model last, and only where your eval shows it holds. ### How do you handle rate limits and overloaded errors from an LLM API? Let the SDK retry: the Anthropic SDKs retry connection errors, 429 and 500 and above twice by default with exponential backoff, and follow the retry-after header. Set a timeout that suits the person waiting. After the retries, send server errors and 529 overloaded to a fallback model that also passes your eval, and then to a person. Errors such as a bad key or a spend cap fail on any model, so stop and alert instead. ### When should I use the Batch API? Use it for work no one is waiting for: nightly backlogs, re-running an eval, tagging old records, summaries for a report. A batch costs 50% of the normal price, holds up to 100,000 requests or 256 MB, and most end within an hour, with 24 hours as the limit. Give each request a custom_id and match results on it, because they come back in any order. ### What should you monitor for an LLM application in production? Log 1 line per call with the model, prompt version, request id, outcome, time taken, tokens in, cached and out, and cost. From those lines watch the failure rate by error, the median and 95th percentile time, the cost per request and the projected month against the budget, and the cached share. Grade a sample of live answers with your eval as well, because a call can succeed and still be wrong. ### How do you reduce the latency of an LLM call? Get the prompt right first, then choose the fastest model that passes your eval, ask for shorter output and set max_tokens, lower the effort setting for simple tasks, and cache long repeated context, which also shortens the time to the first token. For anything a person watches, stream the answer so the first words appear while the rest are written. ## Sources - [Reducing latency](https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/reduce-latency) - [Glossary](https://platform.claude.com/docs/en/about-claude/glossary) - [Errors](https://platform.claude.com/docs/en/api/errors) - [Python SDK](https://platform.claude.com/docs/en/cli-sdks-libraries/sdks/python) - [Anthropic Python SDK, _exceptions.py and _client.py](https://github.com/anthropics/anthropic-sdk-python/blob/main/src/anthropic/_exceptions.py) - [Rate limits](https://platform.claude.com/docs/en/api/rate-limits) - [Batch processing](https://platform.claude.com/docs/en/build-with-claude/batch-processing) - [Prompt caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) - [Pricing](https://platform.claude.com/docs/en/about-claude/pricing) - [Models overview](https://platform.claude.com/docs/en/about-claude/models/overview) - [Effort](https://platform.claude.com/docs/en/build-with-claude/effort) - [Usage and Cost API](https://platform.claude.com/docs/en/manage-claude/usage-cost-api) - [RouteLLM: Learning to Route LLMs with Preference Data](https://arxiv.org/abs/2406.18665) - [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://arxiv.org/abs/2401.09670)