How do you cut the cost of an LLM application at scale?

Find which calls cost the most, then attack the biggest line first. Cache repeated prompt prefixes, batch anything that can wait, route easy requests to a cheaper model, send fewer tokens in and out, and put limits on every user and feature so no single path can run away.

1. Measure cost per feature

Log input, output and cached tokens, the model and the feature for every call. Usually a few prompts make up most of the bill, and often input tokens dominate (long system prompts, big retrieved contexts, chat history resent each turn).

2. Prompt caching

If many requests share a long prefix (instructions, tool definitions, a document), mark it for caching. On the Claude API a cache read costs 0.1 times the normal input price on most models. Writing costs 1.25 times (5-minute lifetime) or 2 times (1-hour lifetime), so caching pays off once a prefix is reused even once or twice. Keep the cached part stable and put anything that varies (the user's question, timestamps) after it.

3. Batch what can wait

Nightly classification, document summaries and evaluations do not need an answer in seconds. Anthropic's Message Batches API halves the price, and most batches finish within an hour. Other providers offer similar discounts.

4. Route by difficulty

Send classification, extraction and simple questions to a small model, and only the hard ones to a large model. Route by rules (request type) or with a cheap classifier, and confirm on your evaluation set that quality holds for each route.

5. Fewer tokens

• Retrieve fewer, better chunks (reranking helps here).
• Summarise or trim long chat histories.
• Ask for shorter outputs, or structured output instead of prose.
• Remove instructions the model no longer needs. Long prompts tend to grow by accretion.

6. Do not call the model at all

Cache whole answers for repeated questions (exact match, or carefully thresholded semantic match). Use plain code for what does not need a model (regex, lookups, rules).

7. Limits

OWASP lists unbounded consumption among its top 10 LLM risks: per-user rate limits, quotas, input size limits and per-request max_tokens, with alerts on spend. One abusive user or one looping agent should not be able to multiply the bill.

Judge by cost per successful task

A cheaper model that fails more often, triggering retries or human escalations, can cost more overall. Compare options on the same evaluation set, by cost per correct result.

A worked example to offer: "A support bot resent a 6,000-token policy on every turn. Caching it and moving intent classification to a small model cut our input cost by about two thirds with no change in answer quality on our test set." Use your own numbers if you have them.