# What is a batch API for LLMs, and when should you use it?

Sending many model requests together to be answered within hours instead of seconds, at a lower price; the right choice for any work no one is waiting on.

- Page: https://tecyfy.com/concepts/batch-processing
- Subject: How AI works (https://tecyfy.com/concepts)
- Level: beginner
- Also called: Batch API, Message Batches, Batch Inference

Not every model call has someone waiting for it. Overnight email triage, tagging last month's tickets, running an [eval set](https://tecyfy.com/concepts/eval-set), summarising a library of documents: all of it can wait an hour. A batch API takes those requests in one go and answers them when it can, for less money.

## How Claude's batch API works

You send up to 100,000 requests, or 256 MB, in one batch. Each is an ordinary Messages request with a `custom_id` you choose, such as the email's id. The batch starts as `in_progress` and becomes `ended` when every request is done. You poll for that, then read the results.

- **Price:** every token is billed at 50% of the standard price.
- **Speed:** most batches finish within an hour. Anything not done within 24 hours expires.
- **Results:** kept for 29 days, and returned in any order, so match each one to its request by `custom_id`, never by position.
- **Failures:** each result is `succeeded`, `errored`, `canceled` or `expired`. An invalid request must be fixed before you send it again; a server error or an expired request can simply be sent again.

[Prompt caching](https://tecyfy.com/concepts/prompt-caching) still works, but only on a best-effort basis, since requests in a batch run in any order and at different times. When every request shares a long prefix, Anthropic suggests the 1-hour cache.

## When to use it

Use a batch whenever the answer is not needed in the next few minutes. Half price is the biggest saving on the bill that needs no change to the prompt, and batches do not compete with your live traffic for [rate limits](https://tecyfy.com/concepts/rate-limit) in the same way.

Do not use it for anything a person is waiting on, and do not assume a batch is quick because the last one was: the 1-hour figure is typical, not promised. Check each request's result type, and send the errored and expired ones again or to a person.

## Related

- [Throughput](https://tecyfy.com/concepts/throughput): How much work a model system gets through in a given time, counted in requests or tokens per minute; on an API it is capped by rate limits, on your own hardware by batching and memory.
- [Prompt Caching](https://tecyfy.com/concepts/prompt-caching): Letting the API keep the processed start of a prompt for a few minutes, so later requests that begin the same way are cheaper and quicker.
- [Model Routing](https://tecyfy.com/concepts/model-routing): Sending each request to the cheapest model that handles it well, by rules, a classifier or a cascade, and checking with an eval that quality holds.
- [Rate Limit](https://tecyfy.com/concepts/rate-limit): The most an API lets you send in a period, counted in requests and tokens per minute. Past it, requests fail with a 429 error until capacity comes back, so code waits and retries.
- [Eval Set](https://tecyfy.com/concepts/eval-set): The list of test cases an LLM app is graded on after every change. Each case is an input and what a good answer must do, so a score going up or down shows whether the change helped.

## Sources

- [Batch processing](https://platform.claude.com/docs/en/build-with-claude/batch-processing)