What is a batch API for LLMs, and when should you use it?

Not every model call has someone waiting for it. Overnight email triage, tagging last month's tickets, running an eval setEval SetThe list of test cases an LLM app is graded on after every change. Each case is an input and what a good answer must do, so a score going up or down shows whether the change helped., summarising a library of documents: all of it can wait an hour. A batch API takes those requests in one go and answers them when it can, for less money.

How Claude's batch API works

You send up to 100,000 requests, or 256 MB, in one batch. Each is an ordinary Messages request with a custom_id you choose, such as the email's id. The batch starts as in_progress and becomes ended when every request is done. You poll for that, then read the results.

• Price: every token is billed at 50% of the standard price.
• Speed: most batches finish within an hour. Anything not done within 24 hours expires.
• Results: kept for 29 days, and returned in any order, so match each one to its request by custom_id, never by position.
• Failures: each result is succeeded, errored, canceled or expired. An invalid request must be fixed before you send it again; a server error or an expired request can simply be sent again.

Prompt cachingPrompt CachingLetting the API keep the processed start of a prompt for a few minutes, so later requests that begin the same way are cheaper and quicker. still works, but only on a best-effort basis, since requests in a batch run in any order and at different times. When every request shares a long prefix, Anthropic suggests the 1-hour cache.

When to use it

Use a batch whenever the answer is not needed in the next few minutes. Half price is the biggest saving on the bill that needs no change to the prompt, and batches do not compete with your live traffic for rate limitsRate LimitThe most an API lets you send in a period, counted in requests and tokens per minute. Past it, requests fail with a 429 error until capacity comes back, so code waits and retries. in the same way.

Do not use it for anything a person is waiting on, and do not assume a batch is quick because the last one was: the 1-hour figure is typical, not promised. Check each request's result type, and send the errored and expired ones again or to a person.