What is a rate limit in an LLM API, and how do you handle one?

Every model API caps how much one account can send. When you go over, the request fails at once with HTTP 429, "too many requests", and nothing is generated. A rate limit is not a bill or a ban: wait, and the same request succeeds. What makes it an engineering problem is that it arrives at your busiest moment, when the most people are waiting.

What is counted

A model API counts more than requests, because 1 request can carry 10 words or a whole book. The Claude API counts 3 things for each model class:

• Requests per minute (RPM).
• Input tokens per minute (ITPM), the tokensTokenThe small piece of text a language model reads and writes, often part of a word. Prices, limits and the context window are all counted in tokens. you send.
• Output tokens per minute (OTPM), the tokens the model writes.

You hit whichever runs out first. A chat app with short messages meets the request limit; an app that sends long documents meets the input limit long before.

2 details change the sums. Input read from the prompt cache does not count toward the input limit on most Claude models, so caching a long shared prompt raises how much you can send, as well as cutting the bill. And the output limit counts only the tokens actually written, so a generous max_tokens costs nothing against it.

Why a burst fails under the limit

The limit is per minute, but capacity is not handed out once a minute. It works like a bucket that refills continuously: each request takes from it, and it tops back up at a steady rate. A limit of 60 requests a minute can behave like 1 a second, so 30 requests fired in the same second can fail even though the minute's total is well under 60. A sudden jump in traffic can also trip a separate limit on how fast usage grows.

The fix is to spread work out. Run a fixed number of requests at a time instead of all at once, and ramp traffic up rather than switching it on.

Waiting the right way

When a 429 arrives, the response usually says how long to wait in a retry-after header. Wait at least that long. If there is no header, wait a little, then longer each time (exponential backoff), with some randomness so many clients do not retry in step.

You often get this for free. The official Claude SDKs retry a 429, a timeout, a 409 and server errors twice with backoff before raising an error, and max_retries changes the count. Raise it for a background job that can wait; keep it low where a person is waiting and a quick "try again" message is better than a long silence.

2 cases need different handling:

• A spend cap is not a rate limit. When an account reaches its monthly spend cap, the Claude API also answers 429, but with no retry-after header and an error code of enforced_spend_limit_reached. Retrying cannot help until the cap is raised, so detect it and alert someone.
• A 529 is not your limit. It means the service is overloaded for everyone. Back off and retry the same way, or send the request to another model.

Seeing it coming

Each response carries headers that say what is left: anthropic-ratelimit-requests-remaining, anthropic-ratelimit-input-tokens-remaining, anthropic-ratelimit-output-tokens-remaining and a reset time for each. Log them with your usage numbers. A dashboard of "remaining" that falls toward 0 at the same hour every day tells you to cache, batch or ask for a higher tier before customers see errors.