# What is a rate limit in an LLM API, and how do you handle one? The most an API lets you send in a period, counted in requests and tokens per minute. Past it, requests fail with a 429 error until capacity comes back, so code waits and retries. - Page: https://tecyfy.com/concepts/rate-limit - Subject: How AI works (https://tecyfy.com/concepts) - Level: intermediate - Also called: Rate Limits, 429 Too Many Requests, RPM and TPM, Retries and Backoff Every model API caps how much one account can send. When you go over, the request fails at once with HTTP 429, "too many requests", and nothing is generated. A rate limit is not a bill or a ban: wait, and the same request succeeds. What makes it an engineering problem is that it arrives at your busiest moment, when the most people are waiting. ## What is counted A model API counts more than requests, because 1 request can carry 10 words or a whole book. The Claude API counts 3 things for each model class: - **Requests per minute (RPM).** - **Input tokens per minute (ITPM)**, the [tokens](https://tecyfy.com/concepts/token) you send. - **Output tokens per minute (OTPM)**, the tokens the model writes. You hit whichever runs out first. A chat app with short messages meets the request limit; an app that sends long documents meets the input limit long before. 2 details change the sums. Input read from the prompt cache does not count toward the input limit on most Claude models, so caching a long shared prompt raises how much you can send, as well as cutting the bill. And the output limit counts only the tokens actually written, so a generous `max_tokens` costs nothing against it. ## Why a burst fails under the limit The limit is per minute, but capacity is not handed out once a minute. It works like a bucket that refills continuously: each request takes from it, and it tops back up at a steady rate. A limit of 60 requests a minute can behave like 1 a second, so 30 requests fired in the same second can fail even though the minute's total is well under 60. A sudden jump in traffic can also trip a separate limit on how fast usage grows. The fix is to spread work out. Run a fixed number of requests at a time instead of all at once, and ramp traffic up rather than switching it on. ## Waiting the right way When a 429 arrives, the response usually says how long to wait in a `retry-after` header. Wait at least that long. If there is no header, wait a little, then longer each time (exponential backoff), with some randomness so many clients do not retry in step. You often get this for free. The official Claude SDKs retry a 429, a timeout, a 409 and server errors twice with backoff before raising an error, and `max_retries` changes the count. Raise it for a background job that can wait; keep it low where a person is waiting and a quick "try again" message is better than a long silence. 2 cases need different handling: - **A spend cap is not a rate limit.** When an account reaches its monthly spend cap, the Claude API also answers 429, but with no `retry-after` header and an error code of `enforced_spend_limit_reached`. Retrying cannot help until the cap is raised, so detect it and alert someone. - **A 529 is not your limit.** It means the service is overloaded for everyone. Back off and retry the same way, or send the request to another model. ## Seeing it coming Each response carries headers that say what is left: `anthropic-ratelimit-requests-remaining`, `anthropic-ratelimit-input-tokens-remaining`, `anthropic-ratelimit-output-tokens-remaining` and a reset time for each. Log them with your usage numbers. A dashboard of "remaining" that falls toward 0 at the same hour every day tells you to cache, batch or ask for a higher tier before customers see errors. ## Related - [API](https://tecyfy.com/concepts#api): Application Programming Interface - a set of protocols and tools for building software applications - [Token](https://tecyfy.com/concepts/token): The small piece of text a language model reads and writes, often part of a word. Prices, limits and the context window are all counted in tokens. - [Streaming](https://tecyfy.com/concepts/streaming): Getting a model's answer in small pieces as it is written, instead of all at once at the end, so people see text start sooner and long answers do not hit a timeout. - [Latency](https://tecyfy.com/concepts/latency): How long a user waits for a model's answer, measured as the time to the first token and the time to the last, and judged by the slowest requests, not the average. ## Sources - [Rate limits](https://platform.claude.com/docs/en/api/rate-limits) - [Errors](https://platform.claude.com/docs/en/api/errors) - [Python SDK](https://platform.claude.com/docs/en/cli-sdks-libraries/sdks/python)