LLM API rate limits explained: RPM, TPM and what to do on a 429

A rate limit is the most an API lets one account send in a period, counted in requests per minute (RPM) and tokens per minute (TPM). Past it, the API answers HTTP 429 Too Many Requests and generates nothing. The fix is never just "send less": it is to read the retry-after header and wait that long, to spread requests out instead of firing them together, and on OpenAI, Azure and Bedrock to stop asking for a far larger max_tokens than you need.

The last of those surprises most teams. On some platforms the limit is checked against what you might use, not what you do use.

Why a burst fails under the limit

The limit is quoted per minute, but capacity is not handed out once a minute. It works like a bucket: each request takes from it, and it refills at a steady rate. Anthropic says so outright: a limit of 60 requests a minute may be enforced as 1 a second. Azure checks requests over 1 or 10-second windows, so 600 RPM is really about 10 a second.

Turn up the traffic and watch where requests start bouncing:

Try itWhy a burst gets 429 errors under the limit

Requests this minute

30

The limit

In the same secondSpread over the minute

Play

0 / 10 left

1. accepted
2. accepted
3. accepted
4. accepted
5. accepted
6. accepted
7. accepted
8. accepted
9. accepted
10. accepted
11. 429
12. 429
13. 429
14. 429
15. 429
16. 429
17. 429
18. 429
19. 429
20. 429
21. 429
22. 429
23. 429
24. 429
25. 429
26. 429
27. 429
28. 429
29. 429
30. 429

10 accepted20 × 429

So 30 requests fired in the same second can fail while the minute's total is well under 60. Running a fixed number of requests at a time, and ramping traffic up instead of switching it on, avoids most of these errors before any retry is needed.

What each platform counts

2 rows decide most designs:

• max_tokens. OpenAI checks your request against the larger of max_tokens and its own estimate; Azure adds max_tokens to its estimate; Bedrock deducts input plus max_tokens the moment the request starts and gives the difference back at the end. Bedrock's own example shows a max_tokens of 32,000 instead of 1,250 cutting the requests a quota allows sharply. On the Claude API it costs nothing against the limit. Set it close to what you need.
• Bedrock's burndown. On Bedrock, output tokens of newer Claude models count several times against the quota: 5 times for Claude up to 4.7, 10 times for Opus 5.5, Sonnet 5 and Fable 5.1, and 15 times for Claude 4.8. A quota that looks generous for input runs out on long answers.

Caching helps beyond the bill: on the Claude API and Bedrock, input read from the prompt cache does not count toward the input limit, so caching a long shared prompt raises how much you can send. If your cache is not hitting, the prompt caching checklist finds out why.

Retry the right way

Every 429 from Anthropic carries a retry-after header; Azure sends retry-after-ms. Wait at least that long. Without a header, back off exponentially with random jitter so a crowd of clients does not retry in step:

import random, time
import anthropic

client = anthropic.Anthropic(max_retries=0)

def call(**kwargs):
    for attempt in range(6):
        try:
            return client.messages.create(**kwargs)
        except anthropic.RateLimitError as e:
            if "enforced_spend_limit_reached" in str(e.body):
                raise  # a spending cap: retrying cannot help
            wait = e.response.headers.get("retry-after")
            time.sleep(float(wait) if wait else min(60, 2 ** attempt) + random.random())
    raise RuntimeError("still rate limited after 6 tries")

The official SDKs already retry a 429 twice with backoff, so the simplest change is often max_retries=5 on a background job. Keep it low where a person is waiting: a quick "busy, try again" beats a minute of silence.

Not every 429 is worth retrying. When an Anthropic account hits its monthly spend cap, the 429 has no retry-after and an error_code of enforced_spend_limit_reached. It fails until someone raises the cap, so alert instead of looping.

See it coming

Every response says how much is left: the anthropic-ratelimit-* headers on Claude and the x-ratelimit-* headers on OpenAI and Azure. Log them beside your usage. A "remaining" line that falls toward 0 at the same hour each day tells you to cache, batch, move background work to a batch API, or ask for a higher tier before customers see errors. On Azure and Bedrock, quota is per region and per model, so a second region can double what you can send. Where capacity must be guaranteed, both sell reserved throughput.

For the idea in one page, see the rate limitRate LimitThe most an API lets you send in a period, counted in requests and tokens per minute. Past it, requests fail with a 429 error until capacity comes back, so code waits and retries. concept. To pick the cheapest model that fits your traffic, the cost calculator prices the same requests on each.