The full answer
Expect errors and design for them: retry temporary failures with exponential backoff and jitter (honouring the retry-after header), don't retry errors that will not change (bad requests, spend limits), control your own request rate so you stay under the provider's limits, and have a fallback for when the model is unavailable: a queue, another model or region, or a hand-off to people.
Know the error types (Claude API as an example)
| Error | Meaning | What to do |
|---|---|---|
| 400 invalid request | Your request is wrong | Fix the code; do not retry |
| 401 / 403 | Key or permission problem | Alert; do not retry |
| 429 rate_limit_error | Over a rate limit (or a spend limit) | Wait for retry-after, then retry |
| 500 api_error | Error on the provider's side | Retry with backoff |
| 504 timeout_error | Took too long | Use streaming for long requests; retry |
| 529 overloaded_error | Provider overloaded across users | Retry with backoff; consider a fallback |
Anthropic's rate limit documentation adds two details worth knowing. Limits are measured in requests, input tokens and output tokens per minute for each model class, using a token bucket that refills continuously, so short bursts can fail even when the minute's average is fine. And a 429 caused by a spend limit has no retry-after header and will keep failing until access returns, so code must tell the two apart rather than retrying forever.
Retry well
- Exponential backoff with random jitter, so many clients do not retry at the same moment.
- A cap on attempts and on total time, then fail clearly.
- Make retried actions safe to repeat (idempotency keys for anything with side effects).
- Most official SDKs retry some errors automatically; know their defaults so you do not double them.
Avoid hitting limits
- A client-side rate limiter or a queue that smooths bursts.
- Batch APIs for work that can wait.
- Prompt caching and shorter prompts, which cut input tokens per minute.
- Ask for higher limits before a launch, with your traffic estimate.
Fallbacks
- Another model or region for the same task, tested on your evaluation set in advance.
- Degrade gracefully: queue the request and notify later, show search results without a generated answer, or hand off to a person.
- A circuit breaker, so a provider outage does not tie up all your workers.
Monitor
Error rates by type, retries, time spent waiting, and how close you are to each limit. Alert before you hit limits, not after.
Summary: "Retry what's temporary, never what's permanent, keep my own traffic under the limits, and always have a plan B."
Last updated 16 October 2026
Sources