Your LLM feature works in testing, but at the busiest hour of the day a share of requests fail with HTTP 429. How do you handle it?

A 429 means the account hit a rate limit, so handle it in 2 layers. Right now: retry correctly, waiting as long as the retry-after header says, or with exponential backoff and jitter when there is none, and never retry a 429 that means a spending cap. Then: send fewer or smaller requests per second, by smoothing bursts, caching, batching background work and setting max_tokens close to what you need, and raise the limit itself if the steady load still exceeds it.

1. Why it fails below the per-minute number

Limits are quoted per minute but enforced continuously, like a bucket that refills at a steady rate. Anthropic says 60 requests a minute may be enforced as 1 a second; Azure checks over 1 or 10-second windows. 30 requests fired in the same second can fail while the minute's total is well under the limit.

2. Retry correctly

import random, time

def with_retry(call, tries=6):
    for attempt in range(tries):
        try:
            return call()
        except RateLimited as e:                 # your SDK's 429 error
            if e.is_spend_cap:                   # e.g. Anthropic's enforced_spend_limit_reached
                raise                            # retrying cannot help: alert instead
            wait = e.retry_after or min(60, 2 ** attempt) + random.random()
            time.sleep(wait)
    raise RuntimeError("still rate limited")

Keep retries few where a person is waiting; a quick "busy, try again" beats a minute of silence.

3. Send less per second

4. See it coming

Log the remaining-capacity headers (anthropic-ratelimit-*, x-ratelimit-*) with each call. A line that hits 0 at the same hour each day is a capacity plan, not a bug: ask for a higher tier or reserved throughput before users notice. See Rate Limit.

What interviewers want: the token-bucket behaviour behind bursts, retry-after and backoff with jitter, spend-cap 429s not retried, smoothing and caching to cut load, max_tokens counting on some platforms, and monitoring the headers to plan capacity.