Every weekday at 9 a.m., clients of your Azure ML online endpoint start getting HTTP 429 errors and slow responses. How do you diagnose and fix it?

The deployment cannot take as many requests at once as arrive at 9 a.m. A managed online endpoint accepts at most 2 × max_concurrent_requests_per_instance × instance count requests in parallel and rejects the rest with 429. Fix it by giving it enough instances before the peak (a schedule rule), scaling on load during the day (metric rules), and setting the concurrency per instance to what one instance can really serve.

1. Confirm which limit you hit

In Azure Monitor, split Requests Per Minute by the Model Status Code dimension and look at latency and CPU per instance across the morning.

A 9 a.m. pattern with 429s and rising latency is almost always the first row: the instances are full.

2. Work out the capacity you need

Measure one instance first: send increasing load at a single instance and find the requests per second it handles within your latency target. Then:

instances needed = peak requests per second ÷ requests per second one instance handles, plus headroom.

If one instance handles 25 requests per second within target and the peak is 180, you need at least 8 instances at 9 a.m., more if you want room for a failed instance or a deployment.

3. Scale ahead of the peak and with the load

Online endpoints autoscale through Azure Monitor autoscale:

• A schedule rule raises the minimum to 8 from 8:30 to 11:00 on weekdays. Instances take minutes to start, so a rule that reacts to CPU at 9:00 arrives late for a peak that is sharp every day.
• Metric rules for the rest of the day: add instances when average CPU stays above 70% for 5 minutes, or when the endpoint's request latency rises; remove one when CPU stays below 30%.
• A maximum that fits your quota and budget.

4. Set concurrency honestly

max_concurrent_requests_per_instance should match what the scoring code can run at once. Setting it high to avoid 429s only moves the queue inside the container, and clients get timeouts (408) instead. If the model is CPU-bound in a single process, a value of 1 or 2 with more instances usually beats a high value on fewer.

5. Help from the client side

• Retry 429s with exponential backoff and jitter, never in a tight loop.
• Batch-style callers (a job scoring thousands of rows at 9 a.m.) belong on a batch endpoint, not the online one.

What interviewers want: reading status codes in Azure Monitor, the parallel request limit and where it comes from, sizing instances from a measured per-instance throughput, schedule-based scaling for a predictable peak plus metric rules, and not hiding overload by raising concurrency.