Twenty teams in your company now call LLM APIs directly, each with its own keys. Design a shared LLM gateway: what it does, how requests flow through it, and what you would not put in it.

A gateway is one internal service every model call goes through. It holds the provider keys, authenticates each team, applies quotas and rate limits, routes to the right model with fallbacks, records usage and cost per team and feature, and applies data rules before anything leaves the company. Keep it thin: it should not own prompts or business logic, and it must add little latency, because every product depends on it.

1. Why build it

• 20 sets of keys, no view of total spend, and no one knows which team caused a rate limit hit that slowed everyone.
• The provider's limits are per organization (and optionally per workspace), so teams share capacity whether they know it or not.
• Security and privacy rules applied 20 times, differently.

2. Request flow

team service ──▶ gateway
   1. authenticate (team, feature, environment from the service identity)
   2. check quota and per-team rate limit
   3. apply data rules (block or redact fields; size limits)
   4. route: model alias → provider and model, region, priority
   5. call provider (streaming passthrough), retry or fall back on 429/5xx
   6. record usage: tokens by type, cost, latency, status, request ID
──▶ response streamed back

3. Key decisions

• Model aliases: teams ask for fast or smart, or a pinned model ID when they need stability. The platform team can move an alias after testing, and teams can pin when they need to.
• Capacity: split the organization's limits into per-team budgets (separate workspaces where the provider supports them), with priority for user-facing traffic over batch jobs. Honour retry-after on 429s, and ramp new heavy workloads gradually.
• Cost attribution: log input, cache write, cache read and output tokens separately per call, priced from a table. Daily cost per team and feature, with alerts.
• Caching: leave prompt caching to the provider (it is cheaper and cache reads do not count toward input rate limits on most Claude models). A gateway-level response cache is risky with personal data; if used, key it by tenant and use it only for public content.
• Fallbacks: a second model or provider for outages, tested in advance, because a fallback that has never run will fail when needed.
• Streaming: pass events through unbuffered; buffering adds latency and memory.

4. What stays out

• Prompts, tools and evaluation belong to product teams, versioned in their code.
• Full conversation logging by default. Log metadata always; content only where policy allows, with retention limits and access controls.

5. Reliability and abuse

• Run the gateway in several zones, with its own load tests. If it is down, everything is down.
• Per-user limits for features open to the public, input size caps, and alerts on unusual spend, because cost-per-use invites abuse.

What interviewers want: a clear scope (auth, quotas, routing, usage, data rules), knowledge that provider limits are shared, model aliases with pinning, token-level cost attribution, honest treatment of caching and logging, tested fallbacks, and the gateway's own reliability.