What is throughput for an LLM, and what limits it on an API and on your own GPU?

Throughput is how much a system gets done per minute: how many emails it can triage, or how many tokens it can read and write. LatencyLatencyHow long a user waits for a model's answer, measured as the time to the first token and the time to the last, and judged by the slowest requests, not the average. is how long one request takes. The 2 pull against each other, and a system can have good throughput and still feel slow to each user.

On a hosted API

Your throughput is mostly set by the provider's rate limitsRate LimitThe most an API lets you send in a period, counted in requests and tokens per minute. Past it, requests fail with a 429 error until capacity comes back, so code waits and retries.. Claude's are counted 3 ways for each model class: requests per minute, input tokens per minute and output tokens per minute. They refill continuously rather than resetting each minute, and going over returns a 429 error that the SDK retries.

2 things raise what you can push through:

• Caching. For most Claude models, tokens read from the prompt cachePrompt CachingLetting the API keep the processed start of a prompt for a few minutes, so later requests that begin the same way are cheaper and quicker. do not count toward the input limit. Anthropic's example: with a limit of 2 million input tokens a minute and 80% of input read from cache, you can process 10 million.
• Batches. Work that can wait goes through batch processingBatch ProcessingSending many model requests together to be answered within hours instead of seconds, at a lower price; the right choice for any work no one is waiting on., which runs outside the normal request flow at half price.

Which of the 3 limits you hit first depends on the job. A classifier that reads long emails and writes a few words runs into the input limit; a writer of long reports runs into the output limit long before the request limit. Log tokens per call and you can see which. When caching and batches are not enough, the answer is a higher usage tier, or spreading load over models whose limits are separate.

On your own hardware

Run an open model yourself and throughput is your problem. A GPU is fastest when it works on many requests at once, so serving engines batch them, and the limit becomes memory: each request in flight keeps a KV cacheKV CacheWork a model saves from the text it has already read, so it is not redone for every new token. Prompt caching keeps that work between requests and charges less for it. that grows with its length. The vLLM paper's contribution was wasting far less of that memory, which let it fit more requests in a batch and raised throughput 2 to 4 times at the same latency. Bigger batches still slow each request a little, so you tune for the latency your users accept and take whatever throughput that leaves.