# What is throughput for an LLM, and what limits it on an API and on your own GPU?

How much work a model system gets through in a given time, counted in requests or tokens per minute; on an API it is capped by rate limits, on your own hardware by batching and memory.

- Page: https://tecyfy.com/concepts/throughput
- Subject: How AI works (https://tecyfy.com/concepts)
- Level: beginner
- Also called: Tokens Per Second, Requests Per Minute

Throughput is how much a system gets done per minute: how many emails it can triage, or how many tokens it can read and write. [Latency](https://tecyfy.com/concepts/latency) is how long one request takes. The 2 pull against each other, and a system can have good throughput and still feel slow to each user.

## On a hosted API

Your throughput is mostly set by the provider's [rate limits](https://tecyfy.com/concepts/rate-limit). Claude's are counted 3 ways for each model class: requests per minute, input tokens per minute and output tokens per minute. They refill continuously rather than resetting each minute, and going over returns a 429 error that the SDK retries.

2 things raise what you can push through:

- **Caching.** For most Claude models, tokens read from the [prompt cache](https://tecyfy.com/concepts/prompt-caching) do not count toward the input limit. Anthropic's example: with a limit of 2 million input tokens a minute and 80% of input read from cache, you can process 10 million.
- **Batches.** Work that can wait goes through [batch processing](https://tecyfy.com/concepts/batch-processing), which runs outside the normal request flow at half price.

Which of the 3 limits you hit first depends on the job. A classifier that reads long emails and writes a few words runs into the input limit; a writer of long reports runs into the output limit long before the request limit. Log tokens per call and you can see which. When caching and batches are not enough, the answer is a higher usage tier, or spreading load over models whose limits are separate.

## On your own hardware

Run an open model yourself and throughput is your problem. A GPU is fastest when it works on many requests at once, so serving engines batch them, and the limit becomes memory: each request in flight keeps a [KV cache](https://tecyfy.com/concepts/kv-cache) that grows with its length. The vLLM paper's contribution was wasting far less of that memory, which let it fit more requests in a batch and raised throughput 2 to 4 times at the same latency. Bigger batches still slow each request a little, so you tune for the latency your users accept and take whatever throughput that leaves.

## Related

- [Latency](https://tecyfy.com/concepts/latency): How long a user waits for a model's answer, measured as the time to the first token and the time to the last, and judged by the slowest requests, not the average.
- [Inference Latency](https://tecyfy.com/concepts/inference-latency): The time a model takes to produce an answer, split into reading the prompt (prefill, which sets the time to first token) and writing the answer one token at a time (decode).
- [Rate Limit](https://tecyfy.com/concepts/rate-limit): The most an API lets you send in a period, counted in requests and tokens per minute. Past it, requests fail with a 429 error until capacity comes back, so code waits and retries.
- [Batch Processing](https://tecyfy.com/concepts/batch-processing): Sending many model requests together to be answered within hours instead of seconds, at a lower price; the right choice for any work no one is waiting on.
- [Prompt Caching](https://tecyfy.com/concepts/prompt-caching): Letting the API keep the processed start of a prompt for a few minutes, so later requests that begin the same way are cheaper and quicker.
- [vLLM](https://tecyfy.com/concepts/vllm): An open-source engine for serving language models fast, built on PagedAttention, that answers requests through the same kinds of API the big model makers offer.

## Sources

- [Rate limits](https://platform.claude.com/docs/en/api/rate-limits)
- [Batch processing](https://platform.claude.com/docs/en/build-with-claude/batch-processing)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://arxiv.org/abs/2309.06180)