# What is vLLM, and how do you serve an open model with it?

An open-source engine for serving language models fast, built on PagedAttention, that answers requests through the same kinds of API the big model makers offer.

- Page: https://tecyfy.com/concepts/vllm
- Subject: How AI works (https://tecyfy.com/concepts)
- Level: advanced
- Also called: vLLM server, vllm serve

vLLM is an open-source serving engine for language models. It started as the system in the 2023 paper that introduced [PagedAttention](https://tecyfy.com/concepts/pagedattention), from researchers at UC Berkeley, and is now one of the most used ways to run an open model behind an API.

## What it does

Running a model for one person is easy: load the weights and generate. Serving many people at once is a different job. The server has to batch requests together to keep the GPU busy, add new requests to a batch as others finish, and fit every request's [KV cache](https://tecyfy.com/concepts/kv-cache) in memory. vLLM does all of this. PagedAttention stores each cache in small blocks so almost none of the memory is wasted, which is where its speed comes from: the paper measured 2 to 4 times the [Throughput](https://tecyfy.com/concepts/throughput) of the leading systems at the same latency.

## Talking to it

`vllm serve` followed by a model's name on Hugging Face downloads the weights and starts a server on port 8000. It answers through OpenAI-compatible endpoints and through the Anthropic messages API at `/v1/messages`, so code written for either maker's SDK can point at it by changing the base address and the model name. That makes it easy to compare a hosted model with an open one on the same [eval set](https://tecyfy.com/concepts/eval-set) without rewriting anything.

## Settings worth knowing

- **Memory share.** vLLM claims a fixed share of the GPU's memory at start, for the weights and then the cache, and sizes its batches to fit.
- **Context length.** A shorter maximum context leaves room for more requests at once.
- **Precision.** Quantized weights, such as 4 or 8-bit, leave more memory for the cache. Test quality at the precision you serve.
- **[LoRA](https://tecyfy.com/concepts/lora) adapters.** With `--enable-lora` and `--lora-modules name=path`, many small adapters share 1 copy of the base model, and each request picks one by putting its name in the `model` field.

## Before it faces the internet

`--api-key` sets a key, but vLLM's own security page says it guards only some endpoints: those under `/v1`, `/v2`, `/inference` and `/cohere`. Others on the same server need no key. The docs say not to rely on it alone. Put the server on a private network or behind a proxy that lets only the routes you mean to expose through.

## Related

- [PagedAttention](https://tecyfy.com/concepts/pagedattention): A way of storing each request's KV cache in small fixed-size blocks, like pages of virtual memory, so a serving engine wastes almost none of it and fits more requests in a batch.
- [KV Cache](https://tecyfy.com/concepts/kv-cache): Work a model saves from the text it has already read, so it is not redone for every new token. Prompt caching keeps that work between requests and charges less for it.
- [Inference](https://tecyfy.com/concepts/inference): Using a trained model to produce an answer. Training builds the model once; inference is every time someone asks it something, and it is what you pay for per token.
- [Throughput](https://tecyfy.com/concepts/throughput): How much work a model system gets through in a given time, counted in requests or tokens per minute; on an API it is capped by rate limits, on your own hardware by batching and memory.
- [LoRA](https://tecyfy.com/concepts/lora): A way to fine-tune a model by freezing its weights and training 2 small matrices beside each chosen layer, so a task needs a tiny file instead of a full copy of the model.
- [Quantization](https://tecyfy.com/concepts/quantization): Storing a model's numbers with less precision, such as 4 bits instead of 16, so it needs far less memory and can run on smaller hardware, at some cost in accuracy.

## Sources

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://arxiv.org/abs/2309.06180)
- [Online serving](https://docs.vllm.ai/en/latest/serving/online_serving/)
- [Security](https://docs.vllm.ai/en/latest/usage/security/)
- [Using LoRA for efficient fine-tuning](https://docs.vllm.ai/en/latest/features/lora/)