What is vLLM, and how do you serve an open model with it?

vLLM is an open-source serving engine for language models. It started as the system in the 2023 paper that introduced PagedAttentionPagedAttentionA way of storing each request's KV cache in small fixed-size blocks, like pages of virtual memory, so a serving engine wastes almost none of it and fits more requests in a batch., from researchers at UC Berkeley, and is now one of the most used ways to run an open model behind an API.

What it does

Running a model for one person is easy: load the weights and generate. Serving many people at once is a different job. The server has to batch requests together to keep the GPU busy, add new requests to a batch as others finish, and fit every request's KV cacheKV CacheWork a model saves from the text it has already read, so it is not redone for every new token. Prompt caching keeps that work between requests and charges less for it. in memory. vLLM does all of this. PagedAttention stores each cache in small blocks so almost none of the memory is wasted, which is where its speed comes from: the paper measured 2 to 4 times the ThroughputThroughputHow much work a model system gets through in a given time, counted in requests or tokens per minute; on an API it is capped by rate limits, on your own hardware by batching and memory. of the leading systems at the same latency.

Talking to it

vllm serve followed by a model's name on Hugging Face downloads the weights and starts a server on port 8000. It answers through OpenAI-compatible endpoints and through the Anthropic messages API at /v1/messages, so code written for either maker's SDK can point at it by changing the base address and the model name. That makes it easy to compare a hosted model with an open one on the same eval setEval SetThe list of test cases an LLM app is graded on after every change. Each case is an input and what a good answer must do, so a score going up or down shows whether the change helped. without rewriting anything.

Settings worth knowing

• Memory share. vLLM claims a fixed share of the GPU's memory at start, for the weights and then the cache, and sizes its batches to fit.
• Context length. A shorter maximum context leaves room for more requests at once.
• Precision. Quantized weights, such as 4 or 8-bit, leave more memory for the cache. Test quality at the precision you serve.
• LoRALoRAA way to fine-tune a model by freezing its weights and training 2 small matrices beside each chosen layer, so a task needs a tiny file instead of a full copy of the model. adapters. With --enable-lora and --lora-modules name=path, many small adapters share 1 copy of the base model, and each request picks one by putting its name in the model field.

Before it faces the internet

--api-key sets a key, but vLLM's own security page says it guards only some endpoints: those under /v1, /v2, /inference and /cohere. Others on the same server need no key. The docs say not to rely on it alone. Put the server on a private network or behind a proxy that lets only the routes you mean to expose through.