How much VRAM do you need to run an LLM locally?

To run a language model on your own GPU, count about 2 GB of VRAM per billion parametersParameterA learned weight or bias value in a neural network; large language models may have billions of parameters. at 16-bit, 1 GB at 8-bit and just over 0.5 GB at 4-bit, then add the KV cache for the conversation. That second part is the one people leave out, and for a long conversation on an older model it can be as large as the model itself.

So an 8B model at 4-bit needs about 5 GB for a short chat and about 9 GB for a 32K-token one. A 70B model at 4-bit needs about 40 GB before the conversation starts, more than the 32 GB of a GeForce RTX 5090.

The weights: parameters times bits

A model is its parameters, and each one takes a fixed number of bits in memory. The sum is parameters × bits ÷ 8, in bytes. A 7B model at 16-bit takes 7 billion × 2 bytes, about 14 GB.

QuantizationQuantizationStoring a model's numbers with less precision, such as 4 bits instead of 16, so it needs far less memory and can run on smaller hardware, at some cost in accuracy. stores each weight in fewer bits, and the memory falls with it:

4-bit is never exactly 4 bits. Every format stores scales next to the numbers so they can be turned back into real values. NVIDIA's NVFP4 adds an 8-bit scale to each block of 16 weights, 4.5 bits a weight, which is what the 4-bit column uses. The Q4_K_M files that llama.cpp and Ollama users download come to 4.89 bits a weight on Llama 3.1 8B, so a 70B Llama in that format is 43.1 GB rather than 39.7.

Fewer bits can change the answers, and how much depends on the model and the task. Test the model at the precision you will run, on your own questions.

The KV cache: the part people forget

A model writes an answer one token at a time, and each new token looks back at every token before it. To avoid working those out again for every new token, it keeps a key and a value for each earlier token in every attention layer. That store is the KV cacheKV CacheWork a model saves from the text it has already read, so it is not redone for every new token. Prompt caching keeps that work between requests and charges less for it., and it grows with every token of the conversation: your prompt, any documents you paste in, and the answer so far.

For each token the cache holds 2 × layers × KV heads × head size numbers, each 2 bytes at 16-bit. Llama 3.3 70B has 80 layers, 8 KV heads and a head size of 128:

2 × 80 × 8 × 128 × 2 bytes = 327,680 bytes, about 328 KB for every token.

A 32K-token conversation is 10.7 GB of cache. At the model's full 128K tokens it is 42.9 GB, more than its 4-bit weights. Llama 3.1 8B does the same at a smaller scale: 131 KB a token, so 17.2 GB of cache at 128K tokens against 4.5 GB of weights at 4-bit.

Choose a model below and move the conversation from 4K to 128K tokens to see which part fills the card.

Try itFit a model on a graphics card

Model

·Llama 3.1 8B·Qwen3.8 27B·Gemma 4 31B·Qwen3.6 35B A3B·Llama 3.3 70B·gpt-oss-120b

Weights

16-bit8-bit4-bit

KV cache

16-bit8-bit

Conversation

32Ktokens

the prompt, any documents and the answer so far

4K32K128K

Memory for 1 conversation

8 GB12 GB16 GB

Weights 4.52 GB

KV cache 4.29 GB

8.81 GBfits a 12 GB card

Why models of a similar size need very different caches

Parameter count says little about the cache. The layout of the attention layers decides it, and newer models are built to keep less:

• Fewer KV heads. Grouped-query attention lets several query heads share one key and value. Llama 3.3 70B has 64 query heads but keeps keys and values for 8, which already makes its cache 8 times smaller than it would otherwise be.
• Windows. Some layers keep only the most recent tokens. In gpt-oss-120b half the layers keep a window of 128 tokens, so its cache is 37 KB a token, a ninth of Llama 3.3 70B's, from a model with more parameters.
• Layers with a fixed memory. Qwen3.8 27B runs full attention in 16 of its 64 layers. The other 48 are Gated DeltaNet layers, which squeeze everything earlier into a state of fixed size instead of keeping every token. Its cache is 66 KB a token plus a fixed 154 MB, so a 128K conversation needs 8.7 GB rather than the 43 GB of Llama 3.3 70B.

Mixture-of-expertsMixture of ExpertsA neural architecture that routes tokens through different expert subnetworks. models change the other half of the sum. Qwen3.6 35B A3B uses about 3 billion parameters for each token, but each token picks its own 8 of 256 experts, so all 35 billion parameters have to be in memory: 20.2 GB at 4-bit. The few active parameters make it fast, not small.

How much VRAM popular models need

At 4-bit, with 1 conversation of 32K tokens and a 16-bit cache, leaving 8% of the card spare:

The spare share is there because the runtime needs memory of its own. vLLM, for example, takes 92% of the GPU's memory by default and fits the weights and the cache inside it. The GPU memory calculator gives the same sums for more models, more GPUs and several conversations at once, and each model has a page of its own, such as Llama 3.3 70B.

The settings that change the answer

The weights are fixed once you pick the precision. The cache is set by choices you make when you start the model:

• The context length. This is the largest lever. Ollama picks a default from the memory it finds: 4K tokens on a card under 24 GiB, 32K from 24 to 48 GiB, and 256K above. vLLM takes the model's own maximum unless you set --max-model-len. If you only chat, a short context leaves room for a larger model. If you paste in long documents, budget the cache for them.
• The cache precision. An 8-bit cache halves it. In vLLM it is --kv-cache-dtype fp8. It works best calibrated, and some layer types, such as sliding windows, lose more from it, so check the answers you care about.
• Conversations at once. Each conversation has its own cache. A server answering 8 people at once needs 8 caches beside 1 copy of the weights.

How to choose a model for your GPU

Start from the card's memory and work backwards. Take off about a tenth for the runtime. Decide the longest conversation you need, and work out its cache from the model's config or the calculator. What is left is the room for weights: divide it by about 0.56 GB per billion parameters at 4-bit, or 1 GB at 8-bit, to get the largest model that fits.

Then prefer a newer model with a small cache over an older one of the same size when you need long conversations. If a model only just fits, shorten the context before you drop to fewer bits. The cache costs memory only while the conversation is long, but lower precision costs quality on every answer.

The open models module of the free AI engineering course runs these sums on a real model, serves it with vLLM and tests what quantization changes.