How much GPU memory do I need to run an LLM?

Its weights plus its KV cache. The weights take 2 bytes a parameter at 16-bit, 1 at 8-bit and about 0.56 at 4-bit, so Llama 3.3 70B needs 141 GB, 70.6 GB or 39.7 GB. The cache adds 328 KB for each token of each conversation, and the server keeps about 8% of the GPU free.

How much VRAM does a 70B model need?

About 141 GB at 16-bit, 70.6 GB at 8-bit and 39.7 GB at 4-bit for Llama 3.3 70B's weights, plus the KV cache: 2.68 GB for 1 conversation of 8,192 tokens. At 16-bit that is 2 H100s; at 4-bit, 1 H100, or 2 GeForce RTX 5090s.

How is the KV cache size calculated?

2 (keys and values) × layers × KV heads × head size × bytes per number, for each token. Llama 3.3 70B has 80 layers, 8 KV heads of 128 numbers, at 2 bytes: 328 KB a token. Multiply by the tokens in a conversation and by the conversations at once. Sliding window, latent and linear attention layers keep less, and an 8-bit cache halves it.

Does a mixture-of-experts model need less memory?

No. Every expert must be in memory, so a mixture of experts needs memory for all its parameters. It is faster, though: each token reads only the experts it uses, so a model with 3 billion active parameters writes tokens about as fast as a 3-billion-parameter model on the same GPU.

Is 4-bit quantization good enough?

Often. A 4-bit model needs about a quarter of the memory of 16-bit and reads a quarter of the bytes for each token, so it runs faster too. Some models ship at 4-bit (gpt-oss in MXFP4, Kimi K2.6 in INT4) and are trained for it; for others, test the 4-bit build on your own tasks before relying on it.

How fast will the model run on my GPU?

At most the GPU's memory bandwidth divided by the bytes read for each token. Llama 3.3 70B at 4-bit on an H100 can write at most 79 tokens a second for 1 conversation. Real servers reach part of that, and with many conversations at once each gets less while the total grows.

Can I run a model across several GPUs?

Yes. vLLM splits a model across the GPUs of a server with tensor parallelism, and across servers with pipeline parallelism. The memory adds up, so 2 GPUs of 80 GB hold a model of about 147 GB, but each GPU still needs room for its share of the cache.

How much GPU memory does a model need?

Llama 3.3 70B needs 141 GB for its weights at 16-bit, 70.6 GB at 8-bit and 39.7 GB at 4-bit, plus 328 KB of KV cache for every token of every conversation.

Memory decides whether an open model runs at all: its weights must fit, with room for the KV cache that grows with each token. Choose a model, a number format, how long the conversations are and how many run at once, and see what fits on each GPU, from a desktop RTX 5090 to an H200 or MI355X.

Size a model Each model

Model

Meta, 70.6B parameters, released in BF16, up to 131,072 tokens.

Tokens in each conversation

tokens

The prompt, documents and the answer so far.

Conversations at once

conversation

Requests the server answers at the same time, each with its own cache.

GPU

NVIDIA, 80 GB, 3,350 GB/s of memory bandwidth.

The answer

Llama 3.3 70B as released (BF16) with 1 conversation of 8,192 tokens needs 144 GB: 2 H100s, at most 47 tokens a second.

Llama 3.3 70B on every GPU

Memory of 2 H100s144 GB of 147 GB usable

Weights 141 GBKV cache 2.68 GBFree 3.41 GB

How many of each GPU hold Llama 3.3 70B with these settings, and the most tokens a second
GPU	Memory	GPUs	Tokens/s, at most
Data center	24 GB	8	17
Desktop	32 GB	8	100
Data center	48 GB	4	24
Data center	80 GB	2	28
Data center	80 GB	2	47
Workstation	96 GB	2	25
Desktop	128 GB	2	3.8
Data center	141 GB	2	67
Data center	180 GB	1	56
Data center	192 GB	1	37
Data center	256 GB	1	42
Data center	288 GB	1	56

The weights. 70.6B parameters at 16 bits each, the size of its released files: 141 GB.
The KV cache. The cache of Llama 3.3 70B has 80 layers that keep every token. Past the windows each token adds 328 KB, so a conversation of 8,192 tokens holds 2.68 GB, and 1 conversation 2.68 GB.
The GPUs. vLLM uses 92% of each GPU's memory by default, so a H100 counts as 73.6 GB. It needs 2, split with tensor parallelism. With the weights loaded, there is room for 18,592 tokens in each of 1 conversation.
The speed. Each token reads all the weights once and every conversation's cache, at 6,700 GB/s, so memory allows at most 47 tokens a second . Real servers reach part of this ceiling, and with many conversations the GPU's compute becomes the limit.

How the calculator works it out

Memory has 2 parts. The weights are every parameter at its number format: 2 bytes at 16-bit, 1 at 8-bit and about 0.56 at 4-bit, since NVIDIA's NVFP4 adds an 8-bit scale for every 16 numbers. As released, a model takes the size of its own files: some ship at 8-bit (FP8) or 4-bit (MXFP4, INT4) already. Llama 3.3 70B, 70.6B parameters, takes 141 GB at 16-bit.

The KV cache keeps the keys and values of every token so the model does not compute them again. For a layer that sees every token it is 2 × KV heads × head size numbers a token; Llama 3.3 70B keeps 328 KB a token, so a conversation of 8,192 tokens holds 2.68 GB, and one at its longest, 131,072 tokens, 42.9 GB. Newer models keep far less: layers with a sliding window keep only the last few hundred or thousand tokens, DeepSeek-style latent attention keeps 1 compressed vector a token, and linear attention or Mamba layers keep a fixed state whatever the length. The calculator reads each model's layers from its own config file.

vLLM takes 92% of each GPU's memory by default, so a GPU counts as 92% of its size. A model too big for 1 GPU is split across 2, 4 or 8 in a server, then across servers of 8. At 16-bit, Llama 3.3 70B needs 2 H100s; at 4-bit, 1 H100.

Writing a token reads the weights it uses once, plus every conversation's cache, so memory speed sets a ceiling on tokens a second: 47 for Llama 3.3 70B at 16-bit, 79 at 4-bit. A mixture of experts reads only its active experts, which is why such models are fast for their size. Real servers reach part of this ceiling, less for small models and models split across many GPUs; with many conversations at once, the GPU's compute becomes the limit instead.

GPU memory for each model

Each page gives a model's memory at 16, 8 and 4-bit, its KV cache, and how many of each GPU hold it.

Questions

How much GPU memory do I need to run an LLM?: Its weights plus its KV cache. The weights take 2 bytes a parameter at 16-bit, 1 at 8-bit and about 0.56 at 4-bit, so Llama 3.3 70B needs 141 GB, 70.6 GB or 39.7 GB. The cache adds 328 KB for each token of each conversation, and the server keeps about 8% of the GPU free.
How much VRAM does a 70B model need?: About 141 GB at 16-bit, 70.6 GB at 8-bit and 39.7 GB at 4-bit for Llama 3.3 70B's weights, plus the KV cache: 2.68 GB for 1 conversation of 8,192 tokens. At 16-bit that is 2 H100s; at 4-bit, 1 H100, or 2 GeForce RTX 5090s.
How is the KV cache size calculated?: 2 (keys and values) × layers × KV heads × head size × bytes per number, for each token. Llama 3.3 70B has 80 layers, 8 KV heads of 128 numbers, at 2 bytes: 328 KB a token. Multiply by the tokens in a conversation and by the conversations at once. Sliding window, latent and linear attention layers keep less, and an 8-bit cache halves it.
Does a mixture-of-experts model need less memory?: No. Every expert must be in memory, so a mixture of experts needs memory for all its parameters. It is faster, though: each token reads only the experts it uses, so a model with 3 billion active parameters writes tokens about as fast as a 3-billion-parameter model on the same GPU.
Is 4-bit quantization good enough?: Often. A 4-bit model needs about a quarter of the memory of 16-bit and reads a quarter of the bytes for each token, so it runs faster too. Some models ship at 4-bit (gpt-oss in MXFP4, Kimi K2.6 in INT4) and are trained for it; for others, test the 4-bit build on your own tasks before relying on it.
How fast will the model run on my GPU?: At most the GPU's memory bandwidth divided by the bytes read for each token. Llama 3.3 70B at 4-bit on an H100 can write at most 79 tokens a second for 1 conversation. Real servers reach part of that, and with many conversations at once each gets less while the total grows.
Can I run a model across several GPUs?: Yes. vLLM splits a model across the GPUs of a server with tensor parallelism, and across servers with pipeline parallelism. The memory adds up, so 2 GPUs of 80 GB hold a model of about 147 GB, but each GPU still needs room for its share of the cache.

Put this calculator on your own site

Copy the code into any web page: a blog post, a course page or your team's wiki. The calculator runs in a frame and shows the same figures as this page, so when the prices change here they change on your page too. It is free and needs no account, and nothing a reader types is sent anywhere.

The code opens the calculator with the settings you have chosen above.

The short script at the end lets the frame grow and shrink with the calculator. Where a page cannot take scripts, choose a fixed height and the frame scrolls.

<iframe src="https://tecyfy.com/embed/gpu-memory-calculator" title="GPU memory calculator" width="100%" height="2400" style="border:0;display:block" loading="lazy" allow="clipboard-write"></iframe>
<p><a href="https://tecyfy.com/ai-agents/gpu-memory-calculator">GPU memory calculator</a> by Tecyfy</p>
<script>addEventListener("message",function(e){if(e.origin!=="https://tecyfy.com"||!e.data||e.data.type!=="tecyfy:height")return;var h=Number(e.data.height);if(!(h>0))return;document.querySelectorAll("iframe").forEach(function(f){if(f.contentWindow===e.source)f.style.height=Math.min(h,20000)+"px"})})</script>

Sources

The calculator

How the calculator works it out

GPU memory for each model

Questions

Put this calculator on your own site

Where each fact comes from