How much GPU memory do I need to run an LLM?

Its weights plus its KV cache. The weights take 2 bytes a parameter at 16-bit, 1 at 8-bit and about 0.56 at 4-bit, so Granite 4.2 30B needs 58.6 GB, 29.3 GB or 16.5 GB. The cache adds 262 KB for each token of each conversation, and the server keeps about 8% of the GPU free.

How much VRAM does a 30B model need?

About 58.6 GB at 16-bit, 29.3 GB at 8-bit and 16.5 GB at 4-bit for Granite 4.2 30B's weights, plus the KV cache: 2.15 GB for 1 conversation of 8,192 tokens. At 16-bit that is 1 H100; at 4-bit, 1 H100, or 1 GeForce RTX 5090.

How is the KV cache size calculated?

2 (keys and values) × layers × KV heads × head size × bytes per number, for each token. Granite 4.2 30B has 64 layers that each keep 2,048 numbers a token, at 2 bytes: 262 KB a token. Multiply by the tokens in a conversation and by the conversations at once. Sliding window, latent and linear attention layers keep less, and an 8-bit cache halves it.

Does a mixture-of-experts model need less memory?

No. Every expert must be in memory, so a mixture of experts needs memory for all its parameters. It is faster, though: each token reads only the experts it uses, so a model with 3 billion active parameters writes tokens about as fast as a 3-billion-parameter model on the same GPU.

Is 4-bit quantization good enough?

Often. A 4-bit model needs about a quarter of the memory of 16-bit and reads a quarter of the bytes for each token, so it runs faster too. Some models ship at 4-bit (gpt-oss in MXFP4, Kimi K2.6 in INT4) and are trained for it; for others, test the 4-bit build on your own tasks before relying on it.

How fast will the model run on my GPU?

At most the GPU's memory bandwidth divided by the bytes read for each token. Granite 4.2 30B at 4-bit on an H100 can write at most 180 tokens a second for 1 conversation. Real servers reach part of that, and with many conversations at once each gets less while the total grows.

Can I run a model across several GPUs?

Yes. vLLM splits a model across the GPUs of a server with tensor parallelism, and across servers with pipeline parallelism. The memory adds up, so 2 GPUs of 80 GB hold a model of about 147 GB, but each GPU still needs room for its share of the cache.

How much GPU memory does a model need?

Granite 4.2 30B needs 58.6 GB for its weights at 16-bit, 29.3 GB at 8-bit and 16.5 GB at 4-bit, plus 262 KB of KV cache for every token of every conversation.

Memory decides whether an open model runs at all: its weights must fit, with room for the KV cache that grows with each token. Choose a model, a number format, how long the conversations are and how many run at once, and see what fits on each GPU, from a desktop RTX 5090 to an H200 or MI355X.

Size a model Each model

Model

IBM, 29.3B parameters, released in BF16, up to 131,072 tokens.

Tokens in each conversation

tokens

The prompt, documents and the answer so far.

Conversations at once

conversation

Requests the server answers at the same time, each with its own cache.

GPU

NVIDIA, 80 GB, 3,350 GB/s of memory bandwidth.

The answer

Granite 4.2 30B as released (BF16) with 1 conversation of 8,192 tokens needs 60.7 GB: 1 H100, at most 55 tokens a second.

Granite 4.2 30B on every GPU

Memory of 1 H10060.7 GB of 73.6 GB usable

Weights 58.6 GBKV cache 2.15 GBFree 12.9 GB

How many of each GPU hold Granite 4.2 30B with these settings, and the most tokens a second
GPU	Memory	GPUs	Tokens/s, at most
Data center	24 GB	4	20
Desktop	32 GB	4	118
Data center	48 GB	2	28
Data center	80 GB	1	34
Data center	80 GB	1	55
Workstation	96 GB	1	30
Desktop	128 GB	1	4.5
Data center	141 GB	1	79
Data center	180 GB	1	132
Data center	192 GB	1	87
Data center	256 GB	1	99
Data center	288 GB	1	132

The weights. 29.3B parameters at 16 bits each, the size of its released files: 58.6 GB.
The KV cache. The cache of Granite 4.2 30B has 64 layers that keep every token. Each token adds 262 KB, so a conversation of 8,192 tokens holds 2.15 GB, and 1 conversation 2.15 GB.
The GPUs. vLLM uses 92% of each GPU's memory by default, so a H100 counts as 73.6 GB. One holds it. With the weights loaded, there is room for 57,397 tokens in each of 1 conversation.
The speed. Each token reads all the weights once and every conversation's cache, at 3,350 GB/s, so memory allows at most 55 tokens a second . Real servers reach part of this ceiling, and with many conversations the GPU's compute becomes the limit.

How much GPU memory does each open model need?

Every model in the calculator, newest first: its weights as released (most at 16-bit), at 8-bit and at 4-bit, and the KV cache of 1 conversation of 32,768 tokens in 16-bit. Add the 2 for the memory to hold, plus room for the serving engine.

How much GPU memory does each open model need?
Model	Maker	Parameters	As released	8-bit	4-bit	KV cache, 32,768 tokens
Nemotron 3.5 Bolt E4B	NVIDIA	8B	16 GB	8 GB	4.5 GB	558 MB
Nemotron 3.5 Bolt 74B A9B 24GB	NVIDIA	74.9B	24.1 GB	74.9 GB	42.1 GB	439 MB
Nemotron 3.5 Super VL 120B A12B	NVIDIA	124B	249 GB	124 GB	70 GB	439 MB
GLM-5.3	Z.ai	753B	756 GB	753 GB	424 GB	3.6 GB
Qwen3.8 2.4T A95B	Qwen	2.4T	4,892 GB	2,446 GB	1,376 GB	3.67 GB
Granite 4.2 30B	IBM	29.3B	58.6 GB	29.3 GB	16.5 GB	8.59 GB
Qwen3.8 27B	Qwen	27.8B	55.6 GB	27.8 GB	15.6 GB	2.3 GB
Nemotron 3.5 Lightning 30B A3B	NVIDIA	31.6B	65.8 GB	31.6 GB	17.8 GB	250 MB
Gemma 4 12B	Google	12B	23.9 GB	12 GB	6.73 GB	872 MB
Qwen3.6 35B A3B	Qwen	36B	71.9 GB	36 GB	20.2 GB	735 MB
Kimi K2.6	Moonshot AI	1T	595 GB	1,027 GB	578 GB	2.3 GB
Mistral Medium 3.5 128B	Mistral AI	128B	134 GB	128 GB	71.8 GB	11.8 GB
Gemma 4 26B A4B	Google	25.8B	51.6 GB	25.8 GB	14.5 GB	881 MB
Gemma 4 31B	Google	31.3B	62.5 GB	31.3 GB	17.6 GB	3.52 GB
Nemotron 3 Super 120B A12B	NVIDIA	124B	247 GB	124 GB	69.5 GB	439 MB
Qwen3.5 9B	Qwen	9.65B	19.3 GB	9.65 GB	5.43 GB	1.13 GB
Qwen3.5 122B A10B	Qwen	125B	250 GB	125 GB	70.4 GB	959 MB
Qwen3.5 397B A17B	Qwen	403B	807 GB	403 GB	227 GB	1.2 GB
Mistral Small 4 119B	Mistral AI	119B	121 GB	119 GB	67.2 GB	755 MB
GLM-4.7-Flash	Z.ai	31.2B	62.4 GB	31.2 GB	17.6 GB	1.77 GB
DeepSeek V3.2	DeepSeek	685B	689 GB	685 GB	386 GB	2.81 GB
Devstral Small 2 24B	Mistral AI	24B	25.8 GB	24 GB	13.5 GB	5.37 GB
Ministral 3 14B	Mistral AI	13.9B	15.7 GB	13.9 GB	7.84 GB	5.37 GB
gpt-oss-120b	OpenAI	117B	65.2 GB	117 GB	65.7 GB	1.21 GB
gpt-oss-20b	OpenAI	20.9B	13.8 GB	20.9 GB	11.8 GB	808 MB
Mistral Small 3.2 24B	Mistral AI	24B	48 GB	24 GB	13.5 GB	5.37 GB
Llama 4 Scout	Meta	109B	217 GB	109 GB	61.1 GB	2.82 GB
Llama 3.3 70B	Meta	70.6B	141 GB	70.6 GB	39.7 GB	10.7 GB
Llama 3.1 8B	Meta	8.03B	16.1 GB	8.03 GB	4.52 GB	4.29 GB

4-bit counts 4.5 bits a weight, for the scales stored beside them. Open a model for how many of each GPU hold it and how fast they serve it.

How the calculator works it out

Memory has 2 parts. The weights are every parameter at its number format: 2 bytes at 16-bit, 1 at 8-bit and about 0.56 at 4-bit, since NVIDIA's NVFP4 adds an 8-bit scale for every 16 numbers. As released, a model takes the size of its own files: some ship at 8-bit (FP8) or 4-bit (MXFP4, INT4) already. Granite 4.2 30B, 29.3B parameters, takes 58.6 GB at 16-bit.

The KV cache keeps the keys and values of every token so the model does not compute them again. For a layer that sees every token it is 2 × KV heads × head size numbers a token; Granite 4.2 30B keeps 262 KB a token, so a conversation of 8,192 tokens holds 2.15 GB, and one at its longest, 131,072 tokens, 34.4 GB. Newer models keep far less: layers with a sliding window keep only the last few hundred or thousand tokens, DeepSeek-style latent attention keeps 1 compressed vector a token, and linear attention or Mamba layers keep a fixed state whatever the length. The calculator reads each model's layers from its own config file.

vLLM takes 92% of each GPU's memory by default, so a GPU counts as 92% of its size. A model too big for 1 GPU is split across 2, 4 or 8 in a server, then across servers of 8. At 16-bit, Granite 4.2 30B needs 1 H100; at 4-bit, 1 H100.

Writing a token reads the weights it uses once, plus every conversation's cache, so memory speed sets a ceiling on tokens a second: 55 for Granite 4.2 30B at 16-bit, 180 at 4-bit. A mixture of experts reads only its active experts, which is why such models are fast for their size. Real servers reach part of this ceiling, less for small models and models split across many GPUs; with many conversations at once, the GPU's compute becomes the limit instead.

GPU memory for each model

Each page gives a model's memory at 16, 8 and 4-bit, its KV cache, and how many of each GPU hold it.

Questions

How much GPU memory do I need to run an LLM?: Its weights plus its KV cache. The weights take 2 bytes a parameter at 16-bit, 1 at 8-bit and about 0.56 at 4-bit, so Granite 4.2 30B needs 58.6 GB, 29.3 GB or 16.5 GB. The cache adds 262 KB for each token of each conversation, and the server keeps about 8% of the GPU free.
How much VRAM does a 30B model need?: About 58.6 GB at 16-bit, 29.3 GB at 8-bit and 16.5 GB at 4-bit for Granite 4.2 30B's weights, plus the KV cache: 2.15 GB for 1 conversation of 8,192 tokens. At 16-bit that is 1 H100; at 4-bit, 1 H100, or 1 GeForce RTX 5090.
How is the KV cache size calculated?: 2 (keys and values) × layers × KV heads × head size × bytes per number, for each token. Granite 4.2 30B has 64 layers that each keep 2,048 numbers a token, at 2 bytes: 262 KB a token. Multiply by the tokens in a conversation and by the conversations at once. Sliding window, latent and linear attention layers keep less, and an 8-bit cache halves it.
Does a mixture-of-experts model need less memory?: No. Every expert must be in memory, so a mixture of experts needs memory for all its parameters. It is faster, though: each token reads only the experts it uses, so a model with 3 billion active parameters writes tokens about as fast as a 3-billion-parameter model on the same GPU.
Is 4-bit quantization good enough?: Often. A 4-bit model needs about a quarter of the memory of 16-bit and reads a quarter of the bytes for each token, so it runs faster too. Some models ship at 4-bit (gpt-oss in MXFP4, Kimi K2.6 in INT4) and are trained for it; for others, test the 4-bit build on your own tasks before relying on it.
How fast will the model run on my GPU?: At most the GPU's memory bandwidth divided by the bytes read for each token. Granite 4.2 30B at 4-bit on an H100 can write at most 180 tokens a second for 1 conversation. Real servers reach part of that, and with many conversations at once each gets less while the total grows.
Can I run a model across several GPUs?: Yes. vLLM splits a model across the GPUs of a server with tensor parallelism, and across servers with pipeline parallelism. The memory adds up, so 2 GPUs of 80 GB hold a model of about 147 GB, but each GPU still needs room for its share of the cache.

Run it in your terminal

The same sums and models as a command you run with npx and a JavaScript library, free under the MIT licence. It prints the memory and, for each GPU, how many hold the model and the most tokens a second.

npx llm-gpu-memory "granite 4.2 30b" --context 32k --users 8

Put this calculator on your own site

Copy the code into any web page: a blog post, a course page or your team's wiki. The calculator runs in a frame and shows the same figures as this page, so when the prices change here they change on your page too. It is free and needs no account, and nothing a reader types is sent anywhere.

The code opens the calculator with the settings you have chosen above.

The short script at the end lets the frame grow and shrink with the calculator. Where a page cannot take scripts, choose a fixed height and the frame scrolls.

<iframe src="https://tecyfy.com/embed/gpu-memory-calculator" title="GPU memory calculator" width="100%" height="2400" style="border:0;display:block" loading="lazy" allow="clipboard-write"></iframe>
<p><a href="https://tecyfy.com/ai-agents/gpu-memory-calculator">GPU memory calculator</a> by Tecyfy</p>
<script>addEventListener("message",function(e){if(e.origin!=="https://tecyfy.com"||!e.data||e.data.type!=="tecyfy:height")return;var h=Number(e.data.height);if(!(h>0))return;document.querySelectorAll("iframe").forEach(function(f){if(f.contentWindow===e.source)f.style.height=Math.min(h,20000)+"px"})})</script>

Sources