How much VRAM does Granite 4.2 30B need?

58.6 GB for its weights at 16-bit, 29.3 GB at 8-bit and 16.5 GB at 4-bit, plus the KV cache: 262 KB for each token, 8.59 GB for a conversation of 32K tokens and 34.4 GB at its longest context of 131,072 tokens. The server also keeps about 8% of each GPU free.

Can Granite 4.2 30B run on 1 GPU?

Yes, at 4-bit: the smallest single GPU here that holds it with a 32K conversation is the GeForce RTX 5090.

How many H100s does Granite 4.2 30B need?

With 1 conversation of 32K tokens, 1 H100 at 16-bit and 1 H100 at 4-bit. Each H100 has 80 GB, of which vLLM uses 92% by default.

Can I run Granite 4.2 30B on an RTX 5090?

The RTX 5090 has 32 GB. With a 32K conversation, Granite 4.2 30B needs 1 GeForce RTX 5090 at 4-bit and 4 GeForce RTX 5090s at 16-bit.

How fast does Granite 4.2 30B run?

At 4-bit on 1 H100, memory lets 1 conversation get at most 134 tokens a second. Real servers reach part of that ceiling, and each conversation gets less when many run at once.

How much GPU memory does Granite 4.2 30B need?

Granite 4.2 30B by IBM has 29.3B parameters. Its weights take 58.6 GB at 16-bit, 29.3 GB at 8-bit and 16.5 GB at 4-bit; released in BF16, its files are 58.6 GB. Each token of a conversation adds 262 KB of KV cache, so a 32K conversation holds 8.59 GB.

Size Granite 4.2 30B for your setup Questions

IBM · released in BF16

Granite 4.2 30B at a glance

Parameters: 29.3B
As released: 58.6 GB; Its BF16 files
At 16-bit: 58.6 GB
At 4-bit: 16.5 GB; NVFP4
KV cache a token: 262 KB; At 16-bit
Longest context: 131,072; Tokens

GPUs needed at each number format

For 1 conversation of 32,768 tokens with a 16-bit KV cache, counting 92% of each GPU's memory as vLLM does. Under each count, the most tokens a second memory allows.

GPU	Memory	16-bit	8-bit	4-bit
L4	24 GB	4 GPUs18 tokens/s	2 GPUs16 tokens/s	2 GPUs24 tokens/s
GeForce RTX 5090	32 GB	4 GPUs107 tokens/s	2 GPUs95 tokens/s	1 GPU72 tokens/s
L40S	48 GB	2 GPUs26 tokens/s	1 GPU23 tokens/s	1 GPU34 tokens/s
A100 80GB	80 GB	1 GPU30 tokens/s	1 GPU54 tokens/s	1 GPU81 tokens/s
H100	80 GB	1 GPU50 tokens/s	1 GPU88 tokens/s	1 GPU134 tokens/s
RTX PRO 6000 Blackwell	96 GB	1 GPU27 tokens/s	1 GPU47 tokens/s	1 GPU72 tokens/s
DGX Spark	128 GB	1 GPU4.1 tokens/s	1 GPU7.2 tokens/s	1 GPU11 tokens/s
H200	141 GB	1 GPU71 tokens/s	1 GPU127 tokens/s	1 GPU192 tokens/s
B200	180 GB	1 GPU119 tokens/s	1 GPU211 tokens/s	1 GPU319 tokens/s
Instinct MI300X	192 GB	1 GPU79 tokens/s	1 GPU140 tokens/s	1 GPU212 tokens/s
Instinct MI325X	256 GB	1 GPU89 tokens/s	1 GPU158 tokens/s	1 GPU239 tokens/s
Instinct MI355X	288 GB	1 GPU119 tokens/s	1 GPU211 tokens/s	1 GPU319 tokens/s

Models of a similar size

Every model

Questions

How much VRAM does Granite 4.2 30B need?: 58.6 GB for its weights at 16-bit, 29.3 GB at 8-bit and 16.5 GB at 4-bit, plus the KV cache: 262 KB for each token, 8.59 GB for a conversation of 32K tokens and 34.4 GB at its longest context of 131,072 tokens. The server also keeps about 8% of each GPU free.
Can Granite 4.2 30B run on 1 GPU?: Yes, at 4-bit: the smallest single GPU here that holds it with a 32K conversation is the GeForce RTX 5090.
How many H100s does Granite 4.2 30B need?: With 1 conversation of 32K tokens, 1 H100 at 16-bit and 1 H100 at 4-bit. Each H100 has 80 GB, of which vLLM uses 92% by default.
Can I run Granite 4.2 30B on an RTX 5090?: The RTX 5090 has 32 GB. With a 32K conversation, Granite 4.2 30B needs 1 GeForce RTX 5090 at 4-bit and 4 GeForce RTX 5090s at 16-bit.
How fast does Granite 4.2 30B run?: At 4-bit on 1 H100, memory lets 1 conversation get at most 134 tokens a second. Real servers reach part of that ceiling, and each conversation gets less when many run at once.

Sources

Granite 4.2 30B at a glance

GPUs needed at each number format

Models of a similar size

Questions

Where each fact comes from