How much VRAM does Nemotron 3.5 Bolt 74B A9B 24GB need?

150 GB for its weights at 16-bit, 74.9 GB at 8-bit and 42.1 GB at 4-bit, plus the KV cache: 8.19 KB for each token, 439 MB for a conversation of 32K tokens and 2.32 GB at its longest context of 262,144 tokens. The server also keeps about 8% of each GPU free.

Can Nemotron 3.5 Bolt 74B A9B 24GB run on 1 GPU?

Yes, at 4-bit: the smallest single GPU here that holds it with a 32K conversation is the L40S.

How many H100s does Nemotron 3.5 Bolt 74B A9B 24GB need?

With 1 conversation of 32K tokens, 4 H100s at 16-bit and 1 H100 at 4-bit. Each H100 has 80 GB, of which vLLM uses 92% by default.

Can I run Nemotron 3.5 Bolt 74B A9B 24GB on an RTX 5090?

The RTX 5090 has 32 GB. With a 32K conversation, Nemotron 3.5 Bolt 74B A9B 24GB needs 2 GeForce RTX 5090s at 4-bit and 8 GeForce RTX 5090s at 16-bit.

How fast does Nemotron 3.5 Bolt 74B A9B 24GB run?

At 4-bit on 1 H100, memory lets 1 conversation get at most 609 tokens a second. Real servers reach part of that ceiling, and each conversation gets less when many run at once.

How much GPU memory does Nemotron 3.5 Bolt 74B A9B 24GB need?

Nemotron 3.5 Bolt 74B A9B 24GB by NVIDIA has 74.9B parameters, 9B of them active for each token. Its weights take 150 GB at 16-bit, 74.9 GB at 8-bit and 42.1 GB at 4-bit; released in 2-bit mixed (IQ2), its files are 24.1 GB. Each token of a conversation adds 8.19 KB of KV cache, so a 32K conversation holds 439 MB.

Size Nemotron 3.5 Bolt 74B A9B 24GB for your setup Questions

NVIDIA · released in 2-bit mixed (IQ2)

Nemotron 3.5 Bolt 74B A9B 24GB at a glance

Parameters: 74.9B; 9B active for each token
As released: 24.1 GB; Its 2-bit mixed (IQ2) files
At 16-bit: 150 GB
At 4-bit: 42.1 GB; NVFP4
KV cache a token: 8.19 KB; At 16-bit
Longest context: 262,144; Tokens

GPUs needed at each number format

For 1 conversation of 32,768 tokens with a 16-bit KV cache, counting 92% of each GPU's memory as vLLM does. Under each count, the most tokens a second memory allows.

GPU	Memory	16-bit	8-bit	4-bit
L4	24 GB	8 GPUs130 tokens/s	4 GPUs127 tokens/s	2 GPUs109 tokens/s
GeForce RTX 5090	32 GB	8 GPUs777 tokens/s	4 GPUs759 tokens/s	2 GPUs651 tokens/s
L40S	48 GB	4 GPUs187 tokens/s	2 GPUs183 tokens/s	1 GPU157 tokens/s
A100 80GB	80 GB	4 GPUs442 tokens/s	2 GPUs432 tokens/s	1 GPU371 tokens/s
H100	80 GB	4 GPUs727 tokens/s	2 GPUs710 tokens/s	1 GPU609 tokens/s
RTX PRO 6000 Blackwell	96 GB	2 GPUs194 tokens/s	1 GPU190 tokens/s	1 GPU326 tokens/s
DGX Spark	128 GB	2 GPUs30 tokens/s	1 GPU29 tokens/s	1 GPU50 tokens/s
H200	141 GB	2 GPUs521 tokens/s	1 GPU509 tokens/s	1 GPU873 tokens/s
B200	180 GB	1 GPU434 tokens/s	1 GPU848 tokens/s	1 GPU1,454 tokens/s
Instinct MI300X	192 GB	1 GPU287 tokens/s	1 GPU562 tokens/s	1 GPU963 tokens/s
Instinct MI325X	256 GB	1 GPU325 tokens/s	1 GPU636 tokens/s	1 GPU1,091 tokens/s
Instinct MI355X	288 GB	1 GPU434 tokens/s	1 GPU848 tokens/s	1 GPU1,454 tokens/s

Models of a similar size

Every model

Questions

How much VRAM does Nemotron 3.5 Bolt 74B A9B 24GB need?: 150 GB for its weights at 16-bit, 74.9 GB at 8-bit and 42.1 GB at 4-bit, plus the KV cache: 8.19 KB for each token, 439 MB for a conversation of 32K tokens and 2.32 GB at its longest context of 262,144 tokens. The server also keeps about 8% of each GPU free.
Can Nemotron 3.5 Bolt 74B A9B 24GB run on 1 GPU?: Yes, at 4-bit: the smallest single GPU here that holds it with a 32K conversation is the L40S.
How many H100s does Nemotron 3.5 Bolt 74B A9B 24GB need?: With 1 conversation of 32K tokens, 4 H100s at 16-bit and 1 H100 at 4-bit. Each H100 has 80 GB, of which vLLM uses 92% by default.
Can I run Nemotron 3.5 Bolt 74B A9B 24GB on an RTX 5090?: The RTX 5090 has 32 GB. With a 32K conversation, Nemotron 3.5 Bolt 74B A9B 24GB needs 2 GeForce RTX 5090s at 4-bit and 8 GeForce RTX 5090s at 16-bit.
How fast does Nemotron 3.5 Bolt 74B A9B 24GB run?: At 4-bit on 1 H100, memory lets 1 conversation get at most 609 tokens a second. Real servers reach part of that ceiling, and each conversation gets less when many run at once.

Sources

Nemotron 3.5 Bolt 74B A9B 24GB at a glance

GPUs needed at each number format

Models of a similar size

Questions

Where each fact comes from