How much GPU memory does Nemotron 3.5 Bolt E4B need?
Nemotron 3.5 Bolt E4B by NVIDIA has 8B parameters. Its weights take 16 GB at 16-bit, 8 GB at 8-bit and 4.5 GB at 4-bit; released in BF16, its files are 16 GB. Each token of a conversation adds 16.4 KB of KV cache, so a 32K conversation holds 558 MB.
NVIDIA · released in BF16
Nemotron 3.5 Bolt E4B at a glance
- Parameters
- 8B
- As released
- 16 GB
- Its BF16 files
- At 16-bit
- 16 GB
- At 4-bit
- 4.5 GB
- NVFP4
- KV cache a token
- 16.4 KB
- At 16-bit
- Longest context
- 65,536
- Tokens
GPUs needed at each number format
For 1 conversation of 32,768 tokens with a 16-bit KV cache, counting 92% of each GPU's memory as vLLM does. Under each count, the most tokens a second memory allows.
| GPU | Memory | 16-bit | 8-bit | 4-bit |
|---|---|---|---|---|
| L4 | 24 GB | 1 GPU18 tokens/s | 1 GPU35 tokens/s | 1 GPU59 tokens/s |
| GeForce RTX 5090 | 32 GB | 1 GPU108 tokens/s | 1 GPU209 tokens/s | 1 GPU354 tokens/s |
| L40S | 48 GB | 1 GPU52 tokens/s | 1 GPU101 tokens/s | 1 GPU171 tokens/s |
| A100 80GB | 80 GB | 1 GPU123 tokens/s | 1 GPU238 tokens/s | 1 GPU403 tokens/s |
| H100 | 80 GB | 1 GPU202 tokens/s | 1 GPU392 tokens/s | 1 GPU663 tokens/s |
| RTX PRO 6000 Blackwell | 96 GB | 1 GPU108 tokens/s | 1 GPU209 tokens/s | 1 GPU354 tokens/s |
| DGX Spark | 128 GB | 1 GPU16 tokens/s | 1 GPU32 tokens/s | 1 GPU54 tokens/s |
| H200 | 141 GB | 1 GPU290 tokens/s | 1 GPU561 tokens/s | 1 GPU949 tokens/s |
| B200 | 180 GB | 1 GPU483 tokens/s | 1 GPU935 tokens/s | 1 GPU1,582 tokens/s |
| Instinct MI300X | 192 GB | 1 GPU320 tokens/s | 1 GPU620 tokens/s | 1 GPU1,048 tokens/s |
| Instinct MI325X | 256 GB | 1 GPU363 tokens/s | 1 GPU701 tokens/s | 1 GPU1,187 tokens/s |
| Instinct MI355X | 288 GB | 1 GPU483 tokens/s | 1 GPU935 tokens/s | 1 GPU1,582 tokens/s |
Models of a similar size
- Llama 3.1 8B8.03B parameters, 4.52 GB at 4-bit
- Qwen3.5 9B9.65B parameters, 5.43 GB at 4-bit
- Gemma 4 12B12B parameters, 6.73 GB at 4-bit
- Ministral 3 14B13.9B parameters, 7.84 GB at 4-bit
- gpt-oss-20b20.9B parameters, 11.8 GB at 4-bit
- Mistral Small 3.2 24B24B parameters, 13.5 GB at 4-bit
Questions
- How much VRAM does Nemotron 3.5 Bolt E4B need?
- 16 GB for its weights at 16-bit, 8 GB at 8-bit and 4.5 GB at 4-bit, plus the KV cache: 16.4 KB for each token, 558 MB for a conversation of 32K tokens and 1.09 GB at its longest context of 65,536 tokens. The server also keeps about 8% of each GPU free.
- Can Nemotron 3.5 Bolt E4B run on 1 GPU?
- Yes, at 4-bit: the smallest single GPU here that holds it with a 32K conversation is the L4.
- How many H100s does Nemotron 3.5 Bolt E4B need?
- With 1 conversation of 32K tokens, 1 H100 at 16-bit and 1 H100 at 4-bit. Each H100 has 80 GB, of which vLLM uses 92% by default.
- Can I run Nemotron 3.5 Bolt E4B on an RTX 5090?
- The RTX 5090 has 32 GB. With a 32K conversation, Nemotron 3.5 Bolt E4B needs 1 GeForce RTX 5090 at 4-bit and 1 GeForce RTX 5090 at 16-bit.
- How fast does Nemotron 3.5 Bolt E4B run?
- At 4-bit on 1 H100, memory lets 1 conversation get at most 663 tokens a second. Real servers reach part of that ceiling, and each conversation gets less when many run at once.
Sources
Where each fact comes from
- 1Hugging Face: nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16
- 2vLLM: Engine arguments (gpu_memory_utilization)
- 3vLLM: Hybrid KV cache manager
- 4vLLM: Parallelism and scaling
- 5NVIDIA: Introducing NVFP4
- 6NVIDIA: L4 specifications
- 7NVIDIA: GeForce RTX 5090 specifications
- 8NVIDIA: L40S specifications
- 9NVIDIA: A100 80GB specifications
- 10NVIDIA: H100 specifications
- 11NVIDIA: RTX PRO 6000 Blackwell specifications
- 12NVIDIA: DGX Spark specifications
- 13NVIDIA: H200 specifications
- 14NVIDIA: B200 specifications
- 15AMD: Instinct MI300X specifications
- 16AMD: Instinct MI325X specifications
- 17AMD: Instinct MI355X specifications