How much GPU memory does Nemotron 3.5 Lightning 30B A3B need?
Nemotron 3.5 Lightning 30B A3B by NVIDIA has 31.6B parameters, 3B of them active for each token. Its weights take 63.2 GB at 16-bit, 31.6 GB at 8-bit and 17.8 GB at 4-bit; released in BF16, its files are 65.8 GB. Each token of a conversation adds 6.14 KB of KV cache, so a 32K conversation holds 250 MB.
NVIDIA · released in BF16
Nemotron 3.5 Lightning 30B A3B at a glance
- Parameters
- 31.6B
- 3B active for each token
- As released
- 65.8 GB
- Its BF16 files
- At 16-bit
- 63.2 GB
- At 4-bit
- 17.8 GB
- NVFP4
- KV cache a token
- 6.14 KB
- At 16-bit
- Longest context
- 262,144
- Tokens
GPUs needed at each number format
For 1 conversation of 32,768 tokens with a 16-bit KV cache, counting 92% of each GPU's memory as vLLM does. Under each count, the most tokens a second memory allows.
| GPU | Memory | 16-bit | 8-bit | 4-bit |
|---|---|---|---|---|
| L4 | 24 GB | 4 GPUs192 tokens/s | 2 GPUs185 tokens/s | 1 GPU155 tokens/s |
| GeForce RTX 5090 | 32 GB | 4 GPUs1,147 tokens/s | 2 GPUs1,103 tokens/s | 1 GPU925 tokens/s |
| L40S | 48 GB | 2 GPUs276 tokens/s | 1 GPU266 tokens/s | 1 GPU446 tokens/s |
| A100 80GB | 80 GB | 1 GPU326 tokens/s | 1 GPU627 tokens/s | 1 GPU1,052 tokens/s |
| H100 | 80 GB | 1 GPU536 tokens/s | 1 GPU1,031 tokens/s | 1 GPU1,729 tokens/s |
| RTX PRO 6000 Blackwell | 96 GB | 1 GPU287 tokens/s | 1 GPU551 tokens/s | 1 GPU925 tokens/s |
| DGX Spark | 128 GB | 1 GPU44 tokens/s | 1 GPU84 tokens/s | 1 GPU141 tokens/s |
| H200 | 141 GB | 1 GPU768 tokens/s | 1 GPU1,477 tokens/s | 1 GPU2,477 tokens/s |
| B200 | 180 GB | 1 GPU1,280 tokens/s | 1 GPU2,461 tokens/s | 1 GPU4,128 tokens/s |
| Instinct MI300X | 192 GB | 1 GPU848 tokens/s | 1 GPU1,631 tokens/s | 1 GPU2,735 tokens/s |
| Instinct MI325X | 256 GB | 1 GPU960 tokens/s | 1 GPU1,846 tokens/s | 1 GPU3,096 tokens/s |
| Instinct MI355X | 288 GB | 1 GPU1,280 tokens/s | 1 GPU2,461 tokens/s | 1 GPU4,128 tokens/s |
Models of a similar size
- Gemma 4 26B A4B25.8B parameters, 14.5 GB at 4-bit
- Qwen3.8 27B27.8B parameters, 15.6 GB at 4-bit
- Granite 4.2 30B29.3B parameters, 16.5 GB at 4-bit
- GLM-4.7-Flash31.2B parameters, 17.6 GB at 4-bit
- Gemma 4 31B31.3B parameters, 17.6 GB at 4-bit
- Qwen3.6 35B A3B36B parameters, 20.2 GB at 4-bit
Questions
- How much VRAM does Nemotron 3.5 Lightning 30B A3B need?
- 63.2 GB for its weights at 16-bit, 31.6 GB at 8-bit and 17.8 GB at 4-bit, plus the KV cache: 6.14 KB for each token, 250 MB for a conversation of 32K tokens and 1.66 GB at its longest context of 262,144 tokens. The server also keeps about 8% of each GPU free.
- Can Nemotron 3.5 Lightning 30B A3B run on 1 GPU?
- Yes, at 4-bit: the smallest single GPU here that holds it with a 32K conversation is the L4.
- How many H100s does Nemotron 3.5 Lightning 30B A3B need?
- With 1 conversation of 32K tokens, 1 H100 at 16-bit and 1 H100 at 4-bit. Each H100 has 80 GB, of which vLLM uses 92% by default.
- Can I run Nemotron 3.5 Lightning 30B A3B on an RTX 5090?
- The RTX 5090 has 32 GB. With a 32K conversation, Nemotron 3.5 Lightning 30B A3B needs 1 GeForce RTX 5090 at 4-bit and 4 GeForce RTX 5090s at 16-bit.
- How fast does Nemotron 3.5 Lightning 30B A3B run?
- At 4-bit on 1 H100, memory lets 1 conversation get at most 1,729 tokens a second. Real servers reach part of that ceiling, and each conversation gets less when many run at once.
Sources
Where each fact comes from
- 1Hugging Face: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16
- 2vLLM: Engine arguments (gpu_memory_utilization)
- 3vLLM: Hybrid KV cache manager
- 4vLLM: Parallelism and scaling
- 5NVIDIA: Introducing NVFP4
- 6NVIDIA: L4 specifications
- 7NVIDIA: GeForce RTX 5090 specifications
- 8NVIDIA: L40S specifications
- 9NVIDIA: A100 80GB specifications
- 10NVIDIA: H100 specifications
- 11NVIDIA: RTX PRO 6000 Blackwell specifications
- 12NVIDIA: DGX Spark specifications
- 13NVIDIA: H200 specifications
- 14NVIDIA: B200 specifications
- 15AMD: Instinct MI300X specifications
- 16AMD: Instinct MI325X specifications
- 17AMD: Instinct MI355X specifications