How much GPU memory does Nemotron 3 Super 120B A12B need?
Nemotron 3 Super 120B A12B by NVIDIA has 124B parameters, 12B of them active for each token. Its weights take 247 GB at 16-bit, 124 GB at 8-bit and 69.5 GB at 4-bit; released in BF16, its files are 247 GB. Each token of a conversation adds 8.19 KB of KV cache, so a 32K conversation holds 439 MB.
NVIDIA · released in BF16
Nemotron 3 Super 120B A12B at a glance
- Parameters
- 124B
- 12B active for each token
- As released
- 247 GB
- Its BF16 files
- At 16-bit
- 247 GB
- At 4-bit
- 69.5 GB
- NVFP4
- KV cache a token
- 8.19 KB
- At 16-bit
- Longest context
- 262,144
- Tokens
GPUs needed at each number format
For 1 conversation of 32,768 tokens with a 16-bit KV cache, counting 92% of each GPU's memory as vLLM does. Under each count, the most tokens a second memory allows.
| GPU | Memory | 16-bit | 8-bit | 4-bit |
|---|---|---|---|---|
| L4 | 24 GB | 16 GPUs98 tokens/s | 8 GPUs193 tokens/s | 4 GPUs167 tokens/s |
| GeForce RTX 5090 | 32 GB | 16 GPUs587 tokens/s | 8 GPUs1,153 tokens/s | 4 GPUs997 tokens/s |
| L40S | 48 GB | 8 GPUs283 tokens/s | 4 GPUs278 tokens/s | 2 GPUs240 tokens/s |
| A100 80GB | 80 GB | 4 GPUs334 tokens/s | 2 GPUs328 tokens/s | 1 GPU284 tokens/s |
| H100 | 80 GB | 4 GPUs548 tokens/s | 2 GPUs539 tokens/s | 1 GPU466 tokens/s |
| RTX PRO 6000 Blackwell | 96 GB | 4 GPUs293 tokens/s | 2 GPUs288 tokens/s | 1 GPU249 tokens/s |
| DGX Spark | 128 GB | 4 GPUs45 tokens/s | 2 GPUs44 tokens/s | 1 GPU38 tokens/s |
| H200 | 141 GB | 2 GPUs393 tokens/s | 1 GPU386 tokens/s | 1 GPU668 tokens/s |
| B200 | 180 GB | 2 GPUs655 tokens/s | 1 GPU643 tokens/s | 1 GPU1,113 tokens/s |
| Instinct MI300X | 192 GB | 2 GPUs434 tokens/s | 1 GPU426 tokens/s | 1 GPU737 tokens/s |
| Instinct MI325X | 256 GB | 2 GPUs491 tokens/s | 1 GPU482 tokens/s | 1 GPU835 tokens/s |
| Instinct MI355X | 288 GB | 1 GPU327 tokens/s | 1 GPU643 tokens/s | 1 GPU1,113 tokens/s |
Models of a similar size
- Llama 3.3 70B70.6B parameters, 39.7 GB at 4-bit
- Llama 4 Scout109B parameters, 61.1 GB at 4-bit
- gpt-oss-120b117B parameters, 65.7 GB at 4-bit
- Mistral Small 4 119B119B parameters, 67.2 GB at 4-bit
- Qwen3.5 122B A10B125B parameters, 70.4 GB at 4-bit
- Mistral Medium 3.5 128B128B parameters, 71.8 GB at 4-bit
Questions
- How much VRAM does Nemotron 3 Super 120B A12B need?
- 247 GB for its weights at 16-bit, 124 GB at 8-bit and 69.5 GB at 4-bit, plus the KV cache: 8.19 KB for each token, 439 MB for a conversation of 32K tokens and 2.32 GB at its longest context of 262,144 tokens. The server also keeps about 8% of each GPU free.
- Can Nemotron 3 Super 120B A12B run on 1 GPU?
- Yes, at 4-bit: the smallest single GPU here that holds it with a 32K conversation is the A100 80GB.
- How many H100s does Nemotron 3 Super 120B A12B need?
- With 1 conversation of 32K tokens, 4 H100s at 16-bit and 1 H100 at 4-bit. Each H100 has 80 GB, of which vLLM uses 92% by default.
- Can I run Nemotron 3 Super 120B A12B on an RTX 5090?
- The RTX 5090 has 32 GB. With a 32K conversation, Nemotron 3 Super 120B A12B needs 4 GeForce RTX 5090s at 4-bit and 16 GeForce RTX 5090s (2 servers of 8) at 16-bit.
- How fast does Nemotron 3 Super 120B A12B run?
- At 4-bit on 1 H100, memory lets 1 conversation get at most 466 tokens a second. Real servers reach part of that ceiling, and each conversation gets less when many run at once.
Sources
Where each fact comes from
- 1Hugging Face: nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16
- 2vLLM: Engine arguments (gpu_memory_utilization)
- 3vLLM: Hybrid KV cache manager
- 4vLLM: Parallelism and scaling
- 5NVIDIA: Introducing NVFP4
- 6NVIDIA: L4 specifications
- 7NVIDIA: GeForce RTX 5090 specifications
- 8NVIDIA: L40S specifications
- 9NVIDIA: A100 80GB specifications
- 10NVIDIA: H100 specifications
- 11NVIDIA: RTX PRO 6000 Blackwell specifications
- 12NVIDIA: DGX Spark specifications
- 13NVIDIA: H200 specifications
- 14NVIDIA: B200 specifications
- 15AMD: Instinct MI300X specifications
- 16AMD: Instinct MI325X specifications
- 17AMD: Instinct MI355X specifications