Skip to the page
Tecyfy
Daily Bytes app

How much GPU memory does Nemotron 3.5 Super VL 120B A12B need?

Nemotron 3.5 Super VL 120B A12B by NVIDIA has 124B parameters, 12B of them active for each token. Its weights take 249 GB at 16-bit, 124 GB at 8-bit and 70 GB at 4-bit; released in BF16, its files are 249 GB. Each token of a conversation adds 8.19 KB of KV cache, so a 32K conversation holds 439 MB.

Size Nemotron 3.5 Super VL 120B A12B for your setupQuestions

NVIDIA · released in BF16

Nemotron 3.5 Super VL 120B A12B at a glance

Parameters
124B
12B active for each token
As released
249 GB
Its BF16 files
At 16-bit
249 GB
At 4-bit
70 GB
NVFP4
KV cache a token
8.19 KB
At 16-bit
Longest context
1,048,576
Tokens

GPUs needed at each number format

For 1 conversation of 32,768 tokens with a 16-bit KV cache, counting 92% of each GPU's memory as vLLM does. Under each count, the most tokens a second memory allows.

GPUMemory16-bit8-bit4-bit
L424 GB16 GPUs98 tokens/s8 GPUs193 tokens/s4 GPUs167 tokens/s
GeForce RTX 509032 GB16 GPUs587 tokens/s8 GPUs1,153 tokens/s4 GPUs997 tokens/s
L40S48 GB8 GPUs283 tokens/s4 GPUs278 tokens/s2 GPUs240 tokens/s
A100 80GB80 GB4 GPUs334 tokens/s2 GPUs328 tokens/s1 GPU284 tokens/s
H10080 GB4 GPUs548 tokens/s2 GPUs539 tokens/s1 GPU466 tokens/s
RTX PRO 6000 Blackwell96 GB4 GPUs293 tokens/s2 GPUs288 tokens/s1 GPU249 tokens/s
DGX Spark128 GB4 GPUs45 tokens/s2 GPUs44 tokens/s1 GPU38 tokens/s
H200141 GB2 GPUs393 tokens/s1 GPU386 tokens/s1 GPU668 tokens/s
B200180 GB2 GPUs655 tokens/s1 GPU643 tokens/s1 GPU1,113 tokens/s
Instinct MI300X192 GB2 GPUs434 tokens/s1 GPU426 tokens/s1 GPU737 tokens/s
Instinct MI325X256 GB2 GPUs491 tokens/s1 GPU482 tokens/s1 GPU835 tokens/s
Instinct MI355X288 GB1 GPU327 tokens/s1 GPU643 tokens/s1 GPU1,113 tokens/s

Models of a similar size

Every model

Questions

How much VRAM does Nemotron 3.5 Super VL 120B A12B need?
249 GB for its weights at 16-bit, 124 GB at 8-bit and 70 GB at 4-bit, plus the KV cache: 8.19 KB for each token, 439 MB for a conversation of 32K tokens and 8.76 GB at its longest context of 1,048,576 tokens. The server also keeps about 8% of each GPU free.
Can Nemotron 3.5 Super VL 120B A12B run on 1 GPU?
Yes, at 4-bit: the smallest single GPU here that holds it with a 32K conversation is the A100 80GB.
How many H100s does Nemotron 3.5 Super VL 120B A12B need?
With 1 conversation of 32K tokens, 4 H100s at 16-bit and 1 H100 at 4-bit. Each H100 has 80 GB, of which vLLM uses 92% by default.
Can I run Nemotron 3.5 Super VL 120B A12B on an RTX 5090?
The RTX 5090 has 32 GB. With a 32K conversation, Nemotron 3.5 Super VL 120B A12B needs 4 GeForce RTX 5090s at 4-bit and 16 GeForce RTX 5090s (2 servers of 8) at 16-bit.
How fast does Nemotron 3.5 Super VL 120B A12B run?
At 4-bit on 1 H100, memory lets 1 conversation get at most 466 tokens a second. Real servers reach part of that ceiling, and each conversation gets less when many run at once.

Sources

Where each fact comes from

  1. 1Hugging Face: nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16
  2. 2vLLM: Engine arguments (gpu_memory_utilization)
  3. 3vLLM: Hybrid KV cache manager
  4. 4vLLM: Parallelism and scaling
  5. 5NVIDIA: Introducing NVFP4
  6. 6NVIDIA: L4 specifications
  7. 7NVIDIA: GeForce RTX 5090 specifications
  8. 8NVIDIA: L40S specifications
  9. 9NVIDIA: A100 80GB specifications
  10. 10NVIDIA: H100 specifications
  11. 11NVIDIA: RTX PRO 6000 Blackwell specifications
  12. 12NVIDIA: DGX Spark specifications
  13. 13NVIDIA: H200 specifications
  14. 14NVIDIA: B200 specifications
  15. 15AMD: Instinct MI300X specifications
  16. 16AMD: Instinct MI325X specifications
  17. 17AMD: Instinct MI355X specifications