How much VRAM does Qwen3.5 122B A10B need?

250 GB for its weights at 16-bit, 125 GB at 8-bit and 70.4 GB at 4-bit, plus the KV cache: 24.6 KB for each token, 959 MB for a conversation of 32K tokens and 6.6 GB at its longest context of 262,144 tokens. The server also keeps about 8% of each GPU free.

Can Qwen3.5 122B A10B run on 1 GPU?

Yes, at 4-bit: the smallest single GPU here that holds it with a 32K conversation is the A100 80GB.

How many H100s does Qwen3.5 122B A10B need?

With 1 conversation of 32K tokens, 4 H100s at 16-bit and 1 H100 at 4-bit. Each H100 has 80 GB, of which vLLM uses 92% by default.

Can I run Qwen3.5 122B A10B on an RTX 5090?

The RTX 5090 has 32 GB. With a 32K conversation, Qwen3.5 122B A10B needs 4 GeForce RTX 5090s at 4-bit and 16 GeForce RTX 5090s (2 servers of 8) at 16-bit.

How fast does Qwen3.5 122B A10B run?

At 4-bit on 1 H100, memory lets 1 conversation get at most 509 tokens a second. Real servers reach part of that ceiling, and each conversation gets less when many run at once.

How much GPU memory does Qwen3.5 122B A10B need?

Qwen3.5 122B A10B by Qwen has 125B parameters, 10B of them active for each token. Its weights take 250 GB at 16-bit, 125 GB at 8-bit and 70.4 GB at 4-bit; released in BF16, its files are 250 GB. Each token of a conversation adds 24.6 KB of KV cache, so a 32K conversation holds 959 MB.

Size Qwen3.5 122B A10B for your setup Questions

Qwen · released in BF16

Qwen3.5 122B A10B at a glance

Parameters: 125B; 10B active for each token
As released: 250 GB; Its BF16 files
At 16-bit: 250 GB
At 4-bit: 70.4 GB; NVFP4
KV cache a token: 24.6 KB; At 16-bit
Longest context: 262,144; Tokens

GPUs needed at each number format

For 1 conversation of 32,768 tokens with a 16-bit KV cache, counting 92% of each GPU's memory as vLLM does. Under each count, the most tokens a second memory allows.

GPU	Memory	16-bit	8-bit	4-bit
L4	24 GB	16 GPUs115 tokens/s	8 GPUs219 tokens/s	4 GPUs182 tokens/s
GeForce RTX 5090	32 GB	16 GPUs684 tokens/s	8 GPUs1,308 tokens/s	4 GPUs1,089 tokens/s
L40S	48 GB	8 GPUs330 tokens/s	4 GPUs315 tokens/s	2 GPUs262 tokens/s
A100 80GB	80 GB	4 GPUs389 tokens/s	2 GPUs372 tokens/s	1 GPU310 tokens/s
H100	80 GB	4 GPUs639 tokens/s	2 GPUs611 tokens/s	1 GPU509 tokens/s
RTX PRO 6000 Blackwell	96 GB	4 GPUs342 tokens/s	2 GPUs327 tokens/s	1 GPU272 tokens/s
DGX Spark	128 GB	4 GPUs52 tokens/s	2 GPUs50 tokens/s	1 GPU41 tokens/s
H200	141 GB	2 GPUs458 tokens/s	1 GPU438 tokens/s	1 GPU729 tokens/s
B200	180 GB	2 GPUs763 tokens/s	1 GPU730 tokens/s	1 GPU1,215 tokens/s
Instinct MI300X	192 GB	2 GPUs506 tokens/s	1 GPU484 tokens/s	1 GPU805 tokens/s
Instinct MI325X	256 GB	2 GPUs573 tokens/s	1 GPU547 tokens/s	1 GPU911 tokens/s
Instinct MI355X	288 GB	1 GPU382 tokens/s	1 GPU730 tokens/s	1 GPU1,215 tokens/s

Models of a similar size

Every model

Questions

How much VRAM does Qwen3.5 122B A10B need?: 250 GB for its weights at 16-bit, 125 GB at 8-bit and 70.4 GB at 4-bit, plus the KV cache: 24.6 KB for each token, 959 MB for a conversation of 32K tokens and 6.6 GB at its longest context of 262,144 tokens. The server also keeps about 8% of each GPU free.
Can Qwen3.5 122B A10B run on 1 GPU?: Yes, at 4-bit: the smallest single GPU here that holds it with a 32K conversation is the A100 80GB.
How many H100s does Qwen3.5 122B A10B need?: With 1 conversation of 32K tokens, 4 H100s at 16-bit and 1 H100 at 4-bit. Each H100 has 80 GB, of which vLLM uses 92% by default.
Can I run Qwen3.5 122B A10B on an RTX 5090?: The RTX 5090 has 32 GB. With a 32K conversation, Qwen3.5 122B A10B needs 4 GeForce RTX 5090s at 4-bit and 16 GeForce RTX 5090s (2 servers of 8) at 16-bit.
How fast does Qwen3.5 122B A10B run?: At 4-bit on 1 H100, memory lets 1 conversation get at most 509 tokens a second. Real servers reach part of that ceiling, and each conversation gets less when many run at once.

Sources

Qwen3.5 122B A10B at a glance

GPUs needed at each number format

Models of a similar size

Questions

Where each fact comes from