How much VRAM does Qwen3.6 35B A3B need?

71.9 GB for its weights at 16-bit, 36 GB at 8-bit and 20.2 GB at 4-bit, plus the KV cache: 20.5 KB for each token, 735 MB for a conversation of 32K tokens and 5.43 GB at its longest context of 262,144 tokens. The server also keeps about 8% of each GPU free.

Can Qwen3.6 35B A3B run on 1 GPU?

Yes, at 4-bit: the smallest single GPU here that holds it with a 32K conversation is the L4.

How many H100s does Qwen3.6 35B A3B need?

With 1 conversation of 32K tokens, 1 H100 at 16-bit and 1 H100 at 4-bit. Each H100 has 80 GB, of which vLLM uses 92% by default.

Can I run Qwen3.6 35B A3B on an RTX 5090?

The RTX 5090 has 32 GB. With a 32K conversation, Qwen3.6 35B A3B needs 1 GeForce RTX 5090 at 4-bit and 4 GeForce RTX 5090s at 16-bit.

How fast does Qwen3.6 35B A3B run?

At 4-bit on 1 H100, memory lets 1 conversation get at most 1,383 tokens a second. Real servers reach part of that ceiling, and each conversation gets less when many run at once.

How much GPU memory does Qwen3.6 35B A3B need?

Qwen3.6 35B A3B by Qwen has 36B parameters, 3B of them active for each token. Its weights take 71.9 GB at 16-bit, 36 GB at 8-bit and 20.2 GB at 4-bit; released in BF16, its files are 71.9 GB. Each token of a conversation adds 20.5 KB of KV cache, so a 32K conversation holds 735 MB.

Size Qwen3.6 35B A3B for your setup Questions

Qwen · released in BF16

Qwen3.6 35B A3B at a glance

Parameters: 36B; 3B active for each token
As released: 71.9 GB; Its BF16 files
At 16-bit: 71.9 GB
At 4-bit: 20.2 GB; NVFP4
KV cache a token: 20.5 KB; At 16-bit
Longest context: 262,144; Tokens

GPUs needed at each number format

For 1 conversation of 32,768 tokens with a 16-bit KV cache, counting 92% of each GPU's memory as vLLM does. Under each count, the most tokens a second memory allows.

GPU	Memory	16-bit	8-bit	4-bit
L4	24 GB	4 GPUs178 tokens/s	2 GPUs161 tokens/s	1 GPU124 tokens/s
GeForce RTX 5090	32 GB	4 GPUs1,064 tokens/s	2 GPUs959 tokens/s	1 GPU740 tokens/s
L40S	48 GB	2 GPUs257 tokens/s	1 GPU231 tokens/s	1 GPU357 tokens/s
A100 80GB	80 GB	1 GPU303 tokens/s	1 GPU546 tokens/s	1 GPU842 tokens/s
H100	80 GB	1 GPU497 tokens/s	1 GPU897 tokens/s	1 GPU1,383 tokens/s
RTX PRO 6000 Blackwell	96 GB	1 GPU266 tokens/s	1 GPU480 tokens/s	1 GPU740 tokens/s
DGX Spark	128 GB	1 GPU41 tokens/s	1 GPU73 tokens/s	1 GPU113 tokens/s
H200	141 GB	1 GPU713 tokens/s	1 GPU1,285 tokens/s	1 GPU1,981 tokens/s
B200	180 GB	1 GPU1,188 tokens/s	1 GPU2,142 tokens/s	1 GPU3,302 tokens/s
Instinct MI300X	192 GB	1 GPU787 tokens/s	1 GPU1,419 tokens/s	1 GPU2,187 tokens/s
Instinct MI325X	256 GB	1 GPU891 tokens/s	1 GPU1,606 tokens/s	1 GPU2,476 tokens/s
Instinct MI355X	288 GB	1 GPU1,188 tokens/s	1 GPU2,142 tokens/s	1 GPU3,302 tokens/s

Models of a similar size

Every model

Questions

How much VRAM does Qwen3.6 35B A3B need?: 71.9 GB for its weights at 16-bit, 36 GB at 8-bit and 20.2 GB at 4-bit, plus the KV cache: 20.5 KB for each token, 735 MB for a conversation of 32K tokens and 5.43 GB at its longest context of 262,144 tokens. The server also keeps about 8% of each GPU free.
Can Qwen3.6 35B A3B run on 1 GPU?: Yes, at 4-bit: the smallest single GPU here that holds it with a 32K conversation is the L4.
How many H100s does Qwen3.6 35B A3B need?: With 1 conversation of 32K tokens, 1 H100 at 16-bit and 1 H100 at 4-bit. Each H100 has 80 GB, of which vLLM uses 92% by default.
Can I run Qwen3.6 35B A3B on an RTX 5090?: The RTX 5090 has 32 GB. With a 32K conversation, Qwen3.6 35B A3B needs 1 GeForce RTX 5090 at 4-bit and 4 GeForce RTX 5090s at 16-bit.
How fast does Qwen3.6 35B A3B run?: At 4-bit on 1 H100, memory lets 1 conversation get at most 1,383 tokens a second. Real servers reach part of that ceiling, and each conversation gets less when many run at once.

Sources

Qwen3.6 35B A3B at a glance

GPUs needed at each number format

Models of a similar size

Questions

Where each fact comes from