How much VRAM does Qwen3.5 397B A17B need?

807 GB for its weights at 16-bit, 403 GB at 8-bit and 227 GB at 4-bit, plus the KV cache: 30.7 KB for each token, 1.2 GB for a conversation of 32K tokens and 8.25 GB at its longest context of 262,144 tokens. The server also keeps about 8% of each GPU free.

Can Qwen3.5 397B A17B run on 1 GPU?

Yes, at 4-bit: the smallest single GPU here that holds it with a 32K conversation is the Instinct MI325X.

How many H100s does Qwen3.5 397B A17B need?

With 1 conversation of 32K tokens, 16 H100s (2 servers of 8) at 16-bit and 4 H100s at 4-bit. Each H100 has 80 GB, of which vLLM uses 92% by default.

Can I run Qwen3.5 397B A17B on an RTX 5090?

The RTX 5090 has 32 GB. With a 32K conversation, Qwen3.5 397B A17B needs 8 GeForce RTX 5090s at 4-bit and 32 GeForce RTX 5090s (4 servers of 8) at 16-bit.

How fast does Qwen3.5 397B A17B run?

At 4-bit on 4 H100s, memory lets 1 conversation get at most 1,245 tokens a second. Real servers reach part of that ceiling, and each conversation gets less when many run at once.

How much GPU memory does Qwen3.5 397B A17B need?

Qwen3.5 397B A17B by Qwen has 403B parameters, 17B of them active for each token. Its weights take 807 GB at 16-bit, 403 GB at 8-bit and 227 GB at 4-bit; released in BF16, its files are 807 GB. Each token of a conversation adds 30.7 KB of KV cache, so a 32K conversation holds 1.2 GB.

Size Qwen3.5 397B A17B for your setup Questions

Qwen · released in BF16

Qwen3.5 397B A17B at a glance

Parameters: 403B; 17B active for each token
As released: 807 GB; Its BF16 files
At 16-bit: 807 GB
At 4-bit: 227 GB; NVFP4
KV cache a token: 30.7 KB; At 16-bit
Longest context: 262,144; Tokens

GPUs needed at each number format

For 1 conversation of 32,768 tokens with a 16-bit KV cache, counting 92% of each GPU's memory as vLLM does. Under each count, the most tokens a second memory allows.

GPU	Memory	16-bit	8-bit	4-bit
L4	24 GB	40 GPUs68 tokens/s	24 GPUs132 tokens/s	16 GPUs223 tokens/s
GeForce RTX 5090	32 GB	32 GPUs407 tokens/s	16 GPUs788 tokens/s	8 GPUs1,332 tokens/s
L40S	48 GB	24 GPUs196 tokens/s	16 GPUs380 tokens/s	8 GPUs642 tokens/s
A100 80GB	80 GB	16 GPUs463 tokens/s	8 GPUs896 tokens/s	4 GPUs758 tokens/s
H100	80 GB	16 GPUs761 tokens/s	8 GPUs1,473 tokens/s	4 GPUs1,245 tokens/s
RTX PRO 6000 Blackwell	96 GB	16 GPUs407 tokens/s	8 GPUs788 tokens/s	4 GPUs666 tokens/s
DGX Spark	128 GB	8 GPUs62 tokens/s	4 GPUs60 tokens/s	2 GPUs51 tokens/s
H200	141 GB	8 GPUs1,091 tokens/s	4 GPUs1,055 tokens/s	2 GPUs892 tokens/s
B200	180 GB	8 GPUs1,818 tokens/s	4 GPUs1,758 tokens/s	2 GPUs1,487 tokens/s
Instinct MI300X	192 GB	8 GPUs1,205 tokens/s	4 GPUs1,165 tokens/s	2 GPUs985 tokens/s
Instinct MI325X	256 GB	4 GPUs682 tokens/s	2 GPUs659 tokens/s	1 GPU558 tokens/s
Instinct MI355X	288 GB	4 GPUs909 tokens/s	2 GPUs879 tokens/s	1 GPU743 tokens/s

Models of a similar size

Every model

Questions

How much VRAM does Qwen3.5 397B A17B need?: 807 GB for its weights at 16-bit, 403 GB at 8-bit and 227 GB at 4-bit, plus the KV cache: 30.7 KB for each token, 1.2 GB for a conversation of 32K tokens and 8.25 GB at its longest context of 262,144 tokens. The server also keeps about 8% of each GPU free.
Can Qwen3.5 397B A17B run on 1 GPU?: Yes, at 4-bit: the smallest single GPU here that holds it with a 32K conversation is the Instinct MI325X.
How many H100s does Qwen3.5 397B A17B need?: With 1 conversation of 32K tokens, 16 H100s (2 servers of 8) at 16-bit and 4 H100s at 4-bit. Each H100 has 80 GB, of which vLLM uses 92% by default.
Can I run Qwen3.5 397B A17B on an RTX 5090?: The RTX 5090 has 32 GB. With a 32K conversation, Qwen3.5 397B A17B needs 8 GeForce RTX 5090s at 4-bit and 32 GeForce RTX 5090s (4 servers of 8) at 16-bit.
How fast does Qwen3.5 397B A17B run?: At 4-bit on 4 H100s, memory lets 1 conversation get at most 1,245 tokens a second. Real servers reach part of that ceiling, and each conversation gets less when many run at once.

Sources

Qwen3.5 397B A17B at a glance

GPUs needed at each number format

Models of a similar size

Questions

Where each fact comes from