How much VRAM does Qwen3.5 9B need?

19.3 GB for its weights at 16-bit, 9.65 GB at 8-bit and 5.43 GB at 4-bit, plus the KV cache: 32.8 KB for each token, 1.13 GB for a conversation of 32K tokens and 8.64 GB at its longest context of 262,144 tokens. The server also keeps about 8% of each GPU free.

Can Qwen3.5 9B run on 1 GPU?

Yes, at 4-bit: the smallest single GPU here that holds it with a 32K conversation is the L4.

How many H100s does Qwen3.5 9B need?

With 1 conversation of 32K tokens, 1 H100 at 16-bit and 1 H100 at 4-bit. Each H100 has 80 GB, of which vLLM uses 92% by default.

Can I run Qwen3.5 9B on an RTX 5090?

The RTX 5090 has 32 GB. With a 32K conversation, Qwen3.5 9B needs 1 GeForce RTX 5090 at 4-bit and 1 GeForce RTX 5090 at 16-bit.

How fast does Qwen3.5 9B run?

At 4-bit on 1 H100, memory lets 1 conversation get at most 511 tokens a second. Real servers reach part of that ceiling, and each conversation gets less when many run at once.

How much GPU memory does Qwen3.5 9B need?

Qwen3.5 9B by Qwen has 9.65B parameters. Its weights take 19.3 GB at 16-bit, 9.65 GB at 8-bit and 5.43 GB at 4-bit; released in BF16, its files are 19.3 GB. Each token of a conversation adds 32.8 KB of KV cache, so a 32K conversation holds 1.13 GB.

Size Qwen3.5 9B for your setup Questions

Qwen · released in BF16

Qwen3.5 9B at a glance

Parameters: 9.65B
As released: 19.3 GB; Its BF16 files
At 16-bit: 19.3 GB
At 4-bit: 5.43 GB; NVFP4
KV cache a token: 32.8 KB; At 16-bit
Longest context: 262,144; Tokens

GPUs needed at each number format

For 1 conversation of 32,768 tokens with a 16-bit KV cache, counting 92% of each GPU's memory as vLLM does. Under each count, the most tokens a second memory allows.

GPU	Memory	16-bit	8-bit	4-bit
L4	24 GB	1 GPU15 tokens/s	1 GPU28 tokens/s	1 GPU46 tokens/s
GeForce RTX 5090	32 GB	1 GPU88 tokens/s	1 GPU166 tokens/s	1 GPU273 tokens/s
L40S	48 GB	1 GPU42 tokens/s	1 GPU80 tokens/s	1 GPU132 tokens/s
A100 80GB	80 GB	1 GPU100 tokens/s	1 GPU189 tokens/s	1 GPU311 tokens/s
H100	80 GB	1 GPU164 tokens/s	1 GPU311 tokens/s	1 GPU511 tokens/s
RTX PRO 6000 Blackwell	96 GB	1 GPU88 tokens/s	1 GPU166 tokens/s	1 GPU273 tokens/s
DGX Spark	128 GB	1 GPU13 tokens/s	1 GPU25 tokens/s	1 GPU42 tokens/s
H200	141 GB	1 GPU235 tokens/s	1 GPU445 tokens/s	1 GPU732 tokens/s
B200	180 GB	1 GPU392 tokens/s	1 GPU742 tokens/s	1 GPU1,220 tokens/s
Instinct MI300X	192 GB	1 GPU259 tokens/s	1 GPU492 tokens/s	1 GPU809 tokens/s
Instinct MI325X	256 GB	1 GPU294 tokens/s	1 GPU557 tokens/s	1 GPU915 tokens/s
Instinct MI355X	288 GB	1 GPU392 tokens/s	1 GPU742 tokens/s	1 GPU1,220 tokens/s

Models of a similar size

Every model

Questions

How much VRAM does Qwen3.5 9B need?: 19.3 GB for its weights at 16-bit, 9.65 GB at 8-bit and 5.43 GB at 4-bit, plus the KV cache: 32.8 KB for each token, 1.13 GB for a conversation of 32K tokens and 8.64 GB at its longest context of 262,144 tokens. The server also keeps about 8% of each GPU free.
Can Qwen3.5 9B run on 1 GPU?: Yes, at 4-bit: the smallest single GPU here that holds it with a 32K conversation is the L4.
How many H100s does Qwen3.5 9B need?: With 1 conversation of 32K tokens, 1 H100 at 16-bit and 1 H100 at 4-bit. Each H100 has 80 GB, of which vLLM uses 92% by default.
Can I run Qwen3.5 9B on an RTX 5090?: The RTX 5090 has 32 GB. With a 32K conversation, Qwen3.5 9B needs 1 GeForce RTX 5090 at 4-bit and 1 GeForce RTX 5090 at 16-bit.
How fast does Qwen3.5 9B run?: At 4-bit on 1 H100, memory lets 1 conversation get at most 511 tokens a second. Real servers reach part of that ceiling, and each conversation gets less when many run at once.

Sources

Qwen3.5 9B at a glance

GPUs needed at each number format

Models of a similar size

Questions

Where each fact comes from