How much GPU memory does Qwen3.8 2.4T A95B need?
Qwen3.8 2.4T A95B by Qwen has 2.4T parameters, 95B of them active for each token. Its weights take 4,892 GB at 16-bit, 2,446 GB at 8-bit and 1,376 GB at 4-bit; released in BF16, its files are 4,892 GB. Each token of a conversation adds 94.2 KB of KV cache, so a 32K conversation holds 3.67 GB.
Qwen · released in BF16
Qwen3.8 2.4T A95B at a glance
- Parameters
- 2.4T
- 95B active for each token
- As released
- 4,892 GB
- Its BF16 files
- At 16-bit
- 4,892 GB
- At 4-bit
- 1,376 GB
- NVFP4
- KV cache a token
- 94.2 KB
- At 16-bit
- Longest context
- 262,144
- Tokens
GPUs needed at each number format
For 1 conversation of 32,768 tokens with a 16-bit KV cache, counting 92% of each GPU's memory as vLLM does. Under each count, the most tokens a second memory allows.
| GPU | Memory | 16-bit | 8-bit | 4-bit |
|---|---|---|---|---|
| L4 | 24 GB | 224 GPUs12 tokens/s | 112 GPUs24 tokens/s | 64 GPUs42 tokens/s |
| GeForce RTX 5090 | 32 GB | 168 GPUs74 tokens/s | 88 GPUs145 tokens/s | 48 GPUs251 tokens/s |
| L40S | 48 GB | 112 GPUs36 tokens/s | 56 GPUs70 tokens/s | 32 GPUs121 tokens/s |
| A100 80GB | 80 GB | 72 GPUs84 tokens/s | 40 GPUs165 tokens/s | 24 GPUs286 tokens/s |
| H100 | 80 GB | 72 GPUs138 tokens/s | 40 GPUs272 tokens/s | 24 GPUs469 tokens/s |
| RTX PRO 6000 Blackwell | 96 GB | 56 GPUs74 tokens/s | 32 GPUs145 tokens/s | 16 GPUs251 tokens/s |
| DGX Spark | 128 GB | 48 GPUs11 tokens/s | 24 GPUs22 tokens/s | 16 GPUs38 tokens/s |
| H200 | 141 GB | 40 GPUs198 tokens/s | 24 GPUs389 tokens/s | 16 GPUs672 tokens/s |
| B200 | 180 GB | 32 GPUs330 tokens/s | 16 GPUs649 tokens/s | 16 GPUs1,121 tokens/s |
| Instinct MI300X | 192 GB | 32 GPUs219 tokens/s | 16 GPUs430 tokens/s | 8 GPUs742 tokens/s |
| Instinct MI325X | 256 GB | 24 GPUs248 tokens/s | 16 GPUs486 tokens/s | 8 GPUs840 tokens/s |
| Instinct MI355X | 288 GB | 24 GPUs330 tokens/s | 16 GPUs649 tokens/s | 8 GPUs1,121 tokens/s |
Models of a similar size
- Qwen3.5 122B A10B125B parameters, 70.4 GB at 4-bit
- Mistral Medium 3.5 128B128B parameters, 71.8 GB at 4-bit
- Qwen3.5 397B A17B403B parameters, 227 GB at 4-bit
- DeepSeek V3.2685B parameters, 386 GB at 4-bit
- GLM-5.3753B parameters, 424 GB at 4-bit
- Kimi K2.61T parameters, 578 GB at 4-bit
Questions
- How much VRAM does Qwen3.8 2.4T A95B need?
- 4,892 GB for its weights at 16-bit, 2,446 GB at 8-bit and 1,376 GB at 4-bit, plus the KV cache: 94.2 KB for each token, 3.67 GB for a conversation of 32K tokens and 25.3 GB at its longest context of 262,144 tokens. The server also keeps about 8% of each GPU free.
- Can Qwen3.8 2.4T A95B run on 1 GPU?
- Not on any single GPU here: at 4-bit with a 32K conversation it needs 24 H100s (3 servers of 8).
- How many H100s does Qwen3.8 2.4T A95B need?
- With 1 conversation of 32K tokens, 72 H100s (9 servers of 8) at 16-bit and 24 H100s (3 servers of 8) at 4-bit. Each H100 has 80 GB, of which vLLM uses 92% by default.
- Can I run Qwen3.8 2.4T A95B on an RTX 5090?
- The RTX 5090 has 32 GB. With a 32K conversation, Qwen3.8 2.4T A95B needs 48 GeForce RTX 5090s (6 servers of 8) at 4-bit and 168 GeForce RTX 5090s (21 servers of 8) at 16-bit.
- How fast does Qwen3.8 2.4T A95B run?
- At 4-bit on 24 H100s (3 servers of 8), memory lets 1 conversation get at most 469 tokens a second. Real servers reach part of that ceiling, and each conversation gets less when many run at once.
Sources
Where each fact comes from
- 1Hugging Face: Qwen/Qwen3.8-2.4T-A95B
- 2vLLM: Engine arguments (gpu_memory_utilization)
- 3vLLM: Hybrid KV cache manager
- 4vLLM: Parallelism and scaling
- 5NVIDIA: Introducing NVFP4
- 6NVIDIA: L4 specifications
- 7NVIDIA: GeForce RTX 5090 specifications
- 8NVIDIA: L40S specifications
- 9NVIDIA: A100 80GB specifications
- 10NVIDIA: H100 specifications
- 11NVIDIA: RTX PRO 6000 Blackwell specifications
- 12NVIDIA: DGX Spark specifications
- 13NVIDIA: H200 specifications
- 14NVIDIA: B200 specifications
- 15AMD: Instinct MI300X specifications
- 16AMD: Instinct MI325X specifications
- 17AMD: Instinct MI355X specifications