How much GPU memory does Kimi K2.6 need?
Kimi K2.6 by Moonshot AI has 1T parameters, 32B of them active for each token. Its weights take 2,054 GB at 16-bit, 1,027 GB at 8-bit and 578 GB at 4-bit; released in INT4, its files are 595 GB. Each token of a conversation adds 70.3 KB of KV cache, so a 32K conversation holds 2.3 GB.
Moonshot AI · released in INT4
Kimi K2.6 at a glance
- Parameters
- 1T
- 32B active for each token
- As released
- 595 GB
- Its INT4 files
- At 16-bit
- 2,054 GB
- At 4-bit
- 578 GB
- NVFP4
- KV cache a token
- 70.3 KB
- At 16-bit
- Longest context
- 262,144
- Tokens
GPUs needed at each number format
For 1 conversation of 32,768 tokens with a 16-bit KV cache, counting 92% of each GPU's memory as vLLM does. Under each count, the most tokens a second memory allows.
| GPU | Memory | 16-bit | 8-bit | 4-bit |
|---|---|---|---|---|
| L4 | 24 GB | 96 GPUs36 tokens/s | 48 GPUs70 tokens/s | 32 GPUs118 tokens/s |
| GeForce RTX 5090 | 32 GB | 72 GPUs216 tokens/s | 40 GPUs418 tokens/s | 24 GPUs706 tokens/s |
| L40S | 48 GB | 48 GPUs104 tokens/s | 24 GPUs202 tokens/s | 16 GPUs340 tokens/s |
| A100 80GB | 80 GB | 32 GPUs246 tokens/s | 16 GPUs476 tokens/s | 8 GPUs803 tokens/s |
| H100 | 80 GB | 32 GPUs404 tokens/s | 16 GPUs781 tokens/s | 8 GPUs1,320 tokens/s |
| RTX PRO 6000 Blackwell | 96 GB | 24 GPUs216 tokens/s | 16 GPUs418 tokens/s | 8 GPUs706 tokens/s |
| DGX Spark | 128 GB | 24 GPUs33 tokens/s | 16 GPUs64 tokens/s | 8 GPUs108 tokens/s |
| H200 | 141 GB | 16 GPUs579 tokens/s | 8 GPUs1,119 tokens/s | 8 GPUs1,891 tokens/s |
| B200 | 180 GB | 16 GPUs965 tokens/s | 8 GPUs1,866 tokens/s | 4 GPUs1,576 tokens/s |
| Instinct MI300X | 192 GB | 16 GPUs639 tokens/s | 8 GPUs1,236 tokens/s | 4 GPUs1,044 tokens/s |
| Instinct MI325X | 256 GB | 16 GPUs724 tokens/s | 8 GPUs1,399 tokens/s | 4 GPUs1,182 tokens/s |
| Instinct MI355X | 288 GB | 8 GPUs965 tokens/s | 4 GPUs933 tokens/s | 4 GPUs1,576 tokens/s |
Models of a similar size
- Qwen3.5 122B A10B125B parameters, 70.4 GB at 4-bit
- Mistral Medium 3.5 128B128B parameters, 71.8 GB at 4-bit
- Qwen3.5 397B A17B403B parameters, 227 GB at 4-bit
- DeepSeek V3.2685B parameters, 386 GB at 4-bit
- GLM-5.3753B parameters, 424 GB at 4-bit
- Qwen3.8 2.4T A95B2.4T parameters, 1,376 GB at 4-bit
Questions
- How much VRAM does Kimi K2.6 need?
- 2,054 GB for its weights at 16-bit, 1,027 GB at 8-bit and 578 GB at 4-bit, plus the KV cache: 70.3 KB for each token, 2.3 GB for a conversation of 32K tokens and 18.4 GB at its longest context of 262,144 tokens. The server also keeps about 8% of each GPU free.
- Can Kimi K2.6 run on 1 GPU?
- Not on any single GPU here: at 4-bit with a 32K conversation it needs 8 H100s.
- How many H100s does Kimi K2.6 need?
- With 1 conversation of 32K tokens, 32 H100s (4 servers of 8) at 16-bit and 8 H100s at 4-bit. Each H100 has 80 GB, of which vLLM uses 92% by default.
- Can I run Kimi K2.6 on an RTX 5090?
- The RTX 5090 has 32 GB. With a 32K conversation, Kimi K2.6 needs 24 GeForce RTX 5090s (3 servers of 8) at 4-bit and 72 GeForce RTX 5090s (9 servers of 8) at 16-bit.
- How fast does Kimi K2.6 run?
- At 4-bit on 8 H100s, memory lets 1 conversation get at most 1,320 tokens a second. Real servers reach part of that ceiling, and each conversation gets less when many run at once.
Sources
Where each fact comes from
- 1Hugging Face: moonshotai/Kimi-K2.6
- 2vLLM: Engine arguments (gpu_memory_utilization)
- 3vLLM: Hybrid KV cache manager
- 4vLLM: Parallelism and scaling
- 5NVIDIA: Introducing NVFP4
- 6NVIDIA: L4 specifications
- 7NVIDIA: GeForce RTX 5090 specifications
- 8NVIDIA: L40S specifications
- 9NVIDIA: A100 80GB specifications
- 10NVIDIA: H100 specifications
- 11NVIDIA: RTX PRO 6000 Blackwell specifications
- 12NVIDIA: DGX Spark specifications
- 13NVIDIA: H200 specifications
- 14NVIDIA: B200 specifications
- 15AMD: Instinct MI300X specifications
- 16AMD: Instinct MI325X specifications
- 17AMD: Instinct MI355X specifications