How much VRAM does Mistral Medium 3.5 128B need?

255 GB for its weights at 16-bit, 128 GB at 8-bit and 71.8 GB at 4-bit, plus the KV cache: 360 KB for each token, 11.8 GB for a conversation of 32K tokens and 94.5 GB at its longest context of 262,144 tokens. The server also keeps about 8% of each GPU free.

Can Mistral Medium 3.5 128B run on 1 GPU?

Yes, at 4-bit: the smallest single GPU here that holds it with a 32K conversation is the RTX PRO 6000 Blackwell.

How many H100s does Mistral Medium 3.5 128B need?

With 1 conversation of 32K tokens, 4 H100s at 16-bit and 2 H100s at 4-bit. Each H100 has 80 GB, of which vLLM uses 92% by default.

Can I run Mistral Medium 3.5 128B on an RTX 5090?

The RTX 5090 has 32 GB. With a 32K conversation, Mistral Medium 3.5 128B needs 4 GeForce RTX 5090s at 4-bit and 16 GeForce RTX 5090s (2 servers of 8) at 16-bit.

How fast does Mistral Medium 3.5 128B run?

At 4-bit on 2 H100s, memory lets 1 conversation get at most 80 tokens a second. Real servers reach part of that ceiling, and each conversation gets less when many run at once.

How much GPU memory does Mistral Medium 3.5 128B need?

Mistral Medium 3.5 128B by Mistral AI has 128B parameters. Its weights take 255 GB at 16-bit, 128 GB at 8-bit and 71.8 GB at 4-bit; released in FP8, its files are 134 GB. Each token of a conversation adds 360 KB of KV cache, so a 32K conversation holds 11.8 GB.

Size Mistral Medium 3.5 128B for your setup Questions

Mistral AI · released in FP8

Mistral Medium 3.5 128B at a glance

Parameters: 128B
As released: 134 GB; Its FP8 files
At 16-bit: 255 GB
At 4-bit: 71.8 GB; NVFP4
KV cache a token: 360 KB; At 16-bit
Longest context: 262,144; Tokens

GPUs needed at each number format

For 1 conversation of 32,768 tokens with a 16-bit KV cache, counting 92% of each GPU's memory as vLLM does. Under each count, the most tokens a second memory allows.

GPU	Memory	16-bit	8-bit	4-bit
L4	24 GB	16 GPUs9 tokens/s	8 GPUs17 tokens/s	4 GPUs14 tokens/s
GeForce RTX 5090	32 GB	16 GPUs54 tokens/s	8 GPUs103 tokens/s	4 GPUs86 tokens/s
L40S	48 GB	8 GPUs26 tokens/s	4 GPUs25 tokens/s	2 GPUs21 tokens/s
A100 80GB	80 GB	4 GPUs31 tokens/s	2 GPUs29 tokens/s	2 GPUs49 tokens/s
H100	80 GB	4 GPUs50 tokens/s	2 GPUs48 tokens/s	2 GPUs80 tokens/s
RTX PRO 6000 Blackwell	96 GB	4 GPUs27 tokens/s	2 GPUs26 tokens/s	1 GPU21 tokens/s
DGX Spark	128 GB	4 GPUs4.1 tokens/s	2 GPUs3.9 tokens/s	1 GPU3.3 tokens/s
H200	141 GB	4 GPUs72 tokens/s	2 GPUs69 tokens/s	1 GPU57 tokens/s
B200	180 GB	2 GPUs60 tokens/s	1 GPU57 tokens/s	1 GPU96 tokens/s
Instinct MI300X	192 GB	2 GPUs40 tokens/s	1 GPU38 tokens/s	1 GPU63 tokens/s
Instinct MI325X	256 GB	2 GPUs45 tokens/s	1 GPU43 tokens/s	1 GPU72 tokens/s
Instinct MI355X	288 GB	2 GPUs60 tokens/s	1 GPU57 tokens/s	1 GPU96 tokens/s

Models of a similar size

Every model

Questions

How much VRAM does Mistral Medium 3.5 128B need?: 255 GB for its weights at 16-bit, 128 GB at 8-bit and 71.8 GB at 4-bit, plus the KV cache: 360 KB for each token, 11.8 GB for a conversation of 32K tokens and 94.5 GB at its longest context of 262,144 tokens. The server also keeps about 8% of each GPU free.
Can Mistral Medium 3.5 128B run on 1 GPU?: Yes, at 4-bit: the smallest single GPU here that holds it with a 32K conversation is the RTX PRO 6000 Blackwell.
How many H100s does Mistral Medium 3.5 128B need?: With 1 conversation of 32K tokens, 4 H100s at 16-bit and 2 H100s at 4-bit. Each H100 has 80 GB, of which vLLM uses 92% by default.
Can I run Mistral Medium 3.5 128B on an RTX 5090?: The RTX 5090 has 32 GB. With a 32K conversation, Mistral Medium 3.5 128B needs 4 GeForce RTX 5090s at 4-bit and 16 GeForce RTX 5090s (2 servers of 8) at 16-bit.
How fast does Mistral Medium 3.5 128B run?: At 4-bit on 2 H100s, memory lets 1 conversation get at most 80 tokens a second. Real servers reach part of that ceiling, and each conversation gets less when many run at once.

Sources

Mistral Medium 3.5 128B at a glance

GPUs needed at each number format

Models of a similar size

Questions

Where each fact comes from