How much VRAM does Mistral Small 4 119B need?

239 GB for its weights at 16-bit, 119 GB at 8-bit and 67.2 GB at 4-bit, plus the KV cache: 23 KB for each token, 755 MB for a conversation of 32K tokens and 24.2 GB at its longest context of 1,048,576 tokens. The server also keeps about 8% of each GPU free.

Can Mistral Small 4 119B run on 1 GPU?

Yes, at 4-bit: the smallest single GPU here that holds it with a 32K conversation is the A100 80GB.

How many H100s does Mistral Small 4 119B need?

With 1 conversation of 32K tokens, 4 H100s at 16-bit and 1 H100 at 4-bit. Each H100 has 80 GB, of which vLLM uses 92% by default.

Can I run Mistral Small 4 119B on an RTX 5090?

The RTX 5090 has 32 GB. With a 32K conversation, Mistral Small 4 119B needs 4 GeForce RTX 5090s at 4-bit and 16 GeForce RTX 5090s (2 servers of 8) at 16-bit.

How fast does Mistral Small 4 119B run?

At 4-bit on 1 H100, memory lets 1 conversation get at most 759 tokens a second. Real servers reach part of that ceiling, and each conversation gets less when many run at once.

How much GPU memory does Mistral Small 4 119B need?

Mistral Small 4 119B by Mistral AI has 119B parameters, 6.5B of them active for each token. Its weights take 239 GB at 16-bit, 119 GB at 8-bit and 67.2 GB at 4-bit; released in FP8, its files are 121 GB. Each token of a conversation adds 23 KB of KV cache, so a 32K conversation holds 755 MB.

Size Mistral Small 4 119B for your setup Questions

Mistral AI · released in FP8

Mistral Small 4 119B at a glance

Parameters: 119B; 6.5B active for each token
As released: 121 GB; Its FP8 files
At 16-bit: 239 GB
At 4-bit: 67.2 GB; NVFP4
KV cache a token: 23 KB; At 16-bit
Longest context: 1,048,576; Tokens

GPUs needed at each number format

For 1 conversation of 32,768 tokens with a 16-bit KV cache, counting 92% of each GPU's memory as vLLM does. Under each count, the most tokens a second memory allows.

GPU	Memory	16-bit	8-bit	4-bit
L4	24 GB	16 GPUs174 tokens/s	8 GPUs331 tokens/s	4 GPUs272 tokens/s
GeForce RTX 5090	32 GB	16 GPUs1,042 tokens/s	8 GPUs1,976 tokens/s	4 GPUs1,625 tokens/s
L40S	48 GB	8 GPUs503 tokens/s	4 GPUs476 tokens/s	2 GPUs392 tokens/s
A100 80GB	80 GB	4 GPUs593 tokens/s	2 GPUs562 tokens/s	1 GPU462 tokens/s
H100	80 GB	4 GPUs974 tokens/s	2 GPUs924 tokens/s	1 GPU759 tokens/s
RTX PRO 6000 Blackwell	96 GB	4 GPUs521 tokens/s	2 GPUs494 tokens/s	1 GPU406 tokens/s
DGX Spark	128 GB	4 GPUs79 tokens/s	2 GPUs75 tokens/s	1 GPU62 tokens/s
H200	141 GB	2 GPUs698 tokens/s	1 GPU662 tokens/s	1 GPU1,088 tokens/s
B200	180 GB	2 GPUs1,163 tokens/s	1 GPU1,103 tokens/s	1 GPU1,814 tokens/s
Instinct MI300X	192 GB	2 GPUs771 tokens/s	1 GPU731 tokens/s	1 GPU1,201 tokens/s
Instinct MI325X	256 GB	2 GPUs872 tokens/s	1 GPU827 tokens/s	1 GPU1,360 tokens/s
Instinct MI355X	288 GB	1 GPU582 tokens/s	1 GPU1,103 tokens/s	1 GPU1,814 tokens/s

Models of a similar size

Every model

Questions

How much VRAM does Mistral Small 4 119B need?: 239 GB for its weights at 16-bit, 119 GB at 8-bit and 67.2 GB at 4-bit, plus the KV cache: 23 KB for each token, 755 MB for a conversation of 32K tokens and 24.2 GB at its longest context of 1,048,576 tokens. The server also keeps about 8% of each GPU free.
Can Mistral Small 4 119B run on 1 GPU?: Yes, at 4-bit: the smallest single GPU here that holds it with a 32K conversation is the A100 80GB.
How many H100s does Mistral Small 4 119B need?: With 1 conversation of 32K tokens, 4 H100s at 16-bit and 1 H100 at 4-bit. Each H100 has 80 GB, of which vLLM uses 92% by default.
Can I run Mistral Small 4 119B on an RTX 5090?: The RTX 5090 has 32 GB. With a 32K conversation, Mistral Small 4 119B needs 4 GeForce RTX 5090s at 4-bit and 16 GeForce RTX 5090s (2 servers of 8) at 16-bit.
How fast does Mistral Small 4 119B run?: At 4-bit on 1 H100, memory lets 1 conversation get at most 759 tokens a second. Real servers reach part of that ceiling, and each conversation gets less when many run at once.

Sources

Mistral Small 4 119B at a glance

GPUs needed at each number format

Models of a similar size

Questions

Where each fact comes from