How much VRAM does Mistral Small 3.2 24B need?

48 GB for its weights at 16-bit, 24 GB at 8-bit and 13.5 GB at 4-bit, plus the KV cache: 164 KB for each token, 5.37 GB for a conversation of 32K tokens and 21.5 GB at its longest context of 131,072 tokens. The server also keeps about 8% of each GPU free.

Can Mistral Small 3.2 24B run on 1 GPU?

Yes, at 4-bit: the smallest single GPU here that holds it with a 32K conversation is the L4.

How many H100s does Mistral Small 3.2 24B need?

With 1 conversation of 32K tokens, 1 H100 at 16-bit and 1 H100 at 4-bit. Each H100 has 80 GB, of which vLLM uses 92% by default.

Can I run Mistral Small 3.2 24B on an RTX 5090?

The RTX 5090 has 32 GB. With a 32K conversation, Mistral Small 3.2 24B needs 1 GeForce RTX 5090 at 4-bit and 2 GeForce RTX 5090s at 16-bit.

How fast does Mistral Small 3.2 24B run?

At 4-bit on 1 H100, memory lets 1 conversation get at most 177 tokens a second. Real servers reach part of that ceiling, and each conversation gets less when many run at once.

How much GPU memory does Mistral Small 3.2 24B need?

Mistral Small 3.2 24B by Mistral AI has 24B parameters. Its weights take 48 GB at 16-bit, 24 GB at 8-bit and 13.5 GB at 4-bit; released in BF16, its files are 48 GB. Each token of a conversation adds 164 KB of KV cache, so a 32K conversation holds 5.37 GB.

Size Mistral Small 3.2 24B for your setup Questions

Mistral AI · released in BF16

Mistral Small 3.2 24B at a glance

Parameters: 24B
As released: 48 GB; Its BF16 files
At 16-bit: 48 GB
At 4-bit: 13.5 GB; NVFP4
KV cache a token: 164 KB; At 16-bit
Longest context: 131,072; Tokens

GPUs needed at each number format

For 1 conversation of 32,768 tokens with a 16-bit KV cache, counting 92% of each GPU's memory as vLLM does. Under each count, the most tokens a second memory allows.

GPU	Memory	16-bit	8-bit	4-bit
L4	24 GB	4 GPUs22 tokens/s	2 GPUs20 tokens/s	1 GPU16 tokens/s
GeForce RTX 5090	32 GB	2 GPUs67 tokens/s	1 GPU61 tokens/s	1 GPU95 tokens/s
L40S	48 GB	2 GPUs32 tokens/s	1 GPU29 tokens/s	1 GPU46 tokens/s
A100 80GB	80 GB	1 GPU38 tokens/s	1 GPU69 tokens/s	1 GPU108 tokens/s
H100	80 GB	1 GPU63 tokens/s	1 GPU114 tokens/s	1 GPU177 tokens/s
RTX PRO 6000 Blackwell	96 GB	1 GPU34 tokens/s	1 GPU61 tokens/s	1 GPU95 tokens/s
DGX Spark	128 GB	1 GPU5.1 tokens/s	1 GPU9.3 tokens/s	1 GPU14 tokens/s
H200	141 GB	1 GPU90 tokens/s	1 GPU163 tokens/s	1 GPU254 tokens/s
B200	180 GB	1 GPU150 tokens/s	1 GPU272 tokens/s	1 GPU424 tokens/s
Instinct MI300X	192 GB	1 GPU99 tokens/s	1 GPU180 tokens/s	1 GPU281 tokens/s
Instinct MI325X	256 GB	1 GPU112 tokens/s	1 GPU204 tokens/s	1 GPU318 tokens/s
Instinct MI355X	288 GB	1 GPU150 tokens/s	1 GPU272 tokens/s	1 GPU424 tokens/s

Models of a similar size

Every model

Questions

How much VRAM does Mistral Small 3.2 24B need?: 48 GB for its weights at 16-bit, 24 GB at 8-bit and 13.5 GB at 4-bit, plus the KV cache: 164 KB for each token, 5.37 GB for a conversation of 32K tokens and 21.5 GB at its longest context of 131,072 tokens. The server also keeps about 8% of each GPU free.
Can Mistral Small 3.2 24B run on 1 GPU?: Yes, at 4-bit: the smallest single GPU here that holds it with a 32K conversation is the L4.
How many H100s does Mistral Small 3.2 24B need?: With 1 conversation of 32K tokens, 1 H100 at 16-bit and 1 H100 at 4-bit. Each H100 has 80 GB, of which vLLM uses 92% by default.
Can I run Mistral Small 3.2 24B on an RTX 5090?: The RTX 5090 has 32 GB. With a 32K conversation, Mistral Small 3.2 24B needs 1 GeForce RTX 5090 at 4-bit and 2 GeForce RTX 5090s at 16-bit.
How fast does Mistral Small 3.2 24B run?: At 4-bit on 1 H100, memory lets 1 conversation get at most 177 tokens a second. Real servers reach part of that ceiling, and each conversation gets less when many run at once.

Sources

Mistral Small 3.2 24B at a glance

GPUs needed at each number format

Models of a similar size

Questions

Where each fact comes from