How much VRAM does gpt-oss-20b need?

41.8 GB for its weights at 16-bit, 20.9 GB at 8-bit and 11.8 GB at 4-bit, plus the KV cache: 24.6 KB for each token, 808 MB for a conversation of 32K tokens and 3.22 GB at its longest context of 131,072 tokens. The server also keeps about 8% of each GPU free.

Can gpt-oss-20b run on 1 GPU?

Yes, at 4-bit: the smallest single GPU here that holds it with a 32K conversation is the L4.

How many H100s does gpt-oss-20b need?

With 1 conversation of 32K tokens, 1 H100 at 16-bit and 1 H100 at 4-bit. Each H100 has 80 GB, of which vLLM uses 92% by default.

Can I run gpt-oss-20b on an RTX 5090?

The RTX 5090 has 32 GB. With a 32K conversation, gpt-oss-20b needs 1 GeForce RTX 5090 at 4-bit and 2 GeForce RTX 5090s at 16-bit.

How fast does gpt-oss-20b run?

At 4-bit on 1 H100, memory lets 1 conversation get at most 1,182 tokens a second. Real servers reach part of that ceiling, and each conversation gets less when many run at once.

How much GPU memory does gpt-oss-20b need?

gpt-oss-20b by OpenAI has 20.9B parameters, 3.6B of them active for each token. Its weights take 41.8 GB at 16-bit, 20.9 GB at 8-bit and 11.8 GB at 4-bit; released in MXFP4, its files are 13.8 GB. Each token of a conversation adds 24.6 KB of KV cache, so a 32K conversation holds 808 MB.

Size gpt-oss-20b for your setup Questions

OpenAI · released in MXFP4

gpt-oss-20b at a glance

Parameters: 20.9B; 3.6B active for each token
As released: 13.8 GB; Its MXFP4 files
At 16-bit: 41.8 GB
At 4-bit: 11.8 GB; NVFP4
KV cache a token: 24.6 KB; At 16-bit
Longest context: 131,072; Tokens

GPUs needed at each number format

For 1 conversation of 32,768 tokens with a 16-bit KV cache, counting 92% of each GPU's memory as vLLM does. Under each count, the most tokens a second memory allows.

GPU	Memory	16-bit	8-bit	4-bit
L4	24 GB	2 GPUs75 tokens/s	1 GPU68 tokens/s	1 GPU106 tokens/s
GeForce RTX 5090	32 GB	2 GPUs448 tokens/s	1 GPU406 tokens/s	1 GPU632 tokens/s
L40S	48 GB	1 GPU108 tokens/s	1 GPU196 tokens/s	1 GPU305 tokens/s
A100 80GB	80 GB	1 GPU255 tokens/s	1 GPU463 tokens/s	1 GPU720 tokens/s
H100	80 GB	1 GPU418 tokens/s	1 GPU760 tokens/s	1 GPU1,182 tokens/s
RTX PRO 6000 Blackwell	96 GB	1 GPU224 tokens/s	1 GPU406 tokens/s	1 GPU632 tokens/s
DGX Spark	128 GB	1 GPU34 tokens/s	1 GPU62 tokens/s	1 GPU96 tokens/s
H200	141 GB	1 GPU599 tokens/s	1 GPU1,089 tokens/s	1 GPU1,694 tokens/s
B200	180 GB	1 GPU999 tokens/s	1 GPU1,815 tokens/s	1 GPU2,823 tokens/s
Instinct MI300X	192 GB	1 GPU662 tokens/s	1 GPU1,202 tokens/s	1 GPU1,871 tokens/s
Instinct MI325X	256 GB	1 GPU749 tokens/s	1 GPU1,361 tokens/s	1 GPU2,118 tokens/s
Instinct MI355X	288 GB	1 GPU999 tokens/s	1 GPU1,815 tokens/s	1 GPU2,823 tokens/s

Models of a similar size

Every model

Questions

How much VRAM does gpt-oss-20b need?: 41.8 GB for its weights at 16-bit, 20.9 GB at 8-bit and 11.8 GB at 4-bit, plus the KV cache: 24.6 KB for each token, 808 MB for a conversation of 32K tokens and 3.22 GB at its longest context of 131,072 tokens. The server also keeps about 8% of each GPU free.
Can gpt-oss-20b run on 1 GPU?: Yes, at 4-bit: the smallest single GPU here that holds it with a 32K conversation is the L4.
How many H100s does gpt-oss-20b need?: With 1 conversation of 32K tokens, 1 H100 at 16-bit and 1 H100 at 4-bit. Each H100 has 80 GB, of which vLLM uses 92% by default.
Can I run gpt-oss-20b on an RTX 5090?: The RTX 5090 has 32 GB. With a 32K conversation, gpt-oss-20b needs 1 GeForce RTX 5090 at 4-bit and 2 GeForce RTX 5090s at 16-bit.
How fast does gpt-oss-20b run?: At 4-bit on 1 H100, memory lets 1 conversation get at most 1,182 tokens a second. Real servers reach part of that ceiling, and each conversation gets less when many run at once.

Sources

gpt-oss-20b at a glance

GPUs needed at each number format

Models of a similar size

Questions

Where each fact comes from