How much VRAM does GLM-4.7-Flash need?

62.4 GB for its weights at 16-bit, 31.2 GB at 8-bit and 17.6 GB at 4-bit, plus the KV cache: 54.1 KB for each token, 1.77 GB for a conversation of 32K tokens and 11 GB at its longest context of 202,752 tokens. The server also keeps about 8% of each GPU free.

Can GLM-4.7-Flash run on 1 GPU?

Yes, at 4-bit: the smallest single GPU here that holds it with a 32K conversation is the L4.

How many H100s does GLM-4.7-Flash need?

With 1 conversation of 32K tokens, 1 H100 at 16-bit and 1 H100 at 4-bit. Each H100 has 80 GB, of which vLLM uses 92% by default.

Can I run GLM-4.7-Flash on an RTX 5090?

The RTX 5090 has 32 GB. With a 32K conversation, GLM-4.7-Flash needs 1 GeForce RTX 5090 at 4-bit and 4 GeForce RTX 5090s at 16-bit.

How fast does GLM-4.7-Flash run?

At 4-bit on 1 H100, memory lets 1 conversation get at most 968 tokens a second. Real servers reach part of that ceiling, and each conversation gets less when many run at once.

How much GPU memory does GLM-4.7-Flash need?

GLM-4.7-Flash by Z.ai has 31.2B parameters, 3B of them active for each token. Its weights take 62.4 GB at 16-bit, 31.2 GB at 8-bit and 17.6 GB at 4-bit; released in BF16, its files are 62.4 GB. Each token of a conversation adds 54.1 KB of KV cache, so a 32K conversation holds 1.77 GB.

Size GLM-4.7-Flash for your setup Questions

Z.ai · released in BF16

GLM-4.7-Flash at a glance

Parameters: 31.2B; 3B active for each token
As released: 62.4 GB; Its BF16 files
At 16-bit: 62.4 GB
At 4-bit: 17.6 GB; NVFP4
KV cache a token: 54.1 KB; At 16-bit
Longest context: 202,752; Tokens

GPUs needed at each number format

For 1 conversation of 32,768 tokens with a 16-bit KV cache, counting 92% of each GPU's memory as vLLM does. Under each count, the most tokens a second memory allows.

GPU	Memory	16-bit	8-bit	4-bit
L4	24 GB	4 GPUs154 tokens/s	2 GPUs126 tokens/s	1 GPU87 tokens/s
GeForce RTX 5090	32 GB	4 GPUs922 tokens/s	2 GPUs751 tokens/s	1 GPU518 tokens/s
L40S	48 GB	2 GPUs222 tokens/s	1 GPU181 tokens/s	1 GPU250 tokens/s
A100 80GB	80 GB	1 GPU262 tokens/s	1 GPU427 tokens/s	1 GPU589 tokens/s
H100	80 GB	1 GPU431 tokens/s	1 GPU702 tokens/s	1 GPU968 tokens/s
RTX PRO 6000 Blackwell	96 GB	1 GPU231 tokens/s	1 GPU375 tokens/s	1 GPU518 tokens/s
DGX Spark	128 GB	1 GPU35 tokens/s	1 GPU57 tokens/s	1 GPU79 tokens/s
H200	141 GB	1 GPU617 tokens/s	1 GPU1,005 tokens/s	1 GPU1,387 tokens/s
B200	180 GB	1 GPU1,029 tokens/s	1 GPU1,676 tokens/s	1 GPU2,311 tokens/s
Instinct MI300X	192 GB	1 GPU682 tokens/s	1 GPU1,110 tokens/s	1 GPU1,531 tokens/s
Instinct MI325X	256 GB	1 GPU772 tokens/s	1 GPU1,257 tokens/s	1 GPU1,733 tokens/s
Instinct MI355X	288 GB	1 GPU1,029 tokens/s	1 GPU1,676 tokens/s	1 GPU2,311 tokens/s

Models of a similar size

Every model

Questions

How much VRAM does GLM-4.7-Flash need?: 62.4 GB for its weights at 16-bit, 31.2 GB at 8-bit and 17.6 GB at 4-bit, plus the KV cache: 54.1 KB for each token, 1.77 GB for a conversation of 32K tokens and 11 GB at its longest context of 202,752 tokens. The server also keeps about 8% of each GPU free.
Can GLM-4.7-Flash run on 1 GPU?: Yes, at 4-bit: the smallest single GPU here that holds it with a 32K conversation is the L4.
How many H100s does GLM-4.7-Flash need?: With 1 conversation of 32K tokens, 1 H100 at 16-bit and 1 H100 at 4-bit. Each H100 has 80 GB, of which vLLM uses 92% by default.
Can I run GLM-4.7-Flash on an RTX 5090?: The RTX 5090 has 32 GB. With a 32K conversation, GLM-4.7-Flash needs 1 GeForce RTX 5090 at 4-bit and 4 GeForce RTX 5090s at 16-bit.
How fast does GLM-4.7-Flash run?: At 4-bit on 1 H100, memory lets 1 conversation get at most 968 tokens a second. Real servers reach part of that ceiling, and each conversation gets less when many run at once.

Sources

GLM-4.7-Flash at a glance

GPUs needed at each number format

Models of a similar size

Questions

Where each fact comes from