How much VRAM does Llama 3.1 8B need?

16.1 GB for its weights at 16-bit, 8.03 GB at 8-bit and 4.52 GB at 4-bit, plus the KV cache: 131 KB for each token, 4.29 GB for a conversation of 32K tokens and 17.2 GB at its longest context of 131,072 tokens. The server also keeps about 8% of each GPU free.

Can Llama 3.1 8B run on 1 GPU?

Yes, at 4-bit: the smallest single GPU here that holds it with a 32K conversation is the L4.

How many H100s does Llama 3.1 8B need?

With 1 conversation of 32K tokens, 1 H100 at 16-bit and 1 H100 at 4-bit. Each H100 has 80 GB, of which vLLM uses 92% by default.

Can I run Llama 3.1 8B on an RTX 5090?

The RTX 5090 has 32 GB. With a 32K conversation, Llama 3.1 8B needs 1 GeForce RTX 5090 at 4-bit and 1 GeForce RTX 5090 at 16-bit.

How fast does Llama 3.1 8B run?

At 4-bit on 1 H100, memory lets 1 conversation get at most 380 tokens a second. Real servers reach part of that ceiling, and each conversation gets less when many run at once.

How much GPU memory does Llama 3.1 8B need?

Llama 3.1 8B by Meta has 8.03B parameters. Its weights take 16.1 GB at 16-bit, 8.03 GB at 8-bit and 4.52 GB at 4-bit; released in BF16, its files are 16.1 GB. Each token of a conversation adds 131 KB of KV cache, so a 32K conversation holds 4.29 GB.

Size Llama 3.1 8B for your setup Questions

Meta · released in BF16

Llama 3.1 8B at a glance

Parameters: 8.03B
As released: 16.1 GB; Its BF16 files
At 16-bit: 16.1 GB
At 4-bit: 4.52 GB; NVFP4
KV cache a token: 131 KB; At 16-bit
Longest context: 131,072; Tokens

GPUs needed at each number format

For 1 conversation of 32,768 tokens with a 16-bit KV cache, counting 92% of each GPU's memory as vLLM does. Under each count, the most tokens a second memory allows.

GPU	Memory	16-bit	8-bit	4-bit
L4	24 GB	1 GPU15 tokens/s	1 GPU24 tokens/s	1 GPU34 tokens/s
GeForce RTX 5090	32 GB	1 GPU88 tokens/s	1 GPU145 tokens/s	1 GPU203 tokens/s
L40S	48 GB	1 GPU42 tokens/s	1 GPU70 tokens/s	1 GPU98 tokens/s
A100 80GB	80 GB	1 GPU100 tokens/s	1 GPU165 tokens/s	1 GPU231 tokens/s
H100	80 GB	1 GPU165 tokens/s	1 GPU272 tokens/s	1 GPU380 tokens/s
RTX PRO 6000 Blackwell	96 GB	1 GPU88 tokens/s	1 GPU145 tokens/s	1 GPU203 tokens/s
DGX Spark	128 GB	1 GPU13 tokens/s	1 GPU22 tokens/s	1 GPU31 tokens/s
H200	141 GB	1 GPU236 tokens/s	1 GPU389 tokens/s	1 GPU545 tokens/s
B200	180 GB	1 GPU393 tokens/s	1 GPU649 tokens/s	1 GPU908 tokens/s
Instinct MI300X	192 GB	1 GPU260 tokens/s	1 GPU430 tokens/s	1 GPU601 tokens/s
Instinct MI325X	256 GB	1 GPU295 tokens/s	1 GPU487 tokens/s	1 GPU681 tokens/s
Instinct MI355X	288 GB	1 GPU393 tokens/s	1 GPU649 tokens/s	1 GPU908 tokens/s

Models of a similar size

Every model

Questions

How much VRAM does Llama 3.1 8B need?: 16.1 GB for its weights at 16-bit, 8.03 GB at 8-bit and 4.52 GB at 4-bit, plus the KV cache: 131 KB for each token, 4.29 GB for a conversation of 32K tokens and 17.2 GB at its longest context of 131,072 tokens. The server also keeps about 8% of each GPU free.
Can Llama 3.1 8B run on 1 GPU?: Yes, at 4-bit: the smallest single GPU here that holds it with a 32K conversation is the L4.
How many H100s does Llama 3.1 8B need?: With 1 conversation of 32K tokens, 1 H100 at 16-bit and 1 H100 at 4-bit. Each H100 has 80 GB, of which vLLM uses 92% by default.
Can I run Llama 3.1 8B on an RTX 5090?: The RTX 5090 has 32 GB. With a 32K conversation, Llama 3.1 8B needs 1 GeForce RTX 5090 at 4-bit and 1 GeForce RTX 5090 at 16-bit.
How fast does Llama 3.1 8B run?: At 4-bit on 1 H100, memory lets 1 conversation get at most 380 tokens a second. Real servers reach part of that ceiling, and each conversation gets less when many run at once.

Sources

Llama 3.1 8B at a glance

GPUs needed at each number format

Models of a similar size

Questions

Where each fact comes from