How much VRAM does Llama 3.3 70B need?

141 GB for its weights at 16-bit, 70.6 GB at 8-bit and 39.7 GB at 4-bit, plus the KV cache: 328 KB for each token, 10.7 GB for a conversation of 32K tokens and 42.9 GB at its longest context of 131,072 tokens. The server also keeps about 8% of each GPU free.

Can Llama 3.3 70B run on 1 GPU?

Yes, at 4-bit: the smallest single GPU here that holds it with a 32K conversation is the A100 80GB.

How many H100s does Llama 3.3 70B need?

With 1 conversation of 32K tokens, 4 H100s at 16-bit and 1 H100 at 4-bit. Each H100 has 80 GB, of which vLLM uses 92% by default.

Can I run Llama 3.3 70B on an RTX 5090?

The RTX 5090 has 32 GB. With a 32K conversation, Llama 3.3 70B needs 2 GeForce RTX 5090s at 4-bit and 8 GeForce RTX 5090s at 16-bit.

How fast does Llama 3.3 70B run?

At 4-bit on 1 H100, memory lets 1 conversation get at most 66 tokens a second. Real servers reach part of that ceiling, and each conversation gets less when many run at once.

How much GPU memory does Llama 3.3 70B need?

Llama 3.3 70B by Meta has 70.6B parameters. Its weights take 141 GB at 16-bit, 70.6 GB at 8-bit and 39.7 GB at 4-bit; released in BF16, its files are 141 GB. Each token of a conversation adds 328 KB of KV cache, so a 32K conversation holds 10.7 GB.

Size Llama 3.3 70B for your setup Questions

Meta · released in BF16

Llama 3.3 70B at a glance

Parameters: 70.6B
As released: 141 GB; Its BF16 files
At 16-bit: 141 GB
At 4-bit: 39.7 GB; NVFP4
KV cache a token: 328 KB; At 16-bit
Longest context: 131,072; Tokens

GPUs needed at each number format

For 1 conversation of 32,768 tokens with a 16-bit KV cache, counting 92% of each GPU's memory as vLLM does. Under each count, the most tokens a second memory allows.

GPU	Memory	16-bit	8-bit	4-bit
L4	24 GB	8 GPUs16 tokens/s	4 GPUs15 tokens/s	4 GPUs24 tokens/s
GeForce RTX 5090	32 GB	8 GPUs94 tokens/s	4 GPUs88 tokens/s	2 GPUs71 tokens/s
L40S	48 GB	4 GPUs23 tokens/s	2 GPUs21 tokens/s	2 GPUs34 tokens/s
A100 80GB	80 GB	4 GPUs54 tokens/s	2 GPUs50 tokens/s	1 GPU40 tokens/s
H100	80 GB	4 GPUs88 tokens/s	2 GPUs82 tokens/s	1 GPU66 tokens/s
RTX PRO 6000 Blackwell	96 GB	2 GPUs24 tokens/s	1 GPU22 tokens/s	1 GPU36 tokens/s
DGX Spark	128 GB	2 GPUs3.6 tokens/s	1 GPU3.4 tokens/s	1 GPU5.4 tokens/s
H200	141 GB	2 GPUs63 tokens/s	1 GPU59 tokens/s	1 GPU95 tokens/s
B200	180 GB	1 GPU53 tokens/s	1 GPU98 tokens/s	1 GPU159 tokens/s
Instinct MI300X	192 GB	1 GPU35 tokens/s	1 GPU65 tokens/s	1 GPU105 tokens/s
Instinct MI325X	256 GB	1 GPU40 tokens/s	1 GPU74 tokens/s	1 GPU119 tokens/s
Instinct MI355X	288 GB	1 GPU53 tokens/s	1 GPU98 tokens/s	1 GPU159 tokens/s

Models of a similar size

Every model

Questions

How much VRAM does Llama 3.3 70B need?: 141 GB for its weights at 16-bit, 70.6 GB at 8-bit and 39.7 GB at 4-bit, plus the KV cache: 328 KB for each token, 10.7 GB for a conversation of 32K tokens and 42.9 GB at its longest context of 131,072 tokens. The server also keeps about 8% of each GPU free.
Can Llama 3.3 70B run on 1 GPU?: Yes, at 4-bit: the smallest single GPU here that holds it with a 32K conversation is the A100 80GB.
How many H100s does Llama 3.3 70B need?: With 1 conversation of 32K tokens, 4 H100s at 16-bit and 1 H100 at 4-bit. Each H100 has 80 GB, of which vLLM uses 92% by default.
Can I run Llama 3.3 70B on an RTX 5090?: The RTX 5090 has 32 GB. With a 32K conversation, Llama 3.3 70B needs 2 GeForce RTX 5090s at 4-bit and 8 GeForce RTX 5090s at 16-bit.
How fast does Llama 3.3 70B run?: At 4-bit on 1 H100, memory lets 1 conversation get at most 66 tokens a second. Real servers reach part of that ceiling, and each conversation gets less when many run at once.

Sources

Llama 3.3 70B at a glance

GPUs needed at each number format

Models of a similar size

Questions

Where each fact comes from