What is FP8, and why are large models shipped in it?
An 8-bit floating-point number format, in 2 kinds, E4M3 and E5M2. It halves a model's memory against BF16, and in tests it matched 16-bit results.
advanced · Also called 8-bit floating point, E4M3, E5M2
A model's weightsWeightA trainable parameter in a neural network that scales input values and is updated during training. are numbers, and the format of each number decides how much memory the model takes. FP32 uses 4 bytes a number, BF16BF16A 16-bit number format with the same range as 32-bit floats but less precision. Half the memory of FP32, and far less likely than FP16 to overflow, so most large models are trained and shipped in it. uses 2, and FP8 uses 1. A model shipped in FP8 needs about half the memory of the same model in BF16, so it fits on fewer GPUs and moves less data for every token it writes.
E4M3 and E5M2
8 bits do not leave much room, so FP8 comes in 2 kinds that spend them differently, as set out in a 2022 paper, FP8 Formats for Deep Learning:
- E4M3: 4 bits of exponent and 3 of mantissa. More precision, less range. It gives up infinities to stretch its range a little. Mistral Large 4's weights, for one, are stored in it.
- E5M2: 5 bits of exponent and 2 of mantissa. More range, less precision, and it keeps the standard IEEE special values.
In their tests, training in FP8 matched 16-bit results on image and language models up to 175 billion parameters without changing any settings, and FP8 also worked as a post-training format for language models that int8 QuantizationQuantizationStoring a model's numbers with less precision, such as 4 bits instead of 16, so it needs far less memory and can run on smaller hardware, at some cost in accuracy. had hurt.
Why scales matter
With so few bits, numbers of very different sizes cannot share one range well. Checkpoints deal with this by storing a scale beside the numbers: Mistral Large 4's FP8 weights carry one scale for each block of 128 by 128.
What FP8 means in practice
It is how some very large open models are shipped. Mistral Large 4, with about 1.05 trillion parameters, comes as an FP8 checkpoint of about 1,061 GB, close to 1 byte a parameter; the same model in BF16 would need about twice that. Mistral's 4-bit NVFP4 version, which targets NVIDIA's Blackwell GPUs, cuts it to about 610 GB. See 4‑bit Quantization4‑bit QuantizationRepresenting weights with 4‑bit integers. for that step.
Our GPU memory calculator shows what each number format does to a model's memory on a given GPU.
Sources