What is BF16 (bfloat16)?
A 16-bit number format with the same range as 32-bit floats but less precision. Half the memory of FP32, and far less likely than FP16 to overflow, so most large models are trained and shipped in it.
intermediate · Also called Bfloat16, Brain Floating Point
Open the config file of almost any recent open model and you will see "torch_dtype": "bfloat16". It is the format the weights were trained and saved in. Knowing what it is explains why a 70B model is about 140 GB to download, and why training in it rarely blows up.
How a float is built
A floating-point number has 3 parts: a sign, an exponent that sets how big the number is, and a mantissa that sets how precise it is. The standard 32-bit float (FP32) uses 8 bits of exponent and 23 of mantissa. Each number takes 4 bytes.
Halving that to 16 bits means giving something up. The 2 common 16-bit formats give up different things:
| Format | Exponent bits | Mantissa bits | Largest value | Bytes |
|---|---|---|---|---|
| FP32 | 8 | 23 | about 3.4 × 10^38 | 4 |
| FP16 | 5 | 10 | 65,504 | 2 |
| BF16 | 8 | 7 | about 3.4 × 10^38 | 2 |
FP16 keeps more precision but its range stops at 65,504. BF16 keeps all 8 exponent bits of FP32, so it covers the same huge range, and pays with a much coarser mantissa: only 2 to 3 significant decimal digits.
Why range beat precision
Google designed bfloat16 (the B is for Google Brain) for its TPUs, and the bet was that neural networks care more about range than about fine precision. In FP16, small gradients can fall below the smallest number it can hold and become zero, so FP16 training needs a trick called gradient scaling (multiplying the loss up before the backward pass) to keep them. BF16 has the range of FP32 and mostly does not. Its rounding is coarse, but training averages over so many steps that the noise washes out.
It became the usual format for training large language models.
What it means in practice
Memory: 2 bytes per ParameterParameterA number inside a model that training sets, a weight or a bias. "70B" in a model's name means about 70 billion of them.. A 7B model is about 14 GB in BF16, half of FP32.
Training: mixed-precision training does most of the arithmetic in BF16 and keeps a 32-bit master copy of the weights for the updates. In PyTorch, torch.autocast with dtype=torch.bfloat16 does this.
Running a model: BF16 is the "as released" size. QuantizationQuantizationStoring a model's numbers with less precision, such as 4 bits instead of 16, so it needs far less memory and can run on smaller hardware, at some cost in accuracy. to 8 or 4 bits shrinks it further for serving, at some cost in quality.
When to use FP16 instead
On an older GPU without BF16 support, FP16 with gradient scaling is the fallback. Otherwise choose BF16 for training. For serving, keep the format the model was released in unless you have measured that a smaller one is good enough.
Sources