# What is BF16 (bfloat16)?

A 16-bit number format with the same range as 32-bit floats but less precision. Half the memory of FP32, and far less likely than FP16 to overflow, so most large models are trained and shipped in it.

- Page: https://tecyfy.com/concepts/bf16
- Subject: How AI works (https://tecyfy.com/concepts)
- Level: intermediate
- Also called: Bfloat16, Brain Floating Point

Open the config file of almost any recent open model and you will see `"torch_dtype": "bfloat16"`. It is the format the weights were trained and saved in. Knowing what it is explains why a 70B model is about 140 GB to download, and why training in it rarely blows up.

## How a float is built

A floating-point number has 3 parts: a sign, an exponent that sets how big the number is, and a mantissa that sets how precise it is. The standard 32-bit float (FP32) uses 8 bits of exponent and 23 of mantissa. Each number takes 4 bytes.

Halving that to 16 bits means giving something up. The 2 common 16-bit formats give up different things:

| Format | Exponent bits | Mantissa bits | Largest value | Bytes |
|---|---|---|---|---|
| FP32 | 8 | 23 | about 3.4 × 10^38 | 4 |
| FP16 | 5 | 10 | 65,504 | 2 |
| BF16 | 8 | 7 | about 3.4 × 10^38 | 2 |

FP16 keeps more precision but its range stops at 65,504. BF16 keeps all 8 exponent bits of FP32, so it covers the same huge range, and pays with a much coarser mantissa: only 2 to 3 significant decimal digits.

## Why range beat precision

Google designed bfloat16 (the B is for Google Brain) for its TPUs, and the bet was that neural networks care more about range than about fine precision. In FP16, small gradients can fall below the smallest number it can hold and become zero, so FP16 training needs a trick called gradient scaling (multiplying the loss up before the backward pass) to keep them. BF16 has the range of FP32 and mostly does not. Its rounding is coarse, but training averages over so many steps that the noise washes out.

It became the usual format for training large language models.

## What it means in practice

**Memory:** 2 bytes per [Parameter](https://tecyfy.com/concepts/parameter). A 7B model is about 14 GB in BF16, half of FP32.

**Training:** mixed-precision training does most of the arithmetic in BF16 and keeps a 32-bit master copy of the weights for the updates. In PyTorch, `torch.autocast` with `dtype=torch.bfloat16` does this.

**Running a model:** BF16 is the "as released" size. [Quantization](https://tecyfy.com/concepts/quantization) to 8 or 4 bits shrinks it further for serving, at some cost in quality.

## When to use FP16 instead

On an older GPU without BF16 support, FP16 with gradient scaling is the fallback. Otherwise choose BF16 for training. For serving, keep the format the model was released in unless you have measured that a smaller one is good enough.

## Related

- [Half‑Precision](https://tecyfy.com/concepts#halfprecision): A 16‑bit floating‑point format used to accelerate training and inference while reducing memory.
- [Quantization](https://tecyfy.com/concepts/quantization): Storing a model's numbers with less precision, such as 4 bits instead of 16, so it needs far less memory and can run on smaller hardware, at some cost in accuracy.
- [Parameter](https://tecyfy.com/concepts/parameter): A number inside a model that training sets, a weight or a bias. "70B" in a model's name means about 70 billion of them.
- [4‑bit Quantization](https://tecyfy.com/concepts#4-bit-quantization): Representing weights with 4‑bit integers.
- [Gradient Checkpointing](https://tecyfy.com/concepts/gradient-checkpointing): A training trick that saves GPU memory by throwing away most in-between results on the forward pass and recomputing them when the backward pass needs them.

## Sources

- [BFloat16, the secret to high performance on Cloud TPUs](https://cloud.google.com/blog/products/ai-machine-learning/bfloat16-the-secret-to-high-performance-on-cloud-tpus)
- [Automatic mixed precision package, torch.amp](https://docs.pytorch.org/docs/stable/amp.html)
- [Model training anatomy](https://huggingface.co/docs/transformers/model_memory_anatomy)