# What is a loss function?

The formula that scores how wrong a model's predictions are during training, as one number to push down.

- Page: https://tecyfy.com/concepts/loss-function
- Subject: How AI works (https://tecyfy.com/concepts)
- Level: beginner
- Also called: Loss, Cost Function, Objective Function

A model cannot improve at something it cannot measure. The loss function is the measurement: after each prediction it says, as one number, how far off the model was. Training is nothing more than changing the weights to make that number smaller.

## How a language model is scored

A [language model](https://tecyfy.com/concepts/large-language-model) is trained to predict the next [Token](https://tecyfy.com/concepts/token). At each position it gives a probability to every token in its vocabulary. The loss looks only at the probability it gave to the token that actually came next.

If the model gave the right token a probability of 0.9, the loss for that position is small. If it gave 0.01, the loss is large. The formula is the negative logarithm of that probability, called cross-entropy loss, averaged over every position in the batch. A confident right answer costs almost nothing, and a confident wrong answer costs a lot, which is the behaviour you want.

A related number you will see is perplexity, which is just the cross-entropy turned back into a plain scale (e raised to the loss). A perplexity of 10 means the model is, on average, as unsure as if it were choosing evenly among 10 tokens.

## Other kinds of loss

The loss is chosen to fit the task:

- **Picking a class** (spam or not, which of 10 labels): cross-entropy, the same idea as above.
- **Predicting a number** (a price, a temperature): mean squared error, the average of the squared gaps.
- **Embeddings:** contrastive losses that pull matching pairs together and push others apart.
- **Diffusion image models:** how far the predicted noise is from the noise that was really added.

## How the loss drives learning

The loss is built to be smooth, so a small change to any weight makes a small, measurable change to the loss. That lets training compute the [Gradient](https://tecyfy.com/concepts#gradient), the direction each weight should move, and [gradient descent](https://tecyfy.com/concepts/gradient-descent) takes the step. A score that only said right or wrong would give no slope to follow.

## What it does not tell you

A low training loss means the model predicts its training text well. It does not mean the model is useful, honest or safe. That is why teams watch the loss on held-out data the model never trained on, and test the finished model on real tasks. A training loss that keeps falling while the held-out loss rises is the model memorising its training data.

## Related

- [Gradient Descent](https://tecyfy.com/concepts/gradient-descent): The way a neural network learns. Measure how wrong it is, work out which way each weight should move to be less wrong, take a small step that way, repeat.
- [Gradient](https://tecyfy.com/concepts#gradient): The vector of partial derivatives of the loss function with respect to model parameters, used to update weights during training.
- [Token](https://tecyfy.com/concepts/token): The small piece of text a language model reads and writes, often part of a word. Prices, limits and the context window are all counted in tokens.
- [Large Language Model](https://tecyfy.com/concepts/large-language-model): A model trained on a huge amount of text to predict the next token. Further training turns it into an assistant that follows instructions.
- [Optimizer](https://tecyfy.com/concepts#optimizer): An algorithm that adjusts model parameters to minimize the loss function, e.g., Adam or SGD.

## Sources

- [torch.nn.CrossEntropyLoss](https://docs.pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html)
- [Perplexity of fixed-length models](https://huggingface.co/docs/transformers/perplexity)
- [Deep Learning, chapter 6, deep feedforward networks](https://www.deeplearningbook.org/contents/mlp.html)