# What is the Adam optimizer?

The most used way to train neural networks. A version of gradient descent that gives every weight its own step size, from a running memory of its past gradients.

- Page: https://tecyfy.com/concepts/adam-optimizer
- Subject: How AI works (https://tecyfy.com/concepts)
- Level: intermediate
- Also called: Adam, AdamW

Plain [gradient descent](https://tecyfy.com/concepts/gradient-descent) moves every weight with the same [Learning Rate](https://tecyfy.com/concepts/learning-rate). That is a problem when some weights get large, steady gradients and others get tiny, noisy ones. Adam, published by Diederik Kingma and Jimmy Ba in 2014, fixes it by giving each weight its own step size. More than a decade later it, or its variant AdamW, still trains most large language models.

## The 2 running averages

For every single weight, Adam keeps 2 numbers that it updates on each step:

- **Momentum:** a running average of the weight's recent [gradients](https://tecyfy.com/concepts#gradient). If the gradient keeps pointing the same way, momentum builds and the steps get longer. If it flips back and forth, the flips cancel out. Like a ball rolling downhill, it keeps going through small bumps.
- **Variance:** a running average of the squared gradients, which measures how large and jumpy they have been. Adam divides the step by the square root of this, so a weight with wild gradients takes careful steps and a weight with small, steady ones takes bolder steps.

The paper's suggested settings (a learning rate of 0.001, and 0.9 and 0.999 for how slowly the 2 averages forget) work well enough across so many problems that most people never change the last 2.

## AdamW

Weight decay pulls weights gently towards zero to stop them growing too large. In the original Adam it was mixed into the gradient, where the variance scaling weakened it unevenly. AdamW, from a 2017 paper by Ilya Loshchilov and Frank Hutter, applies the decay as a separate step. It generalises better, and it is the version most training code means today when it says Adam.

## What it costs in memory

Those 2 averages are stored for every [Parameter](https://tecyfy.com/concepts/parameter), in 32-bit numbers. That is 8 extra bytes per parameter on top of the weights and the gradients. For a 7-billion-parameter model, the optimizer state alone is about 56 GB, more than the 16-bit weights themselves (about 14 GB).

This is why full training needs far more GPU memory than running the same model, and why tricks exist to shrink it: 8-bit Adam from bitsandbytes cuts the state to about 2 bytes per parameter, and [LoRA](https://tecyfy.com/concepts/lora) trains so few parameters that the optimizer state almost disappears.

## When to use something else

Adam is the safe default, not a law. Large labs try newer optimizers that need less memory or fewer steps, and plain SGD with momentum is still common for some image models. If you are fine-tuning, keep Adam or AdamW and spend your effort on the learning rate and the data.

## Related

- [Optimizer](https://tecyfy.com/concepts#optimizer): An algorithm that adjusts model parameters to minimize the loss function, e.g., Adam or SGD.
- [Gradient Descent](https://tecyfy.com/concepts/gradient-descent): The way a neural network learns. Measure how wrong it is, work out which way each weight should move to be less wrong, take a small step that way, repeat.
- [Learning Rate](https://tecyfy.com/concepts/learning-rate): The size of the step a model's weights take on each training update. Too high and training blows up, too low and it barely moves.
- [Gradient](https://tecyfy.com/concepts#gradient): The vector of partial derivatives of the loss function with respect to model parameters, used to update weights during training.
- [Parameter](https://tecyfy.com/concepts/parameter): A number inside a model that training sets, a weight or a bias. "70B" in a model's name means about 70 billion of them.

## Sources

- [Adam, a method for stochastic optimization](https://arxiv.org/abs/1412.6980)
- [Decoupled weight decay regularization](https://arxiv.org/abs/1711.05101)
- [Model training anatomy](https://huggingface.co/docs/transformers/model_memory_anatomy)