# What is gradient descent?

The way a neural network learns. Measure how wrong it is, work out which way each weight should move to be less wrong, take a small step that way, repeat.

- Page: https://tecyfy.com/concepts/gradient-descent
- Subject: How AI works (https://tecyfy.com/concepts)
- Level: beginner
- Also called: Stochastic Gradient Descent, SGD

A neural network starts out as millions or billions of random numbers, its [weights](https://tecyfy.com/concepts#weight). Training has to nudge every one of them towards values that give good answers. Nobody can work those values out directly. Gradient descent finds them by trial and correction, one small step at a time.

## The loop

Every training step does the same 4 things:

1. Run a batch of examples through the model and get its answers.
2. Score how wrong they are with a [Loss Function](https://tecyfy.com/concepts/loss-function). One number, lower is better.
3. Work out the [Gradient](https://tecyfy.com/concepts#gradient): for each weight, which direction raises the loss and how steeply. [Backpropagation](https://tecyfy.com/concepts#backpropagation) computes this for all weights in one backward pass.
4. Move every weight a little the other way, downhill. The size of the step is the [Learning Rate](https://tecyfy.com/concepts/learning-rate).

Then the next batch, and the next. A large language model repeats this loop hundreds of thousands of times.

The usual picture is a hiker in fog on a hillside. They cannot see the valley, but they can feel which way the ground slopes under their feet. Step downhill, feel again, step again. The loss is the height, the weights are the position, and the gradient is the slope.

## Why "stochastic"

The true slope would use every training example at once, which is far too slow for a dataset of trillions of tokens. So each step uses a small random batch instead. The slope from a batch is a noisy estimate of the real one, which is where the name stochastic gradient descent (SGD) comes from. The noise turns out to be cheap and mostly harmless, and each step costs a tiny fraction of a full pass.

## Where it goes wrong

The step size is the setting that matters most. Too large and the weights jump past the low point and the loss bounces or explodes. Too small and training crawls, or settles in a poor spot. The hillside also has flat plateaus and narrow ravines where a fixed step behaves badly.

That is why plain SGD is rarely used alone for language models. Optimizers such as [Adam](https://tecyfy.com/concepts/adam-optimizer) keep a running memory of past gradients and give each weight its own step size. They are still gradient descent underneath: measure the slope, step against it.

## What it means for you

If you fine-tune a model, the loss curve is gradient descent made visible. A curve that falls and flattens is working. One that jumps up or turns into "nan" usually means the learning rate is too high. One that barely moves after many steps usually means it is too low, or the data is not teaching anything new.

## Related

- [Gradient](https://tecyfy.com/concepts#gradient): The vector of partial derivatives of the loss function with respect to model parameters, used to update weights during training.
- [Loss Function](https://tecyfy.com/concepts/loss-function): The formula that scores how wrong a model's predictions are during training, as one number to push down.
- [Learning Rate](https://tecyfy.com/concepts/learning-rate): The size of the step a model's weights take on each training update. Too high and training blows up, too low and it barely moves.
- [Optimizer](https://tecyfy.com/concepts#optimizer): An algorithm that adjusts model parameters to minimize the loss function, e.g., Adam or SGD.
- [Adam Optimizer](https://tecyfy.com/concepts/adam-optimizer): The most used way to train neural networks. A version of gradient descent that gives every weight its own step size, from a running memory of its past gradients.
- [Backpropagation](https://tecyfy.com/concepts#backpropagation): An algorithm for computing gradients in deep networks via the chain rule.

## Sources

- [Deep Learning, chapter 4, numerical computation](https://www.deeplearningbook.org/contents/numerical.html)
- [torch.optim.SGD](https://docs.pytorch.org/docs/stable/generated/torch.optim.SGD.html)
- [Adam, a method for stochastic optimization](https://arxiv.org/abs/1412.6980)