What is gradient descent?

A neural network starts out as millions or billions of random numbers, its weightsWeightA trainable parameter in a neural network that scales input values and is updated during training.. Training has to nudge every one of them towards values that give good answers. Nobody can work those values out directly. Gradient descent finds them by trial and correction, one small step at a time.

The loop

Every training step does the same 4 things:

1. Run a batch of examples through the model and get its answers.
2. Score how wrong they are with a Loss FunctionLoss FunctionThe formula that scores how wrong a model's predictions are during training, as one number to push down.. One number, lower is better.
3. Work out the GradientGradientThe vector of partial derivatives of the loss function with respect to model parameters, used to update weights during training.: for each weight, which direction raises the loss and how steeply. BackpropagationBackpropagationAn algorithm for computing gradients in deep networks via the chain rule. computes this for all weights in one backward pass.
4. Move every weight a little the other way, downhill. The size of the step is the Learning RateLearning RateThe size of the step a model's weights take on each training update. Too high and training blows up, too low and it barely moves..

Then the next batch, and the next. A large language model repeats this loop hundreds of thousands of times.

The usual picture is a hiker in fog on a hillside. They cannot see the valley, but they can feel which way the ground slopes under their feet. Step downhill, feel again, step again. The loss is the height, the weights are the position, and the gradient is the slope.

Why "stochastic"

The true slope would use every training example at once, which is far too slow for a dataset of trillions of tokens. So each step uses a small random batch instead. The slope from a batch is a noisy estimate of the real one, which is where the name stochastic gradient descent (SGD) comes from. The noise turns out to be cheap and mostly harmless, and each step costs a tiny fraction of a full pass.

Where it goes wrong

The step size is the setting that matters most. Too large and the weights jump past the low point and the loss bounces or explodes. Too small and training crawls, or settles in a poor spot. The hillside also has flat plateaus and narrow ravines where a fixed step behaves badly.

That is why plain SGD is rarely used alone for language models. Optimizers such as AdamAdam OptimizerThe most used way to train neural networks. A version of gradient descent that gives every weight its own step size, from a running memory of its past gradients. keep a running memory of past gradients and give each weight its own step size. They are still gradient descent underneath: measure the slope, step against it.

What it means for you

If you fine-tune a model, the loss curve is gradient descent made visible. A curve that falls and flattens is working. One that jumps up or turns into "nan" usually means the learning rate is too high. One that barely moves after many steps usually means it is too low, or the data is not teaching anything new.