What is the learning rate in machine learning?

Every training step works out which way each weight should move to reduce the error. The learning rate says how far to move. It is one number, usually small (0.001, 0.0002, 0.00001), and it decides more about whether training works than almost any other setting.

What it does

Gradient descentGradient DescentThe way a neural network learns. Measure how wrong it is, work out which way each weight should move to be less wrong, take a small step that way, repeat. gives each weight a direction, the negative of its GradientGradientThe vector of partial derivatives of the loss function with respect to model parameters, used to update weights during training.. The update is that direction multiplied by the learning rate. Double the learning rate and every step is twice as long.

Think of walking down into a valley blindfolded. With long strides you get down fast but may stride straight over the bottom and up the far side, again and again. With tiny shuffles you never overshoot, but you may still be on the slope when the day ends.

The 3 ways it shows up in a loss curve

• Too high: the loss jumps around, climbs, or turns into "nan" (not a number). The weights overshoot so far that the arithmetic breaks.
• Too low: the loss falls very slowly and flattens early, well above where it could go.
• About right: a quick fall at first, then a steady, slowing decline.

There is no universal right value. It depends on the model, the batch size, the OptimizerOptimizerAn algorithm that adjusts model parameters to minimize the loss function, e.g., Adam or SGD. and the data. The AdamAdam OptimizerThe most used way to train neural networks. A version of gradient descent that gives every weight its own step size, from a running memory of its past gradients. paper suggests 0.001 as a starting point for Adam, and many projects start there or lower and adjust.

It changes during training

Almost no large model is trained at one fixed rate. A schedule changes it as training goes:

• Warmup: start near zero and rise over the first steps. The original Transformer rose for its first 4,000 steps. Early gradients from random weights are unreliable, and big steps on them can wreck the model before it starts.
• Decay: lower the rate as training goes on, so the late steps make fine adjustments instead of big jumps. Cosine and linear decay are common shapes.

Libraries ship these as schedulers. PyTorch, for one, has them in torch.optim.lr_scheduler, including one that lowers the rate when a validation score stops improving.

When you fine-tune

Fine-tuningFine-TuningTraining an existing model a little further on your own examples so it picks up a task, a format or a style. Often a good prompt does the job first. uses much smaller learning rates than training from scratch. The model already knows a lot, and large steps can erase it. Adapter methods such as LoRALoRAA way to fine-tune a model by freezing its weights and training 2 small matrices beside each chosen layer, so a task needs a tiny file instead of a full copy of the model. train only a few small added matrices and usually take higher rates than full fine-tuning. If a fine-tuned model has forgotten general skills, a lower learning rate or fewer steps is the first thing to try.