What are scaling laws in AI?
The measured pattern that a language model's loss falls predictably as its parameters, training data and compute grow, used to plan how big to make a model and how long to train it.
intermediate · Also called Scaling Laws, Chinchilla Scaling, Compute-Optimal Training
Training a frontier model costs months of time on thousands of GPUs. Before spending that, a lab wants to know roughly how good the result will be. Scaling laws are how it knows: curves, measured on many small training runs, that predict the lossLoss FunctionThe formula that scores how wrong a model's predictions are during training, as one number to push down. of a much larger one.
The 2020 finding
A team at OpenAI, with Jared Kaplan as first author, trained language models across a huge range of sizes and found something surprisingly tidy. The loss fell as a smooth power law in 3 things: the number of parametersParameterA number inside a model that training sets, a weight or a bias. "70B" in a model's name means about 70 billion of them., the amount of training data, and the compute spent. Some of the trends held across more than 7 orders of magnitude, and details such as the exact shape of the network mattered little by comparison.
A power law means steady but diminishing returns. Each 10 times more compute buys about the same fixed drop in loss. It never stops paying, and it never gets cheaper.
The 2022 correction: Chinchilla
The early reading was that, with a fixed compute budget, most of it should go into a bigger model. In 2022 a DeepMind team, with Jordan Hoffmann as first author, tested over 400 models and found the balance was off. For the best result from a given compute budget, parameters and training tokens should grow together: double the model, double the data.
They proved it with Chinchilla, a 70B model trained on 4 times more data than their 280B model Gopher, at the same cost. Chinchilla beat Gopher and several larger models. Its numbers (70B parameters, 1.4 trillion tokens) are where the rule of thumb "about 20 tokens per parameter" comes from.
Why models are now "over-trained"
Chinchilla answers one question: the best model for a fixed training budget. But a popular model is run billions of times after training, and a smaller model is cheaper on every one of those requests. So labs now train small models far past the Chinchilla point. Meta trained the 8B Llama 3 on over 15 trillion tokens, about 75 times the roughly 200 billion the rule suggests, and it was still improving when they stopped.
What scaling laws do not say
They predict loss, not skills. A lower loss usually means a better model, but when a particular ability appears is much harder to forecast. They also assume data of the same quality keeps coming, and good text is finite. That is one reason labs now spend more on data quality, synthetic data and training at the time of answering (reasoning models that think longer), rather than on size alone.
Sources