# What are scaling laws in AI? The measured pattern that a language model's loss falls predictably as its parameters, training data and compute grow, used to plan how big to make a model and how long to train it. - Page: https://tecyfy.com/concepts/scaling-law - Subject: How AI works (https://tecyfy.com/concepts) - Level: intermediate - Also called: Scaling Laws, Chinchilla Scaling, Compute-Optimal Training Training a frontier model costs months of time on thousands of GPUs. Before spending that, a lab wants to know roughly how good the result will be. Scaling laws are how it knows: curves, measured on many small training runs, that predict the [loss](https://tecyfy.com/concepts/loss-function) of a much larger one. ## The 2020 finding A team at OpenAI, with Jared Kaplan as first author, trained language models across a huge range of sizes and found something surprisingly tidy. The loss fell as a smooth power law in 3 things: the number of [parameters](https://tecyfy.com/concepts/parameter), the amount of training data, and the compute spent. Some of the trends held across more than 7 orders of magnitude, and details such as the exact shape of the network mattered little by comparison. A power law means steady but diminishing returns. Each 10 times more compute buys about the same fixed drop in loss. It never stops paying, and it never gets cheaper. ## The 2022 correction: Chinchilla The early reading was that, with a fixed compute budget, most of it should go into a bigger model. In 2022 a DeepMind team, with Jordan Hoffmann as first author, tested over 400 models and found the balance was off. For the best result from a given compute budget, parameters and training tokens should grow together: double the model, double the data. They proved it with Chinchilla, a 70B model trained on 4 times more data than their 280B model Gopher, at the same cost. Chinchilla beat Gopher and several larger models. Its numbers (70B parameters, 1.4 trillion tokens) are where the rule of thumb "about 20 tokens per parameter" comes from. ## Why models are now "over-trained" Chinchilla answers one question: the best model for a fixed training budget. But a popular model is run billions of times after training, and a smaller model is cheaper on every one of those requests. So labs now train small models far past the Chinchilla point. Meta trained the 8B Llama 3 on over 15 trillion tokens, about 75 times the roughly 200 billion the rule suggests, and it was still improving when they stopped. ## What scaling laws do not say They predict loss, not skills. A lower loss usually means a better model, but when a particular ability appears is much harder to forecast. They also assume data of the same quality keeps coming, and good text is finite. That is one reason labs now spend more on data quality, synthetic data and training at the time of answering (reasoning models that think longer), rather than on size alone. ## Related - [Parameter](https://tecyfy.com/concepts/parameter): A number inside a model that training sets, a weight or a bias. "70B" in a model's name means about 70 billion of them. - [Compute Budget](https://tecyfy.com/concepts#compute-budget): The total amount of compute resources allocated for training or inference. - [Large Language Model](https://tecyfy.com/concepts/large-language-model): A model trained on a huge amount of text to predict the next token. Further training turns it into an assistant that follows instructions. - [Loss Function](https://tecyfy.com/concepts/loss-function): The formula that scores how wrong a model's predictions are during training, as one number to push down. - [Token](https://tecyfy.com/concepts/token): The small piece of text a language model reads and writes, often part of a word. Prices, limits and the context window are all counted in tokens. ## Sources - [Scaling laws for neural language models](https://arxiv.org/abs/2001.08361) - [Training compute-optimal large language models](https://arxiv.org/abs/2203.15556) - [Introducing Meta Llama 3](https://ai.meta.com/blog/meta-llama-3/)