Math of the Scaling Law
The “Scaling Laws” for neural language models are the empirical formulas that describe how a model’s performance improves as you increase its size, the amount of data it sees, and the compute used to train it.
The most influential work on this came from Kaplan et al. (OpenAI, 2020), followed by the Chinchilla (DeepMind, 2022) refinements.
1. The Core Variables
To understand the math, we define three primary variables:
- $N$: The number of trainable parameters in the model.
- $D$: The size of the dataset (number of tokens).
- $C$: The total amount of training compute (usually measured in FLOPs—Floating Point Operations).
The relationship between these is approximately $C \approx 6ND$.
2. The Power Law Formula
The central discovery was that the test loss ($L$)—which measures how poorly the model predicts the next word—follows a power law. As you increase $N$, $D$, or $C$, the loss decreases according to the following general form:
$$L(X) \approx \left( \frac{X_c}{X} \right)^\alpha$$
Where:
- $X$ is the variable you are scaling ($N$, $D$, or $C$).
- $X_c$ is a constant.
- $\alpha$ is the scaling exponent (the “slope” of improvement).
Crucially, this means that if you want to reduce the error by a fixed amount, you have to increase your resources exponentially.
3. OpenAI vs. DeepMind (The Chinchilla Shift)
There is a famous mathematical disagreement between two major papers regarding how to spend a “compute budget.”
The Kaplan Scaling Law (OpenAI, 2020)
Kaplan argued that model size ($N$) was the most important factor. If you have 10x more compute, you should put most of it into making the model bigger and only a little into more data.
- Result: This led to massive models trained on relatively “small” datasets (like GPT-3).
The Chinchilla Scaling Law (DeepMind, 2022)
Hoffmann et al. (DeepMind) revisited this with the “Chinchilla” model. They found that Kaplan had “under-trained” his models. They argued that $N$ and $D$ should be scaled equally.
- The Rule of Thumb: For every doubling of model size, you must also double the amount of data.
- The “Chinchilla Optimal” Point: A model is considered “compute-optimal” if it has roughly 20 tokens of data for every 1 parameter.
- Example: A 70B parameter model should be trained on at least 1.4 trillion tokens.
4. Why Scaling Laws Broke the Industry
The math of scaling laws provided a “predictive map.” Before starting a $100 million training run, researchers could train a tiny model for $100, plot the loss on a graph, and mathematically predict exactly how smart the $100M model would be.
This removed the “guesswork” from AI development and gave investors the confidence to build the massive “GPU clusters” we see today.
5. The “Critical Batch Size”
The math also dictates how to parallelize the training. There is a limit to how many GPUs you can throw at a problem before you get diminishing returns. This is calculated via the Critical Batch Size:
$$B_{crit} \approx \frac{A}{L^\beta}$$
As the model gets better (Loss $L$ decreases), the batch size must increase to keep training efficient. This is why modern LLMs are trained with batch sizes in the millions of tokens.
The New Frontier: Inference Scaling
As of 2025–2026, the focus has shifted from Training Scaling Laws to Inference Scaling Laws (often associated with OpenAI’s “o1” or “Strawberry” models). The new math suggests that if you let a model “think” longer before it answers (Search/Reasoning compute), you can get performance gains equivalent to having a much larger model.
Leave a Reply