Training Scaling Laws vs Inference Scaling Laws
The shift from Training Scaling Laws to Inference Scaling Laws is the biggest paradigm shift in AI as of 2026.
While training scaling asks, “How much smarter is the model if I spend $100M on GPUs to train it?”, inference scaling asks, “How much smarter is the model if I give it 10 more seconds to ‘think’ before it speaks?”
1. The Core Difference: System 1 vs. System 2
In cognitive psychology, System 1 is fast, instinctive, and emotional (e.g., “What is 2+2?”). System 2 is slower, more deliberative, and logical (e.g., “Solve for $x$ in $3x^2 + 5 = 17$”).
- Training Scaling (System 1): Focuses on “Pre-training.” It improves the model’s “gut instinct” by seeing more data.
- Inference Scaling (System 2): Focuses on “Test-time Compute.” It allows the model to use search, verification, and self-correction during the actual conversation.
2. The Math of Inference Scaling
Recent research (and models like OpenAI’s o1 or o3) shows that performance doesn’t just scale with parameters ($N$); it scales with Test-Time Compute ($C_{test}$).
The general empirical formula for inference scaling follows a similar power law to training, but the variables change:
$$P \approx A \cdot (C_{test})^\gamma$$
Where:
- $P$ is the performance (accuracy on complex tasks).
- $C_{test}$ is the compute spent during the generation (number of “thought tokens” or search branches).
- $\gamma$ is the scaling exponent for inference (which varies by task difficulty).
The “Compute-Optimal” Trade-off
A breakthrough discovery in 2024/2025 showed that a smaller model using high inference compute can often outperform a massive model using low inference compute.
Example: A 7B parameter model that “thinks” for 30 seconds can beat a 175B model that answers instantly on hard math problems.
3. Two Ways to Scale Inference
There are two primary mathematical approaches to spending compute at test-time:
A. Parallel Scaling (Best-of-N / Verifiers)
The model generates $N$ different answers in parallel. A “Reward Model” or “Verifier” then scores them and picks the best one.
- The Math: If the probability of a single answer being correct is $p$, the probability of the best-of-$N$ being correct is $1 – (1 – p)^N$.
- The Limit: This eventually plateaus because if the model doesn’t “know” the answer, generating 1,000 wrong versions won’t help.
B. Sequential Scaling (Search / Chain-of-Thought)
The model explores a “tree” of possibilities, backtracking when it hits a mistake. This is what models like o1 do.
- The Math: This uses Monte Carlo Tree Search (MCTS) or Process-based Reward Models (PRM).
- The Advantage: Unlike parallel scaling, this can solve problems the model has never seen before by logically walking through the steps.
4. Why This Matters for the Future
We are hitting a “Data Wall”—there isn’t much high-quality text left on the internet to train on. Inference scaling allows AI to keep getting smarter without needing more data. | Era | Scaling Axis | Resource Constraint |
| :— | :— | :— |
| 2020-2023 | Parameters & Data | GPU hours & Web text |
| 2024-2026+ | Inference Compute | Electricity & Real-time Latency |
The “Bitter Lesson” Updated
The “Bitter Lesson” of AI used to be that general methods that leverage compute always win.
- Old Version: More compute during training wins.
- New Version: More compute during inference wins.
Leave a Reply