The 2022 paper “Training Compute-Optimal Large Language Models” (often called the Chinchilla paper) by Hoffmann et al. (DeepMind) fundamentally corrected the earlier scaling laws from OpenAI.
While OpenAI’s Kaplan et al. (2020) suggested that model size ($N$) should grow faster than data size ($D$), the Chinchilla researchers proved that for every doubling of model size, the number of training tokens should also double. This revealed that most famous models at the time (like GPT-3 and Gopher) were significantly undertrained.
1. The Chinchilla Loss Function
The paper models the final training loss $L$ as a function of parameters $N$ and tokens $D$ using a more refined power-law:
$$L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta}$$
Where:
- $E$ is the “irreducible loss” (the entropy of natural language).
- $A, B, \alpha, \beta$ are constants determined by empirical fitting.
- The Big Insight: To minimize loss for a fixed compute budget $C \approx 6ND$, the optimal ratio is roughly 20 tokens per parameter ($D/N \approx 20$).
2. PyTorch Implementation: Compute-Optimal Planner
In a real-world scenario, you don’t “implement” Chinchilla as a layer; you use it to plan your training run. Below is a tool to calculate the optimal $N$ and $D$ for any given budget of FLOPs ($C$).
Python
import torch
import numpy as np
class ChinchillaPlanner:
def __init__(self):
# Constants from the paper (Table 2, Approach 2)
self.alpha = 0.34
self.beta = 0.28
self.A = 406.4
self.B = 410.7
def get_optimal_params_and_tokens(self, compute_budget_flops):
"""
Calculates N and D that minimize loss for a given budget C.
C = 6 * N * D
"""
# Based on the paper's derivation: N_opt = G * (C/6)^a
# where a = beta / (alpha + beta)
a = self.beta / (self.alpha + self.beta)
# G is a constant factor derived from A, B, alpha, beta
G = (self.alpha * self.A / (self.beta * self.B)) ** (1 / (self.alpha + self.beta))
n_opt = G * (compute_budget_flops / 6) ** a
d_opt = compute_budget_flops / (6 * n_opt)
return int(n_opt), int(d_opt)
# Example: If you have the budget of the original GPT-3 (approx 3.14e23 FLOPs)
planner = ChinchillaPlanner()
n, d = planner.get_optimal_params_and_tokens(3.14e23)
print(f"Optimal Model Size: {n/1e9:.1f}B parameters")
print(f"Optimal Data Size: {d/1e12:.2f}T tokens")
# Result: ~15B parameters and ~3.5T tokens (GPT-3 was 175B params / 0.3T tokens)
3. Implementation: LR Schedule for Long Training
A key technical finding in Chinchilla is that the cosine learning rate schedule should be decayed to a final value (usually 10% of the peak) exactly at the end of the token budget $D$.
If you train a model for more tokens than your schedule anticipated, performance degrades. This makes the max_steps parameter in your PyTorch scheduler critical.
Python
from torch.optim.lr_scheduler import CosineAnnealingLR
# If Chinchilla tells us the optimal D is 1.4 Trillion tokens:
total_tokens = 1.4e12
batch_size = 512
seq_len = 2048
total_steps = int(total_tokens / (batch_size * seq_len))
optimizer = torch.optim.AdamW(model.parameters(), lr=2e-4)
# Ensure the decay ends exactly when the optimal data budget is hit
scheduler = CosineAnnealingLR(optimizer, T_max=total_steps, eta_min=2e-5)
4. Why this changed the “LLM Arms Race”
| Before Chinchilla (Kaplan Laws) | After Chinchilla |
| Focus on “Bigger is Better” (175B, 530B params). | Focus on “Better Trained” (70B, 8B params). |
| Models were trained on ~300B tokens. | Models (like Llama 3) are trained on 15T+ tokens. |
| High inference cost (massive models). | Low inference cost (smaller, smarter models). |
The “Chinchilla Point”
The 70B Chinchilla model outperformed the 280B Gopher model despite using the exact same amount of training compute. This was because Chinchilla was trained on 1.4 Trillion tokens, whereas Gopher was trained on only 300 Billion.
Leave a Reply