The era between 2020 and 2022 was when LLM development shifted from “art” to “predictive science.” Researchers discovered that model performance isn’t random—it follows strict mathematical laws—but that these smooth laws occasionally produce “magic” new abilities.
Here are the foundational papers that defined this period.
1. The Discovery of Scaling Laws (2020)
Before this paper, training a giant model was a massive gamble. OpenAI turned it into a math problem.
- Paper: Scaling Laws for Neural Language Models
- Authors: Jared Kaplan, Sam McCandlish, Tom Henighan, et al. (OpenAI)
- The Breakthrough: They discovered that Test Loss (error) follows a Power Law. If you increase your Compute ($C$), Dataset ($D$), or Parameters ($N$), the model’s performance improves in a perfectly straight line on a log-log plot.
- The “Kaplan Rule”: They initially argued that if you have more compute, you should put most of it into making the model bigger ($N$) rather than giving it more data ($D$).
2. The “Chinchilla” Correction (2022)
DeepMind realized that the industry (including GPT-3) was building models that were “too big” and “under-trained.”
- Paper: Training Compute-Optimal Large Language Models
- Authors: Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, et al. (DeepMind)
- The Breakthrough: They proved that for a model to be “compute-optimal,” you must scale parameters and data equally.
- The “Chinchilla Rule”: For every doubling of model size, you should also double the amount of training data. They suggested a ratio of roughly 20 tokens per 1 parameter.
- Impact: This led to the creation of smaller, “denser” models like Llama, which are much cheaper to run but just as smart as giants.
3. The Mystery of Emergence (2022)
While scaling laws predicted that models would get “better” at predicting words, they didn’t explain why models suddenly learned to do math or code.
- Paper: Emergent Abilities of Large Language Models
- Authors: Jason Wei, Yi Tay, Rishi Bommasani, et al. (Google/Stanford)
- The Breakthrough: Defined “Emergence” as an ability that is not present in small models but appears suddenly in large ones.
- The Finding: Tasks like multi-step reasoning, translation, and common-sense logic don’t improve slowly; they stay at 0% accuracy until the model hits a certain “critical mass” (often around 10–100 billion parameters), at which point accuracy spikes to 50%+.
4. The Benchmarking Giant (2022)
To prove these laws and abilities, researchers needed a massive, difficult test that a model couldn’t just “memorize.”
- Paper: Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models (known as BIG-bench)
- Authors: Aarohi Srivastava, Guy Gur-Ari, et al. (Google & 400+ contributors)
- The Breakthrough: A collaborative suite of 200+ tasks (from emoji-understanding to chess) designed to find the limits of LLMs. It provided the data that proved “Emergent Abilities” were a real phenomenon across many different model architectures.
Summary of the Era
| Year | Key Paper | Main Takeaway |
| 2020 | Kaplan (Scaling Laws) | AI performance is mathematically predictable. |
| 2022 | Chinchilla | We need more data, not just more parameters. |
| 2022 | Wei et al. (Emergence) | Big models gain “unpredictable” skills suddenly. |
| 2022 | BIG-bench | Standardized how we measure “intelligence” vs. “scale.” |
The 2024–2026 Update: The “Mirage” Debate
It’s worth noting that a newer foundational paper (“Are Emergent Abilities of Large Language Models a Mirage?”) argued that emergence might just be a trick of how we measure things. For example, if you measure “Correct vs. Incorrect,” a model looks like it suddenly “gets it.” But if you measure “Token Probability,” the model was actually improving smoothly the whole time.
Leave a Reply