Pytorch implementation of paper: Emergent Abilities of Large Language Models

The 2022 paper “Emergent Abilities of Large Language Models” by Wei et al. (Google Research) is a foundational study in AI behavior. It defines an “emergent ability” as a capability that is not present in smaller models but is present in larger models.

Unlike loss, which scales smoothly (as seen in Chinchilla), specific tasks like multi-step math, common-sense reasoning, or code generation often show a “phase transition”—appearing suddenly once a model crosses a certain number of training FLOPs (typically around $10^{22}$ to $10^{24}$ FLOPs).

---

1. Scaling the Evaluation (Not the Model)

You don’t “implement” emergence in a layer; emergence is a property of the scale of the implementation. To study this in PyTorch, you must implement a Multi-Task Evaluation Suite that can track performance across models of different sizes (e.g., 125M, 1.3B, 13B, 175B).

The Benchmarking Logic

The paper focuses on “Few-Shot Prompting” (In-Context Learning). To replicate this, your code must handle the construction of “exemplars” before the query.

Python

import torch
import torch.nn.functional as F

class EmergenceEvaluator:
    def __init__(self, model, tokenizer):
        self.model = model
        self.tokenizer = tokenizer

    def evaluate_few_shot(self, task_description, examples, query):
        """
        Constructs a prompt: [Task] + [Examples] + [Query]
        Examples: List of (input, output) tuples
        """
        prompt = task_description + "\n\n"
        for i, o in examples:
            prompt += f"Input: {i}\nOutput: {o}\n\n"
        
        prompt += f"Input: {query}\nOutput:"
        
        inputs = self.tokenizer(prompt, return_tensors="pt").to(self.model.device)
        
        # We generate the completion and check for 'exact match' accuracy
        with torch.no_grad():
            output = self.model.generate(**inputs, max_new_tokens=20)
            
        decoded = self.tokenizer.decode(output[0], skip_special_tokens=True)
        return decoded.split("Output:")[-1].strip()

---

2. Identifying the “Elbow” in the Curve

The paper’s implementation core is the visualization of the Performance vs. Scale curve. Small models will have near-zero (random) accuracy, while large models will suddenly jump to high accuracy.

Python

import matplotlib.pyplot as plt

def plot_emergence(model_sizes_billions, accuracies, task_name):
    """
    Replicates the 'Phase Transition' plots from the paper.
    """
    plt.figure(figsize=(8, 5))
    plt.plot(model_sizes_billions, accuracies, marker='o', linestyle='-', linewidth=2)
    plt.xscale('log') # Scale laws and emergence are viewed on log scales
    plt.xlabel('Model Parameters (Log Scale)')
    plt.ylabel(f'{task_name} Accuracy')
    plt.title(f'Emergence of {task_name}')
    plt.grid(True, which="both", ls="-", alpha=0.5)
    plt.show()

# Example: 3-digit addition (Figure 2 in the paper)
sizes = [0.1, 1, 10, 100, 175] # Billions
accs  = [0.01, 0.02, 0.05, 0.45, 0.85] # Sudden jump after 10B
# plot_emergence(sizes, accs, "3-digit Addition")

---

3. Critical Emergent Techniques: Chain-of-Thought

One of the most famous emergent abilities discussed is Chain-of-Thought (CoT) reasoning. Models below a certain size (e.g., GPT-3 175B) do not benefit from CoT; in fact, it can degrade their performance. Only once they reach a specific scale does “thinking step-by-step” help them solve complex problems.

Implementation of CoT Logic

Python

def generate_cot_prompt(question):
    # The 'Emergent' trigger is the inclusion of 'Let's think step by step'
    return f"Q: {question}\nA: Let's think step by step."

# Small Model (<1B): Output is usually gibberish or repeats the prompt.
# Large Model (>100B): Output follows a logical sequence of operations.

---

4. Why Emergence is Controversial

Since the 2022 paper, researchers (e.g., Schaeffer et al., 2023) have argued that emergence might be a mirage caused by the metrics used.

• If you use Exact Match (0 or 1), it looks like a sudden jump.
• If you use Token Probability (Perplexity), the improvement is actually smooth and continuous, following the Scaling Laws we discussed previously.

Summary of Emergent Abilities

Ability	Description	Emergence Threshold
Arithmetic	Multi-digit addition/multiplication.	~10B – 100B Params
Big-Bench	Complex linguistic and logic tasks.	Variable by task complexity
CoT Reasoning	Solving word problems via intermediate steps.	~60B+ Params
Instruction Following	Obeying specific constraints in a prompt.	~10B+ Params