Tag: ai

The 2022 paper “Emergent Abilities of Large Language Models” by Wei et al. (Google Research) is a foundational study in AI behavior. It defines an “emergent ability” as a capability that is not present in smaller models but is present in larger models.

Unlike loss, which scales smoothly (as seen in Chinchilla), specific tasks like multi-step math, common-sense reasoning, or code generation often show a “phase transition”—appearing suddenly once a model crosses a certain number of training FLOPs (typically around $10^{22}$ to $10^{24}$ FLOPs).

---

1. Scaling the Evaluation (Not the Model)

You don’t “implement” emergence in a layer; emergence is a property of the scale of the implementation. To study this in PyTorch, you must implement a Multi-Task Evaluation Suite that can track performance across models of different sizes (e.g., 125M, 1.3B, 13B, 175B).

The Benchmarking Logic

The paper focuses on “Few-Shot Prompting” (In-Context Learning). To replicate this, your code must handle the construction of “exemplars” before the query.

Python

import torch
import torch.nn.functional as F

class EmergenceEvaluator:
    def __init__(self, model, tokenizer):
        self.model = model
        self.tokenizer = tokenizer

    def evaluate_few_shot(self, task_description, examples, query):
        """
        Constructs a prompt: [Task] + [Examples] + [Query]
        Examples: List of (input, output) tuples
        """
        prompt = task_description + "\n\n"
        for i, o in examples:
            prompt += f"Input: {i}\nOutput: {o}\n\n"
        
        prompt += f"Input: {query}\nOutput:"
        
        inputs = self.tokenizer(prompt, return_tensors="pt").to(self.model.device)
        
        # We generate the completion and check for 'exact match' accuracy
        with torch.no_grad():
            output = self.model.generate(**inputs, max_new_tokens=20)
            
        decoded = self.tokenizer.decode(output[0], skip_special_tokens=True)
        return decoded.split("Output:")[-1].strip()

---

2. Identifying the “Elbow” in the Curve

The paper’s implementation core is the visualization of the Performance vs. Scale curve. Small models will have near-zero (random) accuracy, while large models will suddenly jump to high accuracy.

Python

import matplotlib.pyplot as plt

def plot_emergence(model_sizes_billions, accuracies, task_name):
    """
    Replicates the 'Phase Transition' plots from the paper.
    """
    plt.figure(figsize=(8, 5))
    plt.plot(model_sizes_billions, accuracies, marker='o', linestyle='-', linewidth=2)
    plt.xscale('log') # Scale laws and emergence are viewed on log scales
    plt.xlabel('Model Parameters (Log Scale)')
    plt.ylabel(f'{task_name} Accuracy')
    plt.title(f'Emergence of {task_name}')
    plt.grid(True, which="both", ls="-", alpha=0.5)
    plt.show()

# Example: 3-digit addition (Figure 2 in the paper)
sizes = [0.1, 1, 10, 100, 175] # Billions
accs  = [0.01, 0.02, 0.05, 0.45, 0.85] # Sudden jump after 10B
# plot_emergence(sizes, accs, "3-digit Addition")

---

3. Critical Emergent Techniques: Chain-of-Thought

One of the most famous emergent abilities discussed is Chain-of-Thought (CoT) reasoning. Models below a certain size (e.g., GPT-3 175B) do not benefit from CoT; in fact, it can degrade their performance. Only once they reach a specific scale does “thinking step-by-step” help them solve complex problems.

Implementation of CoT Logic

Python

def generate_cot_prompt(question):
    # The 'Emergent' trigger is the inclusion of 'Let's think step by step'
    return f"Q: {question}\nA: Let's think step by step."

# Small Model (<1B): Output is usually gibberish or repeats the prompt.
# Large Model (>100B): Output follows a logical sequence of operations.

---

4. Why Emergence is Controversial

Since the 2022 paper, researchers (e.g., Schaeffer et al., 2023) have argued that emergence might be a mirage caused by the metrics used.

• If you use Exact Match (0 or 1), it looks like a sudden jump.
• If you use Token Probability (Perplexity), the improvement is actually smooth and continuous, following the Scaling Laws we discussed previously.

Summary of Emergent Abilities

Ability	Description	Emergence Threshold
Arithmetic	Multi-digit addition/multiplication.	~10B – 100B Params
Big-Bench	Complex linguistic and logic tasks.	Variable by task complexity
CoT Reasoning	Solving word problems via intermediate steps.	~60B+ Params
Instruction Following	Obeying specific constraints in a prompt.	~10B+ Params

The 2022 paper “Training Compute-Optimal Large Language Models” (often called the Chinchilla paper) by Hoffmann et al. (DeepMind) fundamentally corrected the earlier scaling laws from OpenAI.

While OpenAI’s Kaplan et al. (2020) suggested that model size ($N$) should grow faster than data size ($D$), the Chinchilla researchers proved that for every doubling of model size, the number of training tokens should also double. This revealed that most famous models at the time (like GPT-3 and Gopher) were significantly undertrained.

---

1. The Chinchilla Loss Function

The paper models the final training loss $L$ as a function of parameters $N$ and tokens $D$ using a more refined power-law:

$$L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta}$$

Where:

• $E$ is the “irreducible loss” (the entropy of natural language).
• $A, B, \alpha, \beta$ are constants determined by empirical fitting.
• The Big Insight: To minimize loss for a fixed compute budget $C \approx 6ND$, the optimal ratio is roughly 20 tokens per parameter ($D/N \approx 20$).

---

2. PyTorch Implementation: Compute-Optimal Planner

In a real-world scenario, you don’t “implement” Chinchilla as a layer; you use it to plan your training run. Below is a tool to calculate the optimal $N$ and $D$ for any given budget of FLOPs ($C$).

Python

import torch
import numpy as np

class ChinchillaPlanner:
    def __init__(self):
        # Constants from the paper (Table 2, Approach 2)
        self.alpha = 0.34
        self.beta = 0.28
        self.A = 406.4
        self.B = 410.7

    def get_optimal_params_and_tokens(self, compute_budget_flops):
        """
        Calculates N and D that minimize loss for a given budget C.
        C = 6 * N * D
        """
        # Based on the paper's derivation: N_opt = G * (C/6)^a
        # where a = beta / (alpha + beta)
        a = self.beta / (self.alpha + self.beta)
        
        # G is a constant factor derived from A, B, alpha, beta
        G = (self.alpha * self.A / (self.beta * self.B)) ** (1 / (self.alpha + self.beta))
        
        n_opt = G * (compute_budget_flops / 6) ** a
        d_opt = compute_budget_flops / (6 * n_opt)
        
        return int(n_opt), int(d_opt)

# Example: If you have the budget of the original GPT-3 (approx 3.14e23 FLOPs)
planner = ChinchillaPlanner()
n, d = planner.get_optimal_params_and_tokens(3.14e23)

print(f"Optimal Model Size: {n/1e9:.1f}B parameters")
print(f"Optimal Data Size: {d/1e12:.2f}T tokens")
# Result: ~15B parameters and ~3.5T tokens (GPT-3 was 175B params / 0.3T tokens)

---

3. Implementation: LR Schedule for Long Training

A key technical finding in Chinchilla is that the cosine learning rate schedule should be decayed to a final value (usually 10% of the peak) exactly at the end of the token budget $D$.

If you train a model for more tokens than your schedule anticipated, performance degrades. This makes the max_steps parameter in your PyTorch scheduler critical.

Python

from torch.optim.lr_scheduler import CosineAnnealingLR

# If Chinchilla tells us the optimal D is 1.4 Trillion tokens:
total_tokens = 1.4e12
batch_size = 512
seq_len = 2048
total_steps = int(total_tokens / (batch_size * seq_len))

optimizer = torch.optim.AdamW(model.parameters(), lr=2e-4)

# Ensure the decay ends exactly when the optimal data budget is hit
scheduler = CosineAnnealingLR(optimizer, T_max=total_steps, eta_min=2e-5)

---

4. Why this changed the “LLM Arms Race”

Before Chinchilla (Kaplan Laws)	After Chinchilla
Focus on “Bigger is Better” (175B, 530B params).	Focus on “Better Trained” (70B, 8B params).
Models were trained on ~300B tokens.	Models (like Llama 3) are trained on 15T+ tokens.
High inference cost (massive models).	Low inference cost (smaller, smarter models).

The “Chinchilla Point”

The 70B Chinchilla model outperformed the 280B Gopher model despite using the exact same amount of training compute. This was because Chinchilla was trained on 1.4 Trillion tokens, whereas Gopher was trained on only 300 Billion.

The 2020 paper “Scaling Laws for Neural Language Models” by Kaplan et al. (OpenAI) is one of the most significant theoretical contributions to modern AI. It shifted the industry focus from “clever architectures” to “compute-optimal scaling.”

The paper found that language model performance (measured by cross-entropy loss) follows a power law relationship with three primary variables: the number of parameters ($N$), the size of the dataset ($D$), and the amount of compute ($C$).

---

1. The Core Empirical Findings

The researchers discovered that for a Transformer-based model, the test loss $L$ can be predicted using the following power-law formula:

$$L(N, D) \approx \left[ \left( \frac{N_c}{N} \right)^{\frac{\alpha_N}{\alpha_L}} + \left( \frac{D_c}{D} \right)^{\frac{\alpha_D}{\alpha_L}} \right]^{\alpha_L}$$

Where:

• $N$: Number of non-embedding parameters.
• $D$: Number of training tokens.
• Performance is bottlenecked by the smaller of $N$ or $D$. If you have a massive model but a tiny dataset, you will not see scaling benefits.

---

2. PyTorch Implementation: Parameter Counting

A critical part of obeying scaling laws is precisely calculating $N$ (non-embedding parameters). Kaplan et al. noted that embedding parameters do not contribute significantly to the “intelligence” (loss reduction) of the model as it scales.

Python

import torch
import torch.nn as nn

def count_non_embedding_params(model):
    """
    As per Kaplan et al. (2020), scaling laws are driven by 
    the model's internal parameters, excluding the embedding table.
    """
    total_params = sum(p.numel() for p in model.parameters() if p.requires_grad)
    
    # Identify and subtract embedding parameters
    emb_params = 0
    for module in model.modules():
        if isinstance(module, nn.Embedding):
            emb_params += sum(p.numel() for p in module.parameters())
            
    non_emb_params = total_params - emb_params
    return non_emb_params

---

3. Compute Estimation ($C$)

The paper provides a simple “rule of thumb” for the floating-point operations (FLOPs) required for a forward-backward pass of a Transformer. For a model with $N$ parameters and a dataset of $D$ tokens:

$$C \approx 6ND$$

Python

def estimate_training_compute(non_emb_params, tokens_trained):
    """
    Estimates total FLOPs for training.
    6 * N * D covers:
    - 2 ND for the forward pass
    - 4 ND for the backward pass
    """
    return 6 * non_emb_params * tokens_trained

# Example: A 1B parameter model trained on 20B tokens
params = 1e9
tokens = 20e9
total_flops = estimate_training_compute(params, tokens)
print(f"Total Compute: {total_flops:.2e} FLOPs")

---

4. Implementation: The Log-Linear Scheduler

The paper suggests that learning rate schedules and batch sizes should scale with the model size. Specifically, they found that the Critical Batch Size ($B^*$) increases as the loss decreases, following its own power law.

Python

class ScalingLawScheduler:
    """
    Simplified representation of scaling the learning rate 
    based on the model's 'd_model'.
    """
    def __init__(self, optimizer, d_model, warm_up_steps=3000):
        self.optimizer = optimizer
        self.d_model = d_model
        self.warm_up_steps = warm_up_steps
        self.current_step = 0

    def step(self):
        self.current_step += 1
        # Scaling law insight: Larger models require smaller learning rates
        # Base LR is often scaled by (d_model)^-0.5
        lr = (self.d_model ** -0.5) * min(self.current_step ** -0.5, 
                                          self.current_step * (self.warm_up_steps ** -1.5))
        
        for param_group in self.optimizer.param_groups:
            param_group['lr'] = lr

---

5. Implications for Model Design

Finding	Implementation Action
Architecture Irrelevance	Performance depends on total $N$, not the specific shape (width vs. depth).
Overfitting is Rare	On sufficiently large datasets, performance continues to improve; “more is better.”
The “Compute Optimal” Point	For every 10x increase in compute, increase model size by ~5.5x and dataset size by ~1.8x.

Why it Matters

This paper provided the “green light” for the creation of massive models like GPT-3 (175B). It proved that we hadn’t hit a ceiling; we just needed to keep scaling $N$, $D$, and $C$ in the correct proportions.

Note: These laws were later refined by the Chinchilla paper (Hoffmann et al., 2022), which argued that we were actually under-training our models and needed much more data ($D$) per parameter ($N$).

The 2019 paper “Language Models are Unsupervised Multitask Learners” (GPT-2) by Radford et al. (OpenAI) moved the field from “Pre-train then Fine-tune” to Zero-Shot Learning.

The researchers discovered that if you scale the model (up to 1.5B parameters) and the data (WebText), the model begins to perform tasks like translation, summarization, and question-answering without ever being specifically trained for them. Architecturally, it is an evolution of GPT-1 with several critical stability improvements.

1. Key Architectural Changes

GPT-2 introduced two major structural changes to the Transformer blocks to handle deeper stacks (up to 48 layers):

1. Pre-Layer Normalization: LayerNorm is moved to the input of each sub-block (rather than the output).
2. Final LayerNorm: An additional Layer Normalization is added after the final self-attention block.

2. PyTorch Implementation: The GPT-2 Block

This block implements the “Pre-LN” architecture which has since become the standard for almost all Large Language Models (LLMs).

Python

import torch
import torch.nn as nn
import torch.nn.functional as F

class GPT2Block(nn.Module):
    def __init__(self, d_model, n_heads, d_ff, max_len, dropout=0.1):
        super().__init__()
        # Pre-LayerNorm: Norm happens BEFORE the sub-layers
        self.ln_1 = nn.LayerNorm(d_model)
        self.attn = CausalSelfAttention(d_model, n_heads, max_len)
        
        self.ln_2 = nn.LayerNorm(d_model)
        self.mlp = nn.Sequential(
            nn.Linear(d_model, d_ff),
            nn.GELU(),
            nn.Linear(d_ff, d_model),
            nn.Dropout(dropout)
        )

    def forward(self, x):
        # Residual connection around the Norm + Attention
        x = x + self.attn(self.ln_1(x))
        # Residual connection around the Norm + MLP
        x = x + self.mlp(self.ln_2(x))
        return x

---

3. Weight Initialization Scaling

GPT-2 introduced a specific initialization trick: as the model gets deeper, the residual layers contribute more variance. To counter this, the paper scales the weights of residual layers at initialization by a factor of $1/\sqrt{N}$, where $N$ is the number of residual layers.

Python

def init_weights(module, n_layers):
    if isinstance(module, (nn.Linear, nn.Embedding)):
        module.weight.data.normal_(mean=0.0, std=0.02)
        if isinstance(module, nn.Linear) and module.bias is not None:
            module.bias.data.zero_()
    # Scale residual layer weights
    # This is often applied to the projection layers (c_proj) in implementation

---

4. The Full GPT-2 Model

The output of GPT-2 uses “Weight Tying,” where the Embedding matrix and the final Linear output layer share the same weights.

Python

class GPT2(nn.Module):
    def __init__(self, vocab_size, d_model, n_layers, n_heads, d_ff, max_len):
        super().__init__()
        self.wte = nn.Embedding(vocab_size, d_model) # Word Token Embedding
        self.wpe = nn.Embedding(max_len, d_model)   # Word Position Embedding
        self.drop = nn.Dropout(0.1)
        
        self.blocks = nn.ModuleList([
            GPT2Block(d_model, n_heads, d_ff, max_len) for _ in range(n_layers)
        ])
        
        self.ln_f = nn.LayerNorm(d_model)
        self.lm_head = nn.Linear(d_model, vocab_size, bias=False)
        
        # Weight Tying
        self.lm_head.weight = self.wte.weight

    def forward(self, x):
        device = x.device
        b, t = x.size()
        pos = torch.arange(0, t, dtype=torch.long, device=device).unsqueeze(0)
        
        # Combined Embeddings
        x = self.drop(self.wte(x) + self.wpe(pos))
        
        for block in self.blocks:
            x = block(x)
            
        x = self.ln_f(x)
        logits = self.lm_head(x)
        return logits

---

5. Byte-level BPE and “Zero-Shot”

GPT-2 used a 50,257 vocabulary based on Byte-level BPE. This ensured the model could represent any string of characters, which was vital for its ability to handle multiple languages and even code snippets without an “Unknown” token.

Model Variant	Parameters	Layers	dmodel​
GPT-2 Small	117M	12	768
GPT-2 Medium	345M	24	1024
GPT-2 Large	762M	36	1280
GPT-2 Extra Large	1.5B	48	1600

Why it was a breakthrough

GPT-2 proved that Multitask Learning is an emergent property of large-scale language modeling. By simply predicting the next word, the model learned how to translate (when it saw French followed by English in the data) or summarize (when it saw the word “TL;DR”). It moved the goalposts from “learning a task” to “learning a world model” through raw text.

The 2018 paper “Improving Language Understanding by Generative Pre-Training” (GPT-1) by Radford et al. (OpenAI) introduced the concept of Generative Pre-Training. While BERT (released around the same time) was a “bidirectional” encoder, GPT-1 was a unidirectional (left-to-right) decoder.

GPT-1 proved that a model pre-trained on a massive corpus of unlabeled text to predict the next word could be “fine-tuned” on specific tasks (like classification or entailment) with very little data.

1. The Causal Multi-Head Attention

The critical difference between GPT-1 and the standard Transformer encoder is the Causal Mask. To ensure the model only looks at the past when predicting the next word, we mask all “future” tokens by setting their attention scores to $-\infty$.

Python

import torch
import torch.nn as nn
import torch.nn.functional as F

class CausalSelfAttention(nn.Module):
    def __init__(self, d_model, n_heads, max_len):
        super().__init__()
        self.n_heads = n_heads
        self.d_k = d_model // n_heads
        
        self.w_qkv = nn.Linear(d_model, 3 * d_model)
        self.fc = nn.Linear(d_model, d_model)
        
        # Causal mask: lower triangular matrix of ones
        self.register_buffer("mask", torch.tril(torch.ones(max_len, max_len))
                                     .view(1, 1, max_len, max_len))

    def forward(self, x):
        batch_size, seq_len, d_model = x.size()
        
        # Generate Queries, Keys, and Values
        q, k, v = self.w_qkv(x).chunk(3, dim=-1)
        
        # Reshape for multi-head: [batch, heads, seq_len, d_k]
        q = q.view(batch_size, seq_len, self.n_heads, self.d_k).transpose(1, 2)
        k = k.view(batch_size, seq_len, self.n_heads, self.d_k).transpose(1, 2)
        v = v.view(batch_size, seq_len, self.n_heads, self.d_k).transpose(1, 2)
        
        # Scaled Dot-Product Attention
        scores = torch.matmul(q, k.transpose(-2, -1)) / (self.d_k ** 0.5)
        
        # Apply Causal Mask: prevent attending to future tokens
        scores = scores.masked_fill(self.mask[:, :, :seq_len, :seq_len] == 0, float('-inf'))
        
        attn = F.softmax(scores, dim=-1)
        out = torch.matmul(attn, v)
        
        # Re-assemble heads
        out = out.transpose(1, 2).contiguous().view(batch_size, seq_len, d_model)
        return self.fc(out)

---

2. The GPT-1 Decoder Block

GPT-1 uses the “Transformer Decoder” but removes the Encoder-Decoder attention layer (since there is no encoder). It consists only of Masked Self-Attention and a Feed-Forward Network.

Python

class GPTBlock(nn.Module):
    def __init__(self, d_model, n_heads, d_ff, max_len, dropout=0.1):
        super().__init__()
        self.ln_1 = nn.LayerNorm(d_model)
        self.attn = CausalSelfAttention(d_model, n_heads, max_len)
        self.ln_2 = nn.LayerNorm(d_model)
        self.mlp = nn.Sequential(
            nn.Linear(d_model, d_ff),
            nn.GELU(), # GPT-1 used GELU
            nn.Linear(d_ff, d_model),
            nn.Dropout(dropout)
        )

    def forward(self, x):
        # Pre-LayerNorm (standard in GPT models)
        x = x + self.attn(self.ln_1(x))
        x = x + self.mlp(self.ln_2(x))
        return x

---

3. Training Objective: Next Token Prediction

During pre-training, GPT-1 minimizes the negative log-likelihood of the next token given the previous ones:

$$L_1(\mathcal{U}) = \sum_i \log P(u_i | u_{i-k}, \dots, u_{i-1}; \Theta)$$

For fine-tuning, GPT-1 adds a task-specific linear layer but continues to use the language modeling objective as an auxiliary loss to improve generalization.

Python

class GPT1(nn.Module):
    def __init__(self, vocab_size, d_model, n_layers, n_heads, d_ff, max_len):
        super().__init__()
        self.token_emb = nn.Embedding(vocab_size, d_model)
        self.pos_emb = nn.Embedding(max_len, d_model)
        
        self.blocks = nn.ModuleList([
            GPTBlock(d_model, n_heads, d_ff, max_len) for _ in range(n_layers)
        ])
        
        self.ln_f = nn.LayerNorm(d_model)
        self.head = nn.Linear(d_model, vocab_size, bias=False)

    def forward(self, x):
        batch_size, seq_len = x.size()
        pos = torch.arange(seq_len).unsqueeze(0).to(x.device)
        
        x = self.token_emb(x) + self.pos_emb(pos)
        
        for block in self.blocks:
            x = block(x)
            
        x = self.ln_f(x)
        return self.head(x) # Logits for the entire vocab

---

Key Technical Details

• Byte-Pair Encoding (BPE): GPT-1 used a 40,000-merge BPE vocabulary.
• GELU Activation: Like BERT, GPT-1 used the Gaussian Error Linear Unit for smoother gradients.
• Task-Specific Input Transformations: To handle different tasks (Multiple Choice, Entailment, etc.) with the same decoder, GPT-1 introduced special delimiters like [Extract], [Start], and [Delim].

Why it was a breakthrough

Before GPT-1, most models were trained from scratch for each specific task. GPT-1 popularized the “Pre-train then Fine-tune” paradigm, showing that the knowledge gained by simply reading a large library (BooksCorpus) could be transferred to help a model understand logic, sentiment, and grammar.

The 2018 paper “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” by Devlin et al. (Google AI) introduced a shift from the standard “left-to-right” language modeling to a deeply bidirectional approach.

Unlike the original Transformer decoder (which masks future words), BERT uses a Masked Language Model (MLM) objective, allowing the representation of each word to fuse context from both its left and right simultaneously.

1. BERT Embeddings

BERT’s input is a summation of three embeddings: Token, Segment (to distinguish between two sentences), and Position (learned, unlike the sinusoidal ones in the original Transformer).

Python

import torch
import torch.nn as nn

class BERTEmbeddings(nn.Module):
    def __init__(self, vocab_size, hidden_size, max_len, n_segments):
        super().__init__()
        self.tok_embed = nn.Embedding(vocab_size, hidden_size)    # token embedding
        self.pos_embed = nn.Embedding(max_len, hidden_size)       # position embedding
        self.seg_embed = nn.Embedding(n_segments, hidden_size)    # segment(token_type) embedding
        self.norm = nn.LayerNorm(hidden_size)
        self.dropout = nn.Dropout(0.1)

    def forward(self, x, segment_label):
        seq_len = x.size(1)
        pos = torch.arange(seq_len, dtype=torch.long).unsqueeze(0).to(x.device)
        
        # BERT Input = Token + Position + Segment
        embedding = self.tok_embed(x) + self.pos_embed(pos) + self.seg_embed(segment_label)
        return self.dropout(self.norm(embedding))

---

2. The Multi-Head Attention (Bidirectional)

The core “Attention” is identical to the Transformer Encoder, but BERT explicitly removes the “causal mask” used in decoders, allowing every token to attend to every other token.

Python

class ScaledDotProductAttention(nn.Module):
    def forward(self, q, k, v, mask=None):
        d_k = q.size(-1)
        scores = torch.matmul(q, k.transpose(-2, -1)) / (d_k ** 0.5)
        
        if mask is not None:
            # Mask out padding tokens (0)
            scores = scores.masked_fill(mask == 0, -1e9)
            
        attn = torch.softmax(scores, dim=-1)
        return torch.matmul(attn, v)

---

3. The BERT Model & Pre-training Heads

BERT is pre-trained on two tasks:

1. Masked LM (MLM): Predict the [MASK] tokens.
2. Next Sentence Prediction (NSP): Predict if sentence B follows sentence A.

Python

class BERT(nn.Module):
    def __init__(self, vocab_size, hidden_size, n_layers, n_heads, d_ff):
        super().__init__()
        self.embedding = BERTEmbeddings(vocab_size, hidden_size, 512, 2)
        
        # Stack of Transformer Encoder Layers
        self.layers = nn.ModuleList([
            EncoderLayer(hidden_size, n_heads, d_ff, 0.1) for _ in range(n_layers)
        ])
        
        # NSP Head: Binary classification (isNext / NotNext)
        self.nsp_classifier = nn.Linear(hidden_size, 2)
        
        # MLM Head: Predict token from vocabulary
        self.mlm_classifier = nn.Linear(hidden_size, vocab_size)
        self.activation = nn.GELU() # BERT uses GELU instead of ReLU

    def forward(self, x, segment_label, mask):
        x = self.embedding(x, segment_label)
        
        for layer in self.layers:
            x = layer(x, mask)
            
        # Get the [CLS] token (first token) for NSP
        pooled_output = torch.tanh(nn.Linear(x.size(-1), x.size(-1))(x[:, 0]))
        nsp_logits = self.nsp_classifier(pooled_output)
        
        # Get all tokens for MLM
        mlm_logits = self.mlm_classifier(x)
        
        return mlm_logits, nsp_logits

---

Key Technical Details

• The [CLS] Token: BERT inserts a special [CLS] token at the start of every sequence. The final hidden state of this token is used as the aggregate sequence representation for classification tasks.
• GELU Activation: BERT popularized the Gaussian Error Linear Unit, which provides a smoother non-linearity than ReLU.
• Pre-training Strategy: 15% of tokens are chosen for masking. Of those, 80% are replaced with [MASK], 10% with a random token, and 10% are kept unchanged to ensure the model doesn’t only learn to focus on the mask.

Why it was a breakthrough

Before BERT, models like Word2Vec or GloVe gave a single static vector for a word (e.g., “bank” always had the same vector). BERT provides contextualized embeddings, meaning the vector for “bank” in “river bank” is completely different from “bank account” because the bidirectional attention sees the surrounding words before producing the vector.