Pytorch implementation of paper: GPT-2 (2019):”Language Models are Unsupervised Multitask Learners”

The 2019 paper “Language Models are Unsupervised Multitask Learners” (GPT-2) by Radford et al. (OpenAI) moved the field from “Pre-train then Fine-tune” to Zero-Shot Learning.

The researchers discovered that if you scale the model (up to 1.5B parameters) and the data (WebText), the model begins to perform tasks like translation, summarization, and question-answering without ever being specifically trained for them. Architecturally, it is an evolution of GPT-1 with several critical stability improvements.

1. Key Architectural Changes

GPT-2 introduced two major structural changes to the Transformer blocks to handle deeper stacks (up to 48 layers):

1. Pre-Layer Normalization: LayerNorm is moved to the input of each sub-block (rather than the output).
2. Final LayerNorm: An additional Layer Normalization is added after the final self-attention block.

2. PyTorch Implementation: The GPT-2 Block

This block implements the “Pre-LN” architecture which has since become the standard for almost all Large Language Models (LLMs).

Python

import torch
import torch.nn as nn
import torch.nn.functional as F

class GPT2Block(nn.Module):
    def __init__(self, d_model, n_heads, d_ff, max_len, dropout=0.1):
        super().__init__()
        # Pre-LayerNorm: Norm happens BEFORE the sub-layers
        self.ln_1 = nn.LayerNorm(d_model)
        self.attn = CausalSelfAttention(d_model, n_heads, max_len)
        
        self.ln_2 = nn.LayerNorm(d_model)
        self.mlp = nn.Sequential(
            nn.Linear(d_model, d_ff),
            nn.GELU(),
            nn.Linear(d_ff, d_model),
            nn.Dropout(dropout)
        )

    def forward(self, x):
        # Residual connection around the Norm + Attention
        x = x + self.attn(self.ln_1(x))
        # Residual connection around the Norm + MLP
        x = x + self.mlp(self.ln_2(x))
        return x

---

3. Weight Initialization Scaling

GPT-2 introduced a specific initialization trick: as the model gets deeper, the residual layers contribute more variance. To counter this, the paper scales the weights of residual layers at initialization by a factor of $1/\sqrt{N}$, where $N$ is the number of residual layers.

Python

def init_weights(module, n_layers):
    if isinstance(module, (nn.Linear, nn.Embedding)):
        module.weight.data.normal_(mean=0.0, std=0.02)
        if isinstance(module, nn.Linear) and module.bias is not None:
            module.bias.data.zero_()
    # Scale residual layer weights
    # This is often applied to the projection layers (c_proj) in implementation

---

4. The Full GPT-2 Model

The output of GPT-2 uses “Weight Tying,” where the Embedding matrix and the final Linear output layer share the same weights.

Python

class GPT2(nn.Module):
    def __init__(self, vocab_size, d_model, n_layers, n_heads, d_ff, max_len):
        super().__init__()
        self.wte = nn.Embedding(vocab_size, d_model) # Word Token Embedding
        self.wpe = nn.Embedding(max_len, d_model)   # Word Position Embedding
        self.drop = nn.Dropout(0.1)
        
        self.blocks = nn.ModuleList([
            GPT2Block(d_model, n_heads, d_ff, max_len) for _ in range(n_layers)
        ])
        
        self.ln_f = nn.LayerNorm(d_model)
        self.lm_head = nn.Linear(d_model, vocab_size, bias=False)
        
        # Weight Tying
        self.lm_head.weight = self.wte.weight

    def forward(self, x):
        device = x.device
        b, t = x.size()
        pos = torch.arange(0, t, dtype=torch.long, device=device).unsqueeze(0)
        
        # Combined Embeddings
        x = self.drop(self.wte(x) + self.wpe(pos))
        
        for block in self.blocks:
            x = block(x)
            
        x = self.ln_f(x)
        logits = self.lm_head(x)
        return logits

---

5. Byte-level BPE and “Zero-Shot”

GPT-2 used a 50,257 vocabulary based on Byte-level BPE. This ensured the model could represent any string of characters, which was vital for its ability to handle multiple languages and even code snippets without an “Unknown” token.

Model Variant	Parameters	Layers	dmodel​
GPT-2 Small	117M	12	768
GPT-2 Medium	345M	24	1024
GPT-2 Large	762M	36	1280
GPT-2 Extra Large	1.5B	48	1600

Why it was a breakthrough

GPT-2 proved that Multitask Learning is an emergent property of large-scale language modeling. By simply predicting the next word, the model learned how to translate (when it saw French followed by English in the data) or summarize (when it saw the word “TL;DR”). It moved the goalposts from “learning a task” to “learning a world model” through raw text.