The 2019 paper “Language Models are Unsupervised Multitask Learners” (GPT-2) by Radford et al. (OpenAI) moved the field from “Pre-train then Fine-tune” to Zero-Shot Learning.

The researchers discovered that if you scale the model (up to 1.5B parameters) and the data (WebText), the model begins to perform tasks like translation, summarization, and question-answering without ever being specifically trained for them. Architecturally, it is an evolution of GPT-1 with several critical stability improvements.

1. Key Architectural Changes

GPT-2 introduced two major structural changes to the Transformer blocks to handle deeper stacks (up to 48 layers):

1. Pre-Layer Normalization: LayerNorm is moved to the input of each sub-block (rather than the output).
2. Final LayerNorm: An additional Layer Normalization is added after the final self-attention block.

2. PyTorch Implementation: The GPT-2 Block

This block implements the “Pre-LN” architecture which has since become the standard for almost all Large Language Models (LLMs).

Python

import torch
import torch.nn as nn
import torch.nn.functional as F

class GPT2Block(nn.Module):
    def __init__(self, d_model, n_heads, d_ff, max_len, dropout=0.1):
        super().__init__()
        # Pre-LayerNorm: Norm happens BEFORE the sub-layers
        self.ln_1 = nn.LayerNorm(d_model)
        self.attn = CausalSelfAttention(d_model, n_heads, max_len)
        
        self.ln_2 = nn.LayerNorm(d_model)
        self.mlp = nn.Sequential(
            nn.Linear(d_model, d_ff),
            nn.GELU(),
            nn.Linear(d_ff, d_model),
            nn.Dropout(dropout)
        )

    def forward(self, x):
        # Residual connection around the Norm + Attention
        x = x + self.attn(self.ln_1(x))
        # Residual connection around the Norm + MLP
        x = x + self.mlp(self.ln_2(x))
        return x

3. Weight Initialization Scaling

GPT-2 introduced a specific initialization trick: as the model gets deeper, the residual layers contribute more variance. To counter this, the paper scales the weights of residual layers at initialization by a factor of $1/\sqrt{N}$, where $N$ is the number of residual layers.

Python

def init_weights(module, n_layers):
    if isinstance(module, (nn.Linear, nn.Embedding)):
        module.weight.data.normal_(mean=0.0, std=0.02)
        if isinstance(module, nn.Linear) and module.bias is not None:
            module.bias.data.zero_()
    # Scale residual layer weights
    # This is often applied to the projection layers (c_proj) in implementation

4. The Full GPT-2 Model

The output of GPT-2 uses “Weight Tying,” where the Embedding matrix and the final Linear output layer share the same weights.

Python

class GPT2(nn.Module):
    def __init__(self, vocab_size, d_model, n_layers, n_heads, d_ff, max_len):
        super().__init__()
        self.wte = nn.Embedding(vocab_size, d_model) # Word Token Embedding
        self.wpe = nn.Embedding(max_len, d_model)   # Word Position Embedding
        self.drop = nn.Dropout(0.1)
        
        self.blocks = nn.ModuleList([
            GPT2Block(d_model, n_heads, d_ff, max_len) for _ in range(n_layers)
        ])
        
        self.ln_f = nn.LayerNorm(d_model)
        self.lm_head = nn.Linear(d_model, vocab_size, bias=False)
        
        # Weight Tying
        self.lm_head.weight = self.wte.weight

    def forward(self, x):
        device = x.device
        b, t = x.size()
        pos = torch.arange(0, t, dtype=torch.long, device=device).unsqueeze(0)
        
        # Combined Embeddings
        x = self.drop(self.wte(x) + self.wpe(pos))
        
        for block in self.blocks:
            x = block(x)
            
        x = self.ln_f(x)
        logits = self.lm_head(x)
        return logits

5. Byte-level BPE and “Zero-Shot”

GPT-2 used a 50,257 vocabulary based on Byte-level BPE. This ensured the model could represent any string of characters, which was vital for its ability to handle multiple languages and even code snippets without an “Unknown” token.

Why it was a breakthrough

GPT-2 proved that Multitask Learning is an emergent property of large-scale language modeling. By simply predicting the next word, the model learned how to translate (when it saw French followed by English in the data) or summarize (when it saw the word “TL;DR”). It moved the goalposts from “learning a task” to “learning a world model” through raw text.

The 2018 paper “Improving Language Understanding by Generative Pre-Training” (GPT-1) by Radford et al. (OpenAI) introduced the concept of Generative Pre-Training. While BERT (released around the same time) was a “bidirectional” encoder, GPT-1 was a unidirectional (left-to-right) decoder.

GPT-1 proved that a model pre-trained on a massive corpus of unlabeled text to predict the next word could be “fine-tuned” on specific tasks (like classification or entailment) with very little data.

1. The Causal Multi-Head Attention

The critical difference between GPT-1 and the standard Transformer encoder is the Causal Mask. To ensure the model only looks at the past when predicting the next word, we mask all “future” tokens by setting their attention scores to $-\infty$.

Python

import torch
import torch.nn as nn
import torch.nn.functional as F

class CausalSelfAttention(nn.Module):
    def __init__(self, d_model, n_heads, max_len):
        super().__init__()
        self.n_heads = n_heads
        self.d_k = d_model // n_heads
        
        self.w_qkv = nn.Linear(d_model, 3 * d_model)
        self.fc = nn.Linear(d_model, d_model)
        
        # Causal mask: lower triangular matrix of ones
        self.register_buffer("mask", torch.tril(torch.ones(max_len, max_len))
                                     .view(1, 1, max_len, max_len))

    def forward(self, x):
        batch_size, seq_len, d_model = x.size()
        
        # Generate Queries, Keys, and Values
        q, k, v = self.w_qkv(x).chunk(3, dim=-1)
        
        # Reshape for multi-head: [batch, heads, seq_len, d_k]
        q = q.view(batch_size, seq_len, self.n_heads, self.d_k).transpose(1, 2)
        k = k.view(batch_size, seq_len, self.n_heads, self.d_k).transpose(1, 2)
        v = v.view(batch_size, seq_len, self.n_heads, self.d_k).transpose(1, 2)
        
        # Scaled Dot-Product Attention
        scores = torch.matmul(q, k.transpose(-2, -1)) / (self.d_k ** 0.5)
        
        # Apply Causal Mask: prevent attending to future tokens
        scores = scores.masked_fill(self.mask[:, :, :seq_len, :seq_len] == 0, float('-inf'))
        
        attn = F.softmax(scores, dim=-1)
        out = torch.matmul(attn, v)
        
        # Re-assemble heads
        out = out.transpose(1, 2).contiguous().view(batch_size, seq_len, d_model)
        return self.fc(out)

2. The GPT-1 Decoder Block

GPT-1 uses the “Transformer Decoder” but removes the Encoder-Decoder attention layer (since there is no encoder). It consists only of Masked Self-Attention and a Feed-Forward Network.

Python

class GPTBlock(nn.Module):
    def __init__(self, d_model, n_heads, d_ff, max_len, dropout=0.1):
        super().__init__()
        self.ln_1 = nn.LayerNorm(d_model)
        self.attn = CausalSelfAttention(d_model, n_heads, max_len)
        self.ln_2 = nn.LayerNorm(d_model)
        self.mlp = nn.Sequential(
            nn.Linear(d_model, d_ff),
            nn.GELU(), # GPT-1 used GELU
            nn.Linear(d_ff, d_model),
            nn.Dropout(dropout)
        )

    def forward(self, x):
        # Pre-LayerNorm (standard in GPT models)
        x = x + self.attn(self.ln_1(x))
        x = x + self.mlp(self.ln_2(x))
        return x

3. Training Objective: Next Token Prediction

During pre-training, GPT-1 minimizes the negative log-likelihood of the next token given the previous ones:

$$L_1(\mathcal{U}) = \sum_i \log P(u_i | u_{i-k}, \dots, u_{i-1}; \Theta)$$

For fine-tuning, GPT-1 adds a task-specific linear layer but continues to use the language modeling objective as an auxiliary loss to improve generalization.

Python

class GPT1(nn.Module):
    def __init__(self, vocab_size, d_model, n_layers, n_heads, d_ff, max_len):
        super().__init__()
        self.token_emb = nn.Embedding(vocab_size, d_model)
        self.pos_emb = nn.Embedding(max_len, d_model)
        
        self.blocks = nn.ModuleList([
            GPTBlock(d_model, n_heads, d_ff, max_len) for _ in range(n_layers)
        ])
        
        self.ln_f = nn.LayerNorm(d_model)
        self.head = nn.Linear(d_model, vocab_size, bias=False)

    def forward(self, x):
        batch_size, seq_len = x.size()
        pos = torch.arange(seq_len).unsqueeze(0).to(x.device)
        
        x = self.token_emb(x) + self.pos_emb(pos)
        
        for block in self.blocks:
            x = block(x)
            
        x = self.ln_f(x)
        return self.head(x) # Logits for the entire vocab

Key Technical Details

• Byte-Pair Encoding (BPE): GPT-1 used a 40,000-merge BPE vocabulary.
• GELU Activation: Like BERT, GPT-1 used the Gaussian Error Linear Unit for smoother gradients.
• Task-Specific Input Transformations: To handle different tasks (Multiple Choice, Entailment, etc.) with the same decoder, GPT-1 introduced special delimiters like [Extract], [Start], and [Delim].

Why it was a breakthrough

Before GPT-1, most models were trained from scratch for each specific task. GPT-1 popularized the “Pre-train then Fine-tune” paradigm, showing that the knowledge gained by simply reading a large library (BooksCorpus) could be transferred to help a model understand logic, sentiment, and grammar.

The 2018 paper “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” by Devlin et al. (Google AI) introduced a shift from the standard “left-to-right” language modeling to a deeply bidirectional approach.

Unlike the original Transformer decoder (which masks future words), BERT uses a Masked Language Model (MLM) objective, allowing the representation of each word to fuse context from both its left and right simultaneously.

1. BERT Embeddings

BERT’s input is a summation of three embeddings: Token, Segment (to distinguish between two sentences), and Position (learned, unlike the sinusoidal ones in the original Transformer).

Python

import torch
import torch.nn as nn

class BERTEmbeddings(nn.Module):
    def __init__(self, vocab_size, hidden_size, max_len, n_segments):
        super().__init__()
        self.tok_embed = nn.Embedding(vocab_size, hidden_size)    # token embedding
        self.pos_embed = nn.Embedding(max_len, hidden_size)       # position embedding
        self.seg_embed = nn.Embedding(n_segments, hidden_size)    # segment(token_type) embedding
        self.norm = nn.LayerNorm(hidden_size)
        self.dropout = nn.Dropout(0.1)

    def forward(self, x, segment_label):
        seq_len = x.size(1)
        pos = torch.arange(seq_len, dtype=torch.long).unsqueeze(0).to(x.device)
        
        # BERT Input = Token + Position + Segment
        embedding = self.tok_embed(x) + self.pos_embed(pos) + self.seg_embed(segment_label)
        return self.dropout(self.norm(embedding))

2. The Multi-Head Attention (Bidirectional)

The core “Attention” is identical to the Transformer Encoder, but BERT explicitly removes the “causal mask” used in decoders, allowing every token to attend to every other token.

Python

class ScaledDotProductAttention(nn.Module):
    def forward(self, q, k, v, mask=None):
        d_k = q.size(-1)
        scores = torch.matmul(q, k.transpose(-2, -1)) / (d_k ** 0.5)
        
        if mask is not None:
            # Mask out padding tokens (0)
            scores = scores.masked_fill(mask == 0, -1e9)
            
        attn = torch.softmax(scores, dim=-1)
        return torch.matmul(attn, v)

3. The BERT Model & Pre-training Heads

BERT is pre-trained on two tasks:

1. Masked LM (MLM): Predict the [MASK] tokens.
2. Next Sentence Prediction (NSP): Predict if sentence B follows sentence A.

Python

class BERT(nn.Module):
    def __init__(self, vocab_size, hidden_size, n_layers, n_heads, d_ff):
        super().__init__()
        self.embedding = BERTEmbeddings(vocab_size, hidden_size, 512, 2)
        
        # Stack of Transformer Encoder Layers
        self.layers = nn.ModuleList([
            EncoderLayer(hidden_size, n_heads, d_ff, 0.1) for _ in range(n_layers)
        ])
        
        # NSP Head: Binary classification (isNext / NotNext)
        self.nsp_classifier = nn.Linear(hidden_size, 2)
        
        # MLM Head: Predict token from vocabulary
        self.mlm_classifier = nn.Linear(hidden_size, vocab_size)
        self.activation = nn.GELU() # BERT uses GELU instead of ReLU

    def forward(self, x, segment_label, mask):
        x = self.embedding(x, segment_label)
        
        for layer in self.layers:
            x = layer(x, mask)
            
        # Get the [CLS] token (first token) for NSP
        pooled_output = torch.tanh(nn.Linear(x.size(-1), x.size(-1))(x[:, 0]))
        nsp_logits = self.nsp_classifier(pooled_output)
        
        # Get all tokens for MLM
        mlm_logits = self.mlm_classifier(x)
        
        return mlm_logits, nsp_logits

Key Technical Details

• The [CLS] Token: BERT inserts a special [CLS] token at the start of every sequence. The final hidden state of this token is used as the aggregate sequence representation for classification tasks.
• GELU Activation: BERT popularized the Gaussian Error Linear Unit, which provides a smoother non-linearity than ReLU.
• Pre-training Strategy: 15% of tokens are chosen for masking. Of those, 80% are replaced with [MASK], 10% with a random token, and 10% are kept unchanged to ensure the model doesn’t only learn to focus on the mask.

Why it was a breakthrough

Before BERT, models like Word2Vec or GloVe gave a single static vector for a word (e.g., “bank” always had the same vector). BERT provides contextualized embeddings, meaning the vector for “bank” in “river bank” is completely different from “bank account” because the bidirectional attention sees the surrounding words before producing the vector.

The modern LLM landscape (2023–2026) is no longer just about making models bigger. The focus has shifted to efficiency, long-context recall, native multimodality, and test-time reasoning.

Here are the foundational papers and technical reports that define this “Post-Scaling” era.

1. The Return of Sparsity: Mixture of Experts (MoE)

As models hit the trillion-parameter mark, “Dense” models became too expensive. Sparse architectures allow for massive capacity with lower compute costs.

• Paper/Technical Report: Mixtral of Experts (2024)
• Authors: Albert Q. Jiang, Alexandre Sablayrolles, et al. (Mistral AI)
• Contribution: Proved that a Sparse MoE model (specifically 8x7B) could outperform much larger dense models like Llama 2 70B while being faster at inference.
• DeepSeek-V3/V4 (2025-2026): Introduced Multi-head Latent Attention (MLA) and advanced load balancing, further optimizing how “experts” are routed.

2. Efficiency & Serving Breakthroughs

Training a model is only half the battle; running it for millions of users requires mathematical innovations in memory management.

• Paper: Efficient Memory Management for Large Language Model Serving with PagedAttention (2023)
• Authors: Woosuk Kwon, Zhuohan Li, et al.
• Contribution: Introduced vLLM and PagedAttention. By treating KV cache memory like virtual memory in an OS, they reduced memory waste by 96%, enabling the high-speed APIs we use today.

3. The “Inference Scaling” Revolution (System 2)

This represents the shift from “predicting the next word” to “searching for the right answer.”

• Models: OpenAI o1/o3 and Gemini 2.0/3.0 Thinking (2024–2026)
• Key Concept: These models use Chain-of-Thought (CoT) Reinforcement Learning and Search (like Monte Carlo Tree Search).
• The Insight: Scaling compute during inference (letting the model think longer) provides a performance boost that parameters alone cannot achieve.

4. Native Multimodality (Omni-Models)

Earlier multimodal models were “Frankenstein” systems (a CLIP image encoder stitched to a GPT text decoder). Modern models are natively multimodal.

• Paper/Release: Gemini: A Family of Highly Capable Multimodal Models (2023) / GPT-4o (2024)
• Contribution: These models were trained on text, images, audio, and video simultaneously using a single end-to-end transformer. This allows for near-zero latency in voice conversations and “vision” that understands nuances like human emotion.

5. Long Context & The Needle in a Haystack

The “context window” expanded from 4k tokens to millions, changing how we use LLMs as databases.

• Paper: Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context (2024)
• Contribution: Introduced the “Needle In A Haystack” test as the gold standard for long-context retrieval. It proved that models could maintain 99%+ recall across 1M+ tokens, effectively replacing some RAG (Retrieval-Augmented Generation) use cases.

Summary of the Current Landscape

This DeepSeek-V3 technical walkthrough provides a visual tour of modern architectures, explaining how features like Sparse Attention and Multi-head Latent Attention (MLA) differ from the original 2017 Transformer.

The era between 2020 and 2022 was when LLM development shifted from “art” to “predictive science.” Researchers discovered that model performance isn’t random—it follows strict mathematical laws—but that these smooth laws occasionally produce “magic” new abilities.

Here are the foundational papers that defined this period.

1. The Discovery of Scaling Laws (2020)

Before this paper, training a giant model was a massive gamble. OpenAI turned it into a math problem.

• Paper: Scaling Laws for Neural Language Models
• Authors: Jared Kaplan, Sam McCandlish, Tom Henighan, et al. (OpenAI)
• The Breakthrough: They discovered that Test Loss (error) follows a Power Law. If you increase your Compute ($C$), Dataset ($D$), or Parameters ($N$), the model’s performance improves in a perfectly straight line on a log-log plot.
• The “Kaplan Rule”: They initially argued that if you have more compute, you should put most of it into making the model bigger ($N$) rather than giving it more data ($D$).

2. The “Chinchilla” Correction (2022)

DeepMind realized that the industry (including GPT-3) was building models that were “too big” and “under-trained.”

• Paper: Training Compute-Optimal Large Language Models
• Authors: Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, et al. (DeepMind)
• The Breakthrough: They proved that for a model to be “compute-optimal,” you must scale parameters and data equally.
• The “Chinchilla Rule”: For every doubling of model size, you should also double the amount of training data. They suggested a ratio of roughly 20 tokens per 1 parameter.
  • Impact: This led to the creation of smaller, “denser” models like Llama, which are much cheaper to run but just as smart as giants.

3. The Mystery of Emergence (2022)

While scaling laws predicted that models would get “better” at predicting words, they didn’t explain why models suddenly learned to do math or code.

• Paper: Emergent Abilities of Large Language Models
• Authors: Jason Wei, Yi Tay, Rishi Bommasani, et al. (Google/Stanford)
• The Breakthrough: Defined “Emergence” as an ability that is not present in small models but appears suddenly in large ones.
• The Finding: Tasks like multi-step reasoning, translation, and common-sense logic don’t improve slowly; they stay at 0% accuracy until the model hits a certain “critical mass” (often around 10–100 billion parameters), at which point accuracy spikes to 50%+.

4. The Benchmarking Giant (2022)

To prove these laws and abilities, researchers needed a massive, difficult test that a model couldn’t just “memorize.”

• Paper: Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models (known as BIG-bench)
• Authors: Aarohi Srivastava, Guy Gur-Ari, et al. (Google & 400+ contributors)
• The Breakthrough: A collaborative suite of 200+ tasks (from emoji-understanding to chess) designed to find the limits of LLMs. It provided the data that proved “Emergent Abilities” were a real phenomenon across many different model architectures.

Summary of the Era

The 2024–2026 Update: The “Mirage” Debate

It’s worth noting that a newer foundational paper (“Are Emergent Abilities of Large Language Models a Mirage?”) argued that emergence might just be a trick of how we measure things. For example, if you measure “Correct vs. Incorrect,” a model looks like it suddenly “gets it.” But if you measure “Token Probability,” the model was actually improving smoothly the whole time.

The “Great Divergence” occurred shortly after the Transformer was born, as the AI world split into two camps: those who wanted models to understand (Google’s BERT) and those who wanted models to generate (OpenAI’s GPT).

These are the foundational papers that defined that era and the “Scale War” that followed.

1. The “Understanding” Path (Encoder-Only)

Google focused on using the Encoder half of the Transformer to create a “Swiss Army Knife” for language understanding.

• BERT (2018):“BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”
  • Authors: Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova.
  • The Breakthrough: Introduced Masked Language Modeling (MLM). By hiding 15% of words and making the model guess them using both left and right context, BERT became the best “reader” in history.
• RoBERTa (2019):“RoBERTa: A Robustly Optimized BERT Pretraining Approach”
  • Authors: Yinhan Liu, Myle Ott, et al. (Facebook AI).
  • The Breakthrough: Proved that BERT was “significantly undertrained.” By training longer, on more data, and removing the “Next Sentence Prediction” task, they reached much higher performance, showing that Scale > Architectural tweaks.

2. The “Generative” Path (Decoder-Only)

OpenAI bet everything on the Decoder half, believing that if a model can perfectly predict the next word, it will eventually understand everything else.

• GPT-1 (2018):“Improving Language Understanding by Generative Pre-Training”
  • Authors: Alec Radford, Karthik Narasimhan, et al.
  • The Breakthrough: The first to show that a generative model, pre-trained on a diverse corpus (BookCorpus), could be fine-tuned for many different tasks.
• GPT-2 (2019):“Language Models are Unsupervised Multitask Learners”
  • Authors: Alec Radford, Jeffrey Wu, et al.
  • The Breakthrough: This started the Scale War. It jumped to 1.5 billion parameters and proved “Zero-Shot” learning: the model could do tasks (like translation) it was never specifically trained for, simply because it had “read the whole internet.”

3. The Bridge: The Unified Path

While GPT and BERT were fighting, Google released a “best of both worlds” model that remains a staple in research.

• T5 (2019):“Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer”
  • Authors: Colin Raffel, Noam Shazeer, et al.
  • The Breakthrough: Proposed that every NLP task (classification, translation, etc.) should be treated as a “text-to-text” problem. It used the full Encoder-Decoder stack and introduced the C4 dataset, which became the standard for training massive models.

Summary: The Divergence in a Nutshell

The “End” of the Divergence

By 2021, the GPT Philosophy essentially “won” the Scale War for general-purpose AI. While BERT is still used for specific business tasks (like categorizing emails), the world moved toward Generative models because they demonstrated “Emergent Intelligence” at massive scales that encoders simply couldn’t match.

While the 2017 “Attention Is All You Need” paper is the undisputed crown jewel of the Transformer era, it didn’t emerge from a vacuum. It was the culmination of a three-year “arms race” to fix the bottleneck of LSTMs.

Here are the foundational papers that built the bridge from LSTMs to the Transformer.

1. The Birth of Modern Attention (2014)

Before this paper, “Attention” wasn’t really a thing in NLP. We just tried to squash entire sentences into a single fixed-length vector.

• Paper: Neural Machine Translation by Jointly Learning to Align and Translate
• Authors: Dzmitry Bahdanau, Kyunghyun Cho, & Yoshua Bengio
• Contribution: Introduced “Bahdanau Attention” (or Additive Attention). It allowed the model to search for relevant parts of a source sentence for each word it generated. It proved that “aligning” and “translating” could happen simultaneously.

2. Refining the Math: Dot-Product Attention (2015)

This paper simplified the math of attention, moving toward the speed and efficiency we see in modern LLMs.

• Paper: Effective Approaches to Attention-based Neural Machine Translation
• Authors: Minh-Thang Luong, Hieu Pham, & Christopher D. Manning
• Contribution: Introduced “Global” vs. “Local” attention and, crucially, “Dot-Product Attention.” This mathematical simplification was much faster to compute than Bahdanau’s version and is a direct ancestor of the “Scaled Dot-Product Attention” used in GPT and Gemini today.

3. The Quest for Parallelism (2017)

Just months before the Transformer paper, Facebook (now Meta) tried to solve the LSTM speed problem using Convolutions (CNNs) instead of Attention.

• Paper: Convolutional Sequence to Sequence Learning
• Authors: Jonas Gehring, Michael Auli, David Grangier, et al.
• Contribution: They showed that you could process sequences in parallel using CNNs, outperforming LSTMs. This put immense pressure on the research community to find a “fully parallel” architecture.

4. The Turning Point: “Attention Is All You Need” (2017)

The Google Brain team took the “Attention” idea from Bahdanau and the “Parallelism” goal from Gehring and realized: You don’t need LSTMs or CNNs at all.

• Paper: Attention Is All You Need
• Authors: Ashish Vaswani, Noam Shazeer, Niki Parmar, et al.
• Core Innovations:
  • Self-Attention: Instead of the decoder looking at the encoder, every word in a sentence looks at every other word in that same sentence.
  • Multi-Head Attention: The model can focus on different things at once (e.g., one “head” focuses on grammar, another on the subject of the sentence).
  • Positional Encoding: Since there is no “order” in a parallel system, they used sine/cosine waves to tell the model where each word sits in a sentence.

Comparison: How Attention Changed Everything

Fun Fact: The authors of “Attention Is All You Need” originally used the name “Transformer” because the model transforms one sequence into another, but also as a nod to the fact that it was “transforming” the field of AI.