The “Great Divergence” occurred shortly after the Transformer was born, as the AI world split into two camps: those who wanted models to understand (Google’s BERT) and those who wanted models to generate (OpenAI’s GPT).
These are the foundational papers that defined that era and the “Scale War” that followed.
1. The “Understanding” Path (Encoder-Only)
Google focused on using the Encoder half of the Transformer to create a “Swiss Army Knife” for language understanding.
- BERT (2018):“BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”
- Authors: Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova.
- The Breakthrough: Introduced Masked Language Modeling (MLM). By hiding 15% of words and making the model guess them using both left and right context, BERT became the best “reader” in history.
- RoBERTa (2019):“RoBERTa: A Robustly Optimized BERT Pretraining Approach”
- Authors: Yinhan Liu, Myle Ott, et al. (Facebook AI).
- The Breakthrough: Proved that BERT was “significantly undertrained.” By training longer, on more data, and removing the “Next Sentence Prediction” task, they reached much higher performance, showing that Scale > Architectural tweaks.
2. The “Generative” Path (Decoder-Only)
OpenAI bet everything on the Decoder half, believing that if a model can perfectly predict the next word, it will eventually understand everything else.
- GPT-1 (2018):“Improving Language Understanding by Generative Pre-Training”
- Authors: Alec Radford, Karthik Narasimhan, et al.
- The Breakthrough: The first to show that a generative model, pre-trained on a diverse corpus (BookCorpus), could be fine-tuned for many different tasks.
- GPT-2 (2019):“Language Models are Unsupervised Multitask Learners”
- Authors: Alec Radford, Jeffrey Wu, et al.
- The Breakthrough: This started the Scale War. It jumped to 1.5 billion parameters and proved “Zero-Shot” learning: the model could do tasks (like translation) it was never specifically trained for, simply because it had “read the whole internet.”
3. The Bridge: The Unified Path
While GPT and BERT were fighting, Google released a “best of both worlds” model that remains a staple in research.
- T5 (2019):“Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer”
- Authors: Colin Raffel, Noam Shazeer, et al.
- The Breakthrough: Proposed that every NLP task (classification, translation, etc.) should be treated as a “text-to-text” problem. It used the full Encoder-Decoder stack and introduced the C4 dataset, which became the standard for training massive models.
Summary: The Divergence in a Nutshell
| Feature | BERT Philosophy | GPT Philosophy |
| Primary Goal | Feature Extraction (Understanding) | Next-Token Prediction (Generation) |
| Context | Bidirectional (Looks left & right) | Unidirectional (Looks only at the past) |
| Ideal Task | Search, Sentiment, NER | Writing, Coding, Chat |
| Outcome | Revolutionized Google Search | Led to ChatGPT & the LLM era |
The “End” of the Divergence
By 2021, the GPT Philosophy essentially “won” the Scale War for general-purpose AI. While BERT is still used for specific business tasks (like categorizing emails), the world moved toward Generative models because they demonstrated “Emergent Intelligence” at massive scales that encoders simply couldn’t match.
Leave a Reply