While the 2017 “Attention Is All You Need” paper is the undisputed crown jewel of the Transformer era, it didn’t emerge from a vacuum. It was the culmination of a three-year “arms race” to fix the bottleneck of LSTMs.
Here are the foundational papers that built the bridge from LSTMs to the Transformer.
1. The Birth of Modern Attention (2014)
Before this paper, “Attention” wasn’t really a thing in NLP. We just tried to squash entire sentences into a single fixed-length vector.
- Paper: Neural Machine Translation by Jointly Learning to Align and Translate
- Authors: Dzmitry Bahdanau, Kyunghyun Cho, & Yoshua Bengio
- Contribution: Introduced “Bahdanau Attention” (or Additive Attention). It allowed the model to search for relevant parts of a source sentence for each word it generated. It proved that “aligning” and “translating” could happen simultaneously.
2. Refining the Math: Dot-Product Attention (2015)
This paper simplified the math of attention, moving toward the speed and efficiency we see in modern LLMs.
- Paper: Effective Approaches to Attention-based Neural Machine Translation
- Authors: Minh-Thang Luong, Hieu Pham, & Christopher D. Manning
- Contribution: Introduced “Global” vs. “Local” attention and, crucially, “Dot-Product Attention.” This mathematical simplification was much faster to compute than Bahdanau’s version and is a direct ancestor of the “Scaled Dot-Product Attention” used in GPT and Gemini today.
3. The Quest for Parallelism (2017)
Just months before the Transformer paper, Facebook (now Meta) tried to solve the LSTM speed problem using Convolutions (CNNs) instead of Attention.
- Paper: Convolutional Sequence to Sequence Learning
- Authors: Jonas Gehring, Michael Auli, David Grangier, et al.
- Contribution: They showed that you could process sequences in parallel using CNNs, outperforming LSTMs. This put immense pressure on the research community to find a “fully parallel” architecture.
4. The Turning Point: “Attention Is All You Need” (2017)
The Google Brain team took the “Attention” idea from Bahdanau and the “Parallelism” goal from Gehring and realized: You don’t need LSTMs or CNNs at all.
- Paper: Attention Is All You Need
- Authors: Ashish Vaswani, Noam Shazeer, Niki Parmar, et al.
- Core Innovations:
- Self-Attention: Instead of the decoder looking at the encoder, every word in a sentence looks at every other word in that same sentence.
- Multi-Head Attention: The model can focus on different things at once (e.g., one “head” focuses on grammar, another on the subject of the sentence).
- Positional Encoding: Since there is no “order” in a parallel system, they used sine/cosine waves to tell the model where each word sits in a sentence.
Comparison: How Attention Changed Everything
| Feature | LSTM Era (Pre-2017) | Transformer Era (Post-2017) |
| Processing | Sequential (Word 1 → Word 2) | Parallel (All words at once) |
| Relationship | Proximity-based (Memory fades) | Global (Every word “sees” every word) |
| Speed | Slow (Hard to use multiple GPUs) | Fast (Scales across thousands of GPUs) |
Fun Fact: The authors of “Attention Is All You Need” originally used the name “Transformer” because the model transforms one sequence into another, but also as a nod to the fact that it was “transforming” the field of AI.
Leave a Reply