The paper “Convolutional Sequence to Sequence Learning” (Gehring et al., 2017) by Facebook AI Research (FAIR) was a landmark shift in NMT. Unlike LSTMs or GRUs, which are inherently sequential, this model uses Convolutional Neural Networks (CNNs).
Because CNNs can be computed in parallel, this architecture is significantly faster to train than RNNs. It also uses Gated Linear Units (GLU) and a sophisticated Position Embedding system to maintain word order.
1. Position Embeddings
Since CNNs have no natural concept of “time” or “order,” we must explicitly add positional information to the word embeddings.
Python
import torch
import torch.nn as nn
import torch.nn.functional as F
class Encoder(nn.Module):
def __init__(self, input_dim, emb_dim, hid_dim, n_layers, kernel_size, dropout, device, max_length=100):
super().__init__()
self.device = device
self.scale = torch.sqrt(torch.FloatTensor([0.5])).to(device)
self.tok_embedding = nn.Embedding(input_dim, emb_dim)
self.pos_embedding = nn.Embedding(max_length, emb_dim)
# Linear layer to map embedding dim to hidden dim
self.emb2hid = nn.Linear(emb_dim, hid_dim)
self.hid2emb = nn.Linear(hid_dim, emb_dim)
# Convolutional blocks
self.convs = nn.ModuleList([nn.Conv1d(in_channels=hid_dim,
out_channels=2 * hid_dim,
kernel_size=kernel_size,
padding=(kernel_size - 1) // 2)
for _ in range(n_layers)])
self.dropout = nn.Dropout(dropout)
def forward(self, src):
# src: [batch_size, src_len]
batch_size = src.shape[0]
src_len = src.shape[1]
# Create position tensor
pos = torch.arange(0, src_len).unsqueeze(0).repeat(batch_size, 1).to(self.device)
# Combined embedding
e = self.dropout(self.tok_embedding(src) + self.pos_embedding(pos))
# Element-wise scaling as per paper
conv_input = self.emb2hid(e).permute(0, 2, 1)
for conv in self.convs:
# Residual connection
convolved = conv(self.dropout(conv_input))
# Gated Linear Unit (GLU) activation
# Splits the 2*hid_dim channels into two, applies sigmoid to one
convolved = F.glu(convolved, dim=1)
# Add residual and scale
conv_input = (conv_input + convolved) * self.scale
# Extract combined and element-wise representations
combined = self.hid2emb(conv_input.permute(0, 2, 1))
return conv_input, combined
2. The Gated Linear Unit (GLU)
The paper uses GLUs instead of ReLU or Tanh. A GLU takes a vector $A$ and $B$ (split from the convolution output) and computes $A \otimes \sigma(B)$. This allows the network to control which information flows through the hierarchy.
3. The Decoder with Multi-Step Attention
The decoder uses a similar CNN structure but includes an attention mechanism in every layer. It looks at the encoder’s state to decide which source words are relevant.
Python
class Decoder(nn.Module):
def __init__(self, output_dim, emb_dim, hid_dim, n_layers, kernel_size, dropout, trg_pad_idx, device, max_length=100):
super().__init__()
self.kernel_size = kernel_size
self.trg_pad_idx = trg_pad_idx
self.device = device
self.scale = torch.sqrt(torch.FloatTensor([0.5])).to(device)
self.tok_embedding = nn.Embedding(output_dim, emb_dim)
self.pos_embedding = nn.Embedding(max_length, emb_dim)
self.emb2hid = nn.Linear(emb_dim, hid_dim)
self.hid2emb = nn.Linear(hid_dim, emb_dim)
self.attn_hid2emb = nn.Linear(hid_dim, emb_dim)
self.fc_out = nn.Linear(emb_dim, output_dim)
self.convs = nn.ModuleList([nn.Conv1d(hid_dim, 2 * hid_dim, kernel_size, padding=kernel_size - 1)
for _ in range(n_layers)])
self.dropout = nn.Dropout(dropout)
def calculate_attention(self, embedded, convolved, encoder_convolved, encoder_combined):
# convolved: [batch_size, hid_dim, trg_len]
# encoder_convolved: [batch_size, hid_dim, src_len]
# 1. Map convolved back to embedding space
convolved_emb = self.attn_hid2emb(convolved.permute(0, 2, 1))
# 2. Add current target embedding (Residual)
query = (convolved_emb + embedded) * self.scale
# 3. Dot product attention with encoder_convolved
energy = torch.matmul(query, encoder_convolved.permute(0, 2, 1))
attention = F.softmax(energy, dim=2)
# 4. Weighted sum of encoder_combined
weighted = torch.matmul(attention, encoder_combined)
# 5. Map back to hid_dim
return self.emb2hid(weighted).permute(0, 2, 1), attention
def forward(self, trg, encoder_convolved, encoder_combined):
# trg: [batch_size, trg_len]
batch_size = trg.shape[0]
trg_len = trg.shape[1]
pos = torch.arange(0, trg_len).unsqueeze(0).repeat(batch_size, 1).to(self.device)
embedded = self.dropout(self.tok_embedding(trg) + self.pos_embedding(pos))
conv_input = self.emb2hid(embedded).permute(0, 2, 1)
for conv in self.convs:
# Casual Padding: ensure the conv doesn't "see" the future
padding = torch.zeros(batch_size, conv_input.shape[1], self.kernel_size - 1).to(self.device)
padded_input = torch.cat((padding, conv_input), dim=2)
convolved = conv(self.dropout(padded_input))
convolved = F.glu(convolved[:, :, :conv_input.shape[2] + self.kernel_size - 1], dim=1)
# Multi-step Attention
attn_weighted, _ = self.calculate_attention(embedded, convolved, encoder_convolved, encoder_combined)
# Residual + Attention + Scale
convolved = (convolved + attn_weighted) * self.scale
conv_input = (conv_input + convolved) * self.scale
output = self.fc_out(self.hid2emb(conv_input.permute(0, 2, 1)))
return output
Key Takeaways from the Paper
- Parallelism: Unlike LSTMs, the entire source sentence is processed at once through the convolutional layers.
- Causal Convolutions: In the decoder, we use specific padding to ensure that the prediction for word $t$ only depends on words $1 \dots t-1$.
- Multi-Step Attention: Applying attention at every layer allows the model to refine its “focus” as it goes deeper into the architecture.
- Deep Residuals: Because the network is deep, residual connections and the $\sqrt{0.5}$ scaling factor are critical for stable training.
This architecture served as the bridge between RNN-based translation and the Transformer (Attention is All You Need), which would arrive just a few months later.