The paper “Effective Approaches to Attention-based Neural Machine Translation” (Luong et al., 2015) introduced what is now commonly known as Luong Attention. It differs from Bahdanau attention by being “top-down”—calculating attention after the decoder’s RNN step—and by offering different scoring functions (Dot, General, and Concat).
1. The Global Attention Mechanism
The core of Luong’s approach is the calculation of a variable-length context vector $c_t$ that is derived from the encoder hidden states $H$. The paper proposes three ways to calculate the alignment score between the current decoder hidden state $s_t$ and an encoder hidden state $h_s$:
$$score(s_t, h_s) =
\begin{cases}
s_t^\top h_s & \text{dot} \
s_t^\top W_a h_s & \text{general} \
v_a^\top \tanh(W_a [s_t; h_s]) & \text{concat}
\end{cases}$$
Python
import torch
import torch.nn as nn
import torch.nn.functional as F
class LuongAttention(nn.Module):
def __init__(self, method, hidden_dim):
super(LuongAttention, self).__init__()
self.method = method
self.hidden_dim = hidden_dim
if self.method == 'general':
self.attn = nn.Linear(hidden_dim, hidden_dim)
elif self.method == 'concat':
self.attn = nn.Linear(hidden_dim * 2, hidden_dim)
self.v = nn.Parameter(torch.FloatTensor(hidden_dim))
def forward(self, decoder_hidden, encoder_outputs):
# decoder_hidden: [1, batch_size, hidden_dim]
# encoder_outputs: [src_len, batch_size, hidden_dim]
src_len = encoder_outputs.size(0)
batch_size = encoder_outputs.size(1)
# Calculate alignment scores
if self.method == 'dot':
# dot product between decoder_hidden and encoder_outputs
attn_energies = torch.sum(decoder_hidden * encoder_outputs, dim=2) # [src_len, batch_size]
elif self.method == 'general':
energy = self.attn(encoder_outputs) # [src_len, batch_size, hidden_dim]
attn_energies = torch.sum(decoder_hidden * energy, dim=2)
elif self.method == 'concat':
# Concatenate decoder hidden and encoder outputs
combined = torch.cat((decoder_hidden.expand(src_len, -1, -1), encoder_outputs), dim=2)
energy = torch.tanh(self.attn(combined)) # [src_len, batch_size, hidden_dim]
attn_energies = torch.sum(self.v * energy, dim=2)
# Normalize scores to probabilities (softmax over src_len)
return F.softmax(attn_energies.t(), dim=1).unsqueeze(1) # [batch_size, 1, src_len]
2. The Attentional Decoder
In Luong attention, the sequence is: RNN Step $\rightarrow$ Attention $\rightarrow$ Context Vector $\rightarrow$ Hidden-to-Output.
Python
class LuongDecoder(nn.Module):
def __init__(self, output_dim, emb_dim, hidden_dim, dropout, attention):
super().__init__()
self.attention = attention
self.embedding = nn.Embedding(output_dim, emb_dim)
self.rnn = nn.LSTM(emb_dim, hidden_dim, dropout=dropout)
# Combined layer to produce 'attentional hidden state' tilde{h}_t
self.concat = nn.Linear(hidden_dim * 2, hidden_dim)
self.out = nn.Linear(hidden_dim, output_dim)
self.dropout = nn.Dropout(dropout)
def forward(self, input, hidden, cell, encoder_outputs):
# input: [batch_size]
input = input.unsqueeze(0)
embedded = self.dropout(self.embedding(input))
# 1. Standard RNN step
# luong uses s_t (current) to calculate attention, unlike bahdanau (s_{t-1})
output, (hidden, cell) = self.rnn(embedded, (hidden, cell))
# 2. Calculate attention weights and context vector
a = self.attention(output, encoder_outputs) # [batch_size, 1, src_len]
encoder_outputs = encoder_outputs.permute(1, 0, 2) # [batch_size, src_len, hidden_dim]
# context vector is weighted sum of encoder outputs
context = torch.bmm(a, encoder_outputs) # [batch_size, 1, hidden_dim]
# 3. Concatenate RNN output and context vector
output = output.squeeze(0) # [batch_size, hidden_dim]
context = context.squeeze(1) # [batch_size, hidden_dim]
# Eq. 5 from paper: h_tilde = tanh(W_c [c_t; h_t])
concat_input = torch.cat((output, context), 1)
concat_output = torch.tanh(self.concat(concat_input))
# 4. Final prediction
prediction = self.out(concat_output)
return prediction, hidden, cell
Key Differences from Bahdanau (2015)
| Feature | Bahdanau (NMT) | Luong (Effective NMT) |
| RNN Structure | Bidirectional GRU | Unidirectional LSTM (often multi-layer) |
| Attention Timing | Before RNN step (uses $s_{t-1}$) | After RNN step (uses $s_t$) |
| Alignment Score | Concat only | Dot, General, or Concat |
| Context Path | Fed into next RNN step | Fed into a final hidden-to-output layer |
Summary of “Local” Attention
The paper also introduced Local Attention, which only looks at a small window of the source sentence centered around a predicted position $p_t$. This is more computationally efficient for very long sequences but is less commonly used today than the Global Attention implemented above.