Pytorch implementation of paper: Effective Approaches to Attention-based Neural Machine Translation

The paper “Effective Approaches to Attention-based Neural Machine Translation” (Luong et al., 2015) introduced what is now commonly known as Luong Attention. It differs from Bahdanau attention by being “top-down”—calculating attention after the decoder’s RNN step—and by offering different scoring functions (Dot, General, and Concat).

1. The Global Attention Mechanism

The core of Luong’s approach is the calculation of a variable-length context vector $c_t$ that is derived from the encoder hidden states $H$. The paper proposes three ways to calculate the alignment score between the current decoder hidden state $s_t$ and an encoder hidden state $h_s$:

$$score(s_t, h_s) =

\begin{cases}

s_t^\top h_s & \text{dot} \

s_t^\top W_a h_s & \text{general} \

v_a^\top \tanh(W_a [s_t; h_s]) & \text{concat}

\end{cases}$$

Python

import torch
import torch.nn as nn
import torch.nn.functional as F

class LuongAttention(nn.Module):
    def __init__(self, method, hidden_dim):
        super(LuongAttention, self).__init__()
        self.method = method
        self.hidden_dim = hidden_dim

        if self.method == 'general':
            self.attn = nn.Linear(hidden_dim, hidden_dim)
        elif self.method == 'concat':
            self.attn = nn.Linear(hidden_dim * 2, hidden_dim)
            self.v = nn.Parameter(torch.FloatTensor(hidden_dim))

    def forward(self, decoder_hidden, encoder_outputs):
        # decoder_hidden: [1, batch_size, hidden_dim]
        # encoder_outputs: [src_len, batch_size, hidden_dim]
        
        src_len = encoder_outputs.size(0)
        batch_size = encoder_outputs.size(1)

        # Calculate alignment scores
        if self.method == 'dot':
            # dot product between decoder_hidden and encoder_outputs
            attn_energies = torch.sum(decoder_hidden * encoder_outputs, dim=2) # [src_len, batch_size]
        
        elif self.method == 'general':
            energy = self.attn(encoder_outputs) # [src_len, batch_size, hidden_dim]
            attn_energies = torch.sum(decoder_hidden * energy, dim=2)
            
        elif self.method == 'concat':
            # Concatenate decoder hidden and encoder outputs
            combined = torch.cat((decoder_hidden.expand(src_len, -1, -1), encoder_outputs), dim=2)
            energy = torch.tanh(self.attn(combined)) # [src_len, batch_size, hidden_dim]
            attn_energies = torch.sum(self.v * energy, dim=2)

        # Normalize scores to probabilities (softmax over src_len)
        return F.softmax(attn_energies.t(), dim=1).unsqueeze(1) # [batch_size, 1, src_len]

---

2. The Attentional Decoder

In Luong attention, the sequence is: RNN Step $\rightarrow$ Attention $\rightarrow$ Context Vector $\rightarrow$ Hidden-to-Output.

Python

class LuongDecoder(nn.Module):
    def __init__(self, output_dim, emb_dim, hidden_dim, dropout, attention):
        super().__init__()
        self.attention = attention
        self.embedding = nn.Embedding(output_dim, emb_dim)
        self.rnn = nn.LSTM(emb_dim, hidden_dim, dropout=dropout)
        
        # Combined layer to produce 'attentional hidden state' tilde{h}_t
        self.concat = nn.Linear(hidden_dim * 2, hidden_dim)
        self.out = nn.Linear(hidden_dim, output_dim)
        self.dropout = nn.Dropout(dropout)

    def forward(self, input, hidden, cell, encoder_outputs):
        # input: [batch_size]
        input = input.unsqueeze(0)
        embedded = self.dropout(self.embedding(input))

        # 1. Standard RNN step
        # luong uses s_t (current) to calculate attention, unlike bahdanau (s_{t-1})
        output, (hidden, cell) = self.rnn(embedded, (hidden, cell))

        # 2. Calculate attention weights and context vector
        a = self.attention(output, encoder_outputs) # [batch_size, 1, src_len]
        encoder_outputs = encoder_outputs.permute(1, 0, 2) # [batch_size, src_len, hidden_dim]
        
        # context vector is weighted sum of encoder outputs
        context = torch.bmm(a, encoder_outputs) # [batch_size, 1, hidden_dim]

        # 3. Concatenate RNN output and context vector
        output = output.squeeze(0) # [batch_size, hidden_dim]
        context = context.squeeze(1) # [batch_size, hidden_dim]
        
        # Eq. 5 from paper: h_tilde = tanh(W_c [c_t; h_t])
        concat_input = torch.cat((output, context), 1)
        concat_output = torch.tanh(self.concat(concat_input))

        # 4. Final prediction
        prediction = self.out(concat_output)
        
        return prediction, hidden, cell

---

Key Differences from Bahdanau (2015)

Feature	Bahdanau (NMT)	Luong (Effective NMT)
RNN Structure	Bidirectional GRU	Unidirectional LSTM (often multi-layer)
Attention Timing	Before RNN step (uses $s_{t-1}$)	After RNN step (uses $s_t$)
Alignment Score	Concat only	Dot, General, or Concat
Context Path	Fed into next RNN step	Fed into a final hidden-to-output layer

Summary of “Local” Attention

The paper also introduced Local Attention, which only looks at a small window of the source sentence centered around a predicted position $p_t$. This is more computationally efficient for very long sequences but is less commonly used today than the Global Attention implemented above.