The 2005 paper by Alex Graves and Jürgen Schmidhuber was a major milestone for sequence labeling, particularly in speech recognition (phoneme classification). While bidirectional RNNs existed since 1997, Graves and Schmidhuber combined them with the LSTM‘s ability to handle long-term dependencies.
The core idea is to process the input sequence using two separate hidden layers—one moving forward in time and one moving backward—and then concatenating their outputs at each time step.
1. The Bidirectional Logic
For any given time step $t$, the model has access to:
- Past context: Via the forward layer $\overrightarrow{h}_t$.
- Future context: Via the backward layer $\overleftarrow{h}_t$.
The final representation for that time step is typically the concatenation: $h_t = [\overrightarrow{h}_t ; \overleftarrow{h}_t]$.
2. PyTorch Implementation
While PyTorch provides a bidirectional=True flag in its standard nn.LSTM, here is how you build it from the ground up using the LSTM cells we’ve discussed to understand the data flow.
Python
import torch
import torch.nn as nn
class BidirectionalLSTM(nn.Module):
def __init__(self, input_size, hidden_size, output_size):
super(BidirectionalLSTM, self).__init__()
self.hidden_size = hidden_size
# Two independent LSTM cells
self.forward_cell = nn.LSTMCell(input_size, hidden_size)
self.backward_cell = nn.LSTMCell(input_size, hidden_size)
# The output layer takes the concatenated hidden states (2 * hidden_size)
self.fc = nn.Linear(hidden_size * 2, output_size)
def forward(self, x):
# x shape: [batch_size, seq_len, input_size]
batch_size, seq_len, _ = x.size()
device = x.device
# 1. Initialize states for both directions
h_f = torch.zeros(batch_size, self.hidden_size).to(device)
c_f = torch.zeros(batch_size, self.hidden_size).to(device)
h_b = torch.zeros(batch_size, self.hidden_size).to(device)
c_b = torch.zeros(batch_size, self.hidden_size).to(device)
# Lists to store hidden states for each time step
forward_states = [None] * seq_len
backward_states = [None] * seq_len
# 2. Forward Pass
for t in range(seq_len):
h_f, c_f = self.forward_cell(x[:, t, :], (h_f, c_f))
forward_states[t] = h_f
# 3. Backward Pass
for t in reversed(range(seq_len)):
h_b, c_b = self.backward_cell(x[:, t, :], (h_b, c_b))
backward_states[t] = h_b
# 4. Concatenation & Output
# Combine [forward_t, backward_t] for every t
combined_outputs = []
for f, b in zip(forward_states, backward_states):
combined_outputs.append(torch.cat((f, b), dim=1))
# Stack to shape: [seq_len, batch_size, hidden_size * 2]
out = torch.stack(combined_outputs)
# 5. Framewise Classification (as per the paper)
# Apply the linear layer to every time step
logits = self.fc(out)
return logits # [seq_len, batch_size, output_size]
3. Why this was a breakthrough for Speech
In speech recognition (phoneme classification), the sound you are hearing now is often disambiguated by the sound that comes immediately after it.
- Standard LSTM: Only knows what has been said so far. It has to “guess” until the next phoneme arrives.
- Bidirectional LSTM: Can “wait” to see the entire utterance before deciding what the phoneme at $t=5$ was, using context from $t=6, 7, 8…$
4. Key Implementation Nuance: Padding
When using bidirectional models on mini-batches of varying lengths, you must be careful with padding. If you don’t use pack_padded_sequence, the “backward” layer will start by processing a bunch of zero-padding tokens, which can corrupt the initial hidden state for the actual data.