The paper “Learning to Forget: Continual Prediction with LSTM” (Gers, Schmidhuber, and Cummins, 2000) is the most significant update to the original LSTM. It introduced the Forget Gate, which allows the network to reset its internal state when a sequence ends or information becomes obsolete.
Without this gate, the cell state ($c_t$) can grow indefinitely, leading to numerical instability and a “saturated” memory that can’t learn new patterns.
1. The Core Innovation: The Forget Gate
In the 1997 version, $c_t = c_{t-1} + i_t g_t$. In the 2000 version, the forget gate $f_t$ was added to scale the previous cell state:
$$c_t = f_t \odot c_{t-1} + i_t \odot g_t$$
2. PyTorch Implementation
This implementation follows the precise equations from the 2000 paper, including the forget gate logic.
Python
import torch
import torch.nn as nn
class ForgetGateLSTMCell(nn.Module):
def __init__(self, input_size, hidden_size):
super(ForgetGateLSTMCell, self).__init__()
self.input_size = input_size
self.hidden_size = hidden_size
# Linear layers for the 4 internal components
# (f: forget, i: input, g: cell candidate, o: output)
self.x2h = nn.Linear(input_size, 4 * hidden_size)
self.h2h = nn.Linear(hidden_size, 4 * hidden_size)
def forward(self, x, states):
# states: (h_prev, c_prev)
h_prev, c_prev = states
# Calculate all gate inputs at once
pre_gates = self.x2h(x) + self.h2h(h_prev)
# Split into individual gates
# In the 2000 paper, these are:
# y_in_f (forget), y_in_i (input), y_in_c (cell), y_in_o (output)
f_pre, i_pre, g_pre, o_pre = pre_gates.chunk(4, 1)
# Gate activations (usually Sigmoid for gates, Tanh for cell)
f = torch.sigmoid(f_pre) # Forget Gate: f_t
i = torch.sigmoid(i_pre) # Input Gate: i_t
o = torch.sigmoid(o_pre) # Output Gate: o_t
g = torch.tanh(g_pre) # Cell Input: g_t
# --- THE FORGET GATE UPDATE ---
# c_t = f_t * c_{t-1} + i_t * g_t
c_next = (f * c_prev) + (i * g)
# Hidden state output
h_next = o * torch.tanh(c_next)
return h_next, c_next
3. Comparison: 1997 vs. 2000
The 2000 paper changed how the LSTM handles “continual” tasks where the data stream isn’t chopped into neat, independent segments.
| Feature | Original LSTM (1997) | Forget Gate LSTM (2000) |
| Cell Update | $c_t = c_{t-1} + i_t g_t$ | $c_t = f_t c_{t-1} + i_t g_t$ |
| Gradient Flow | Constant (1.0) | Controlled by $f_t$ |
| Memory | Never resets; can saturate | Resets based on input context |
| Ideal For | Short, segmented sequences | Long-term, continuous time series |
4. Why “Learning to Forget” Matters
The authors demonstrated that on tasks like the Embedded Reber Grammar, the original LSTM would fail once the sequences became too long because the internal state would drift. By “forgetting,” the model can “clear its desk” once a specific sub-task is finished, allowing the same neurons to be reused for the next part of the sequence.
If you are training this on modern hardware, you’ll find that $f_t$ is often initialized with a bias of 1.0. This ensures that at the start of training, the gate is wide open (not forgetting anything), preventing the vanishing gradient problem from returning prematurely.
Leave a Reply