Reference: https://threepointformula.wordpress.com/wp-admin/post.php?post=1567
This paper, authored by Sepp Hochreiter in 1991, is the seminal work that first identified the vanishing and exploding gradient problem in recurrent neural networks (RNNs). While it didn’t propose the full LSTM architecture as we know it today (that came in 1997), it laid the mathematical groundwork for “Constant Error Carousels” to solve the gradient issues.
Since the thesis focuses on the fundamental failure of standard RNNs to capture long-term dependencies, a PyTorch implementation involves demonstrating a Vanilla RNN and the logic used to analyze gradient flow.
1. The Vanilla RNN (The Problematic Architecture)
Hochreiter’s thesis proved that as we backpropagate through time, the gradient is multiplied by the weight matrix repeatedly. If the weights are small, the gradient vanishes; if large, it explodes.
Python
import torch
import torch.nn as nn
class VanillaRNN(nn.Module):
def __init__(self, input_size, hidden_size, output_size):
super(VanillaRNN, self).__init__()
self.hidden_size = hidden_size
# Standard RNN equations:
# h_t = tanh(W_ih * x_t + b_ih + W_hh * h_{t-1} + b_hh)
self.rnn = nn.RNN(input_size, hidden_size, batch_first=True)
self.fc = nn.Linear(hidden_size, output_size)
def forward(self, x):
# Initialize hidden state with zeros
h0 = torch.zeros(1, x.size(0), self.hidden_size).to(x.device)
# Forward propagate RNN
out, hn = self.rnn(x, h0)
# Reshape for the linear layer
out = self.fc(out[:, -1, :])
return out
2. Monitoring Gradient Decay (The Thesis Core)
To replicate Hochreiter’s findings, we need to track the norm of the gradients of the hidden states across many time steps. In a “dynamic neural network,” the gradient $\frac{\partial E}{\partial h_t}$ can be expressed as:
$$\frac{\partial E}{\partial h_t} = \frac{\partial E}{\partial h_T} \prod_{k=t}^{T-1} \frac{\partial h_{k+1}}{\partial h_k}$$
If the spectral radius of the weight matrix $W$ is less than 1, the product $\prod \frac{\partial h_{k+1}}{\partial h_k}$ tends toward zero.
Python
def analyze_gradients(model, input_data, target):
model.zero_grad()
output = model(input_data)
loss = nn.MSELoss()(output, target)
loss.backward()
# Access gradients of the recurrent weight matrix
# This is where Hochreiter observed the 'vanishing' effect
for name, param in model.named_parameters():
if 'weight_hh' in name:
grad_norm = param.grad.norm()
print(f"Gradient Norm for {name}: {grad_norm.item()}")
3. The “Constant Error Carousel” (The Solution)
The thesis proposed that to prevent the gradient from vanishing, the internal state should be updated linearly, effectively making $\frac{\partial h_{k+1}}{\partial h_k} \approx 1$.
In PyTorch, this logic eventually became the LSTM. Below is a simplified implementation showing the “Linear Path” (Cell State) that Hochreiter argued for:
Python
class ConstantErrorCarousel(nn.Module):
"""
Simplified logic inspired by Hochreiter's 1991 thesis:
Maintaining a linear flow of information.
"""
def __init__(self, input_size, hidden_size):
super().__init__()
self.hidden_size = hidden_size
self.input_weights = nn.Linear(input_size, hidden_size)
def forward(self, x, state=None):
if state is None:
state = torch.zeros(x.size(0), self.hidden_size)
# Instead of h_t = tanh(W * h_{t-1} + ...),
# we focus on h_t = h_{t-1} + transformation(x)
# This additive update ensures the gradient flow is roughly 1.0
new_info = torch.tanh(self.input_weights(x))
new_state = state + new_info
return new_state
Summary of Findings in the Paper
- The Vanishing Gradient: In deep or long-recurrent networks, the error signal effectively disappears, making it impossible for the model to learn relationships spanning more than 5–10 steps.
- Sensitivity Analysis: Hochreiter used this to show that the “importance” of an input at time $t$ to an output at time $T$ decays exponentially.
- Path to LSTM: This thesis was the “Warning Shot” for the AI community, leading directly to the 1997 LSTM paper where Gating Units were introduced to control this linear flow.
Leave a Reply