Vanishing vs Exploding Gradients

In deep learning, particularly with recurrent neural networks (RNNs) or very deep architectures, the “vanishing” and “exploding” gradient problems are two sides of the same coin. Both stem from the way we update weights using the Chain Rule during backpropagation.

---

1. The Core Issue

When calculating the gradient of the loss function with respect to a weight in an early layer, we multiply several gradients together. If we represent the gradient as $G$:

• Vanishing: If the gradients are small (e.g., $< 1$), multiplying many of them causes the product to shrink exponentially toward zero. Early layers stop learning.
• Exploding: If the gradients are large (e.g., $> 1$), the product grows exponentially. This leads to “NaN” (Not a Number) errors or weights that oscillate wildly.

---

2. Comparison Table

Feature	Vanishing Gradient	Exploding Gradient
Symptom	Training slows down or stops; loss plateaus.	Model weights become NaN; loss fluctuates or goes to infinity.
Activation Cause	Sigmoid or Tanh (slopes are $< 1$ at extremes).	Large weights + many layers + no normalization.
Common Architectures	Deep Vanilla CNNs, standard RNNs.	RNNs with long sequences, very deep networks without scaling.
PyTorch Detection	weight.grad values are extremely close to 0.	weight.grad values are huge or NaN.

---

3. How to Fix Them in PyTorch

Addressing Vanishing Gradients

• Activation Functions: Use ReLU ($f(x) = \max(0, x)$) or its variants (Leaky ReLU). ReLU has a constant gradient of 1 for all positive inputs, preventing the shrink.
• Weight Initialization: Use He Initialization (for ReLU) or Xavier/Glorot (for Sigmoid/Tanh). This keeps the variance of activations consistent across layers.Pythontorch.nn.init.kaiming_normal_(layer.weight, nonlinearity='relu')
• Batch Normalization: nn.BatchNorm2d or nn.LayerNorm re-centers and re-scales data, keeping it in the “active” range of activation functions.
• Residual Connections: Skip connections (ResNets) allow the gradient to “bypass” layers, flowing directly to earlier parts of the net.

Addressing Exploding Gradients

• Gradient Clipping: This is the most common fix in PyTorch, especially for RNNs. It forces the gradient norm to stay below a certain threshold.Python# After loss.backward(), before optimizer.step() torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
• Weight Regularization: Adding L2 penalty (weight_decay in your optimizer) keeps weight magnitudes small.
• Smaller Learning Rate: A high learning rate compounds the explosion; lowering it can provide stability.

---

4. Debugging in PyTorch

If you suspect your gradients are misbehaving, you can inspect them manually after calling loss.backward():

Python

for name, param in model.named_parameters():
    if param.grad is not None:
        print(f"{name} grad norm: {param.grad.norm()}")

• If the norms are consistently $1e-7$ or smaller, they are vanishing.
• If they are $1e+5$ or NaN, they are exploding.