Pytorch implementation of paper: Peephole Connections (2000): Gers and Schmidhuber introduced “Peephole” connections, allowing gates to “look” at the cell state directly for better timing.

The Peephole LSTM (Gers et al., 2000) was introduced to solve tasks where the network needs to measure precise time intervals. In a standard LSTM, the gates only see the previous hidden state $h_{t-1}$ and current input $x_t$. If the output gate is closed, the rest of the network has no idea what is happening inside the “vault” of the cell state $c_t$.

Peephole connections allow the gates to “peep” at the cell state $c_t$ even when the output gate is closed, providing the gates with the internal context necessary for precise timing.

1. The Mathematical Change

The gates are modified to include the cell state in their activation. Note that the cell state used for the input and forget gates is $c_{t-1}$, while the output gate uses the updated $c_t$:

• Forget Gate: $f_t = \sigma(W_f x_t + U_f h_{t-1} + \mathbf{V_f c_{t-1}} + b_f)$
• Input Gate: $i_t = \sigma(W_i x_t + U_i h_{t-1} + \mathbf{V_i c_{t-1}} + b_i)$
• Output Gate: $o_t = \sigma(W_o x_t + U_o h_{t-1} + \mathbf{V_o c_{t}} + b_o)$

2. PyTorch Implementation

In PyTorch, the peephole connections are implemented as diagonal weight matrices (element-wise multiplication) rather than full matrix multiplications to keep the parameter count manageable.

Python

import torch
import torch.nn as nn

class PeepholeLSTMCell(nn.Module):
    def __init__(self, input_size, hidden_size):
        super(PeepholeLSTMCell, self).__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        
        # Standard weights for x and h
        self.x2h = nn.Linear(input_size, 4 * hidden_size)
        self.h2h = nn.Linear(hidden_size, 4 * hidden_size)
        
        # Peephole weights (diagonal/element-wise)
        # Note: We only need 3 peephole vectors (f, i, o)
        self.w_pc = nn.Parameter(torch.Tensor(hidden_size)) # for input gate
        self.w_pf = nn.Parameter(torch.Tensor(hidden_size)) # for forget gate
        self.w_po = nn.Parameter(torch.Tensor(hidden_size)) # for output gate
        
        self.reset_parameters()

    def reset_parameters(self):
        nn.init.orthogonal_(self.x2h.weight)
        nn.init.orthogonal_(self.h2h.weight)
        nn.init.constant_(self.w_pc, 0)
        nn.init.constant_(self.w_pf, 0)
        nn.init.constant_(self.w_po, 0)

    def forward(self, x, states):
        h_prev, c_prev = states
        
        # 1. Pre-calculate x and h transformations
        pre_gates = self.x2h(x) + self.h2h(h_prev)
        f_pre, i_pre, g_pre, o_pre = pre_gates.chunk(4, 1)
        
        # 2. Forget Gate (sees c_prev)
        f = torch.sigmoid(f_pre + self.w_pf * c_prev)
        
        # 3. Input Gate (sees c_prev)
        i = torch.sigmoid(i_pre + self.w_pc * c_prev)
        
        # 4. Cell State Update
        g = torch.tanh(g_pre)
        c_next = (f * c_prev) + (i * g)
        
        # 5. Output Gate (sees updated c_next)
        o = torch.sigmoid(o_pre + self.w_po * c_next)
        
        # 6. Hidden State
        h_next = o * torch.tanh(c_next)
        
        return h_next, c_next

---

3. Key Technical Notes

• Temporal Precision: Peephole connections are particularly famous for allowing LSTMs to learn the “Counting” task or “Clock” tasks where the network must emit a pulse every $N$ steps.
• Parameter Count: Using diagonal peephole connections only adds $3 \times \text{hidden\_size}$ parameters, which is negligible compared to the $W$ and $U$ matrices.
• Why $c_{next}$ for Output Gate? The output gate is responsible for deciding which parts of the already updated memory should be visible to the rest of the network, hence it uses the current state rather than the previous one.