Pytorch implementation of paper: CTC (2006): Graves introduced Connectionist Temporal Classification, a way to train LSTMs on unsegmented sequence data (like handwriting or audio).

The 2006 paper “Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks” by Alex Graves et al. solved a massive bottleneck in AI: the need for perfectly “aligned” data.

Before CTC, if you wanted to train an LSTM to recognize the word “Hello” in audio, you had to manually label exactly which milliseconds corresponded to the ‘H’, the ‘e’, and so on. CTC allows the model to output a sequence of characters and “aligns” them to the target text automatically using a hidden “blank” token.

1. The CTC Algorithm Logic

CTC introduces a special blank symbol ($\epsilon$). The model predicts a probability distribution over the alphabet plus this blank for every time step. To get the final prediction, CTC:

1. Collapses repeated characters (e.g., h h e l l l o $\rightarrow$ h e l o).
2. Removes the blank tokens (e.g., h - e - l - l - o $\rightarrow$ hello).

---

2. PyTorch Implementation

PyTorch has a highly optimized nn.CTCLoss because the “forward-backward” algorithm required to calculate the loss is computationally expensive.

Python

import torch
import torch.nn as nn

class CTCModel(nn.Module):
    def __init__(self, input_size, hidden_size, num_classes):
        super(CTCModel, self).__init__()
        # 1. Bidirectional LSTM to capture context
        self.lstm = nn.LSTM(input_size, hidden_size, 
                            bidirectional=True, batch_first=True)
        
        # 2. Fully connected layer to map to character probabilities
        # num_classes must include the "blank" token (usually index 0)
        self.fc = nn.Linear(hidden_size * 2, num_classes)
        
        # 3. Log Softmax is required for CTCLoss
        self.softmax = nn.LogSoftmax(dim=2)

    def forward(self, x):
        # x: [batch, seq_len, features]
        lstm_out, _ = self.lstm(x)
        logits = self.fc(lstm_out)
        return self.softmax(logits)

# --- Usage & Training Setup ---

# Example parameters
num_chars = 26 # a-z
num_classes = num_chars + 1 # +1 for CTC blank token
model = CTCModel(input_size=40, hidden_size=256, num_classes=num_classes)

# Define CTC Loss
# zero_infinity=True helps with stability in early training
ctc_loss = nn.CTCLoss(blank=0, zero_infinity=True)

# Dummy Data
# input: [seq_len, batch, features] (Note: CTC expects seq_len first by default)
input_data = torch.randn(50, 16, 40) 
# targets: [sum_of_target_lengths] (1D tensor of all labels concatenated)
targets = torch.randint(1, num_classes, (16, 10)) 

# Lengths are required for CTC to know where sequences end
input_lengths = torch.full(size=(16,), fill_value=50, dtype=torch.long)
target_lengths = torch.full(size=(16,), fill_value=10, dtype=torch.long)

# Forward pass
log_probs = model(input_data.transpose(0, 1)).transpose(0, 1)

# Calculate Loss
loss = ctc_loss(log_probs, targets, input_lengths, target_lengths)
loss.backward()

---

3. Key Concepts of CTC

• The Blank Token: It acts as a “buffer” between characters. This allows the model to predict the same character twice (like the ‘l’ in ‘hello’) without them being collapsed into one.
• Input vs. Target Length: For CTC to work, the input sequence length ($T$) must be greater than or equal to the target sequence length ($L$). If your audio is 1 second long but you have 500 characters of text, the math breaks.
• Decoding: During inference, we usually use Greedy Decoding (taking the argmax at each step and collapsing) or Beam Search Decoding (exploring multiple paths) to find the most likely sentence.

4. Why it was a breakthrough

CTC enabled “End-to-End” speech recognition. You could feed raw audio and the corresponding transcript directly into a loss function, and the model would learn the alignment by itself. This paved the way for modern systems like DeepSpeech and the voice assistants we use today.