Connectionist Temporal Classification

• Connectionist Temporal Classification (CTC) is a sequence-level objective and alignment-marginalization method for training models that map an input sequence to a shorter, ordered target sequence, without requiring frame-level alignment, key characteristics are monotonic alignment, conditional independence of path symbols, and efficient forward-backward computation.
• Applications of CTC range from speech and handwriting recognition to gesture, sign-language decoding, summarization, and machine translation, utilizing models like RNNs, CNNs, and self-attention networks, making it a versatile tool across various sequence-to-sequence tasks.
• Key problems addressed by CTC include the monotonicity assumption, which limits it to order-preserving tasks, and the conditional independence assumption, which can be mitigated by integrating external language models, attentive architectures, graph-based supervision, or by using variational and latent-variable approaches, depending on the applications and trade-offs from CNT.

Connectionist Temporal Classification (CTC) is a sequence-level objective and alignment-marginalization method for training models that map an input sequence to a shorter, ordered target sequence without requiring frame-level alignment. It introduces a special blank symbol, defines frame-level paths containing labels, repetitions, and blanks, and assigns a target sequence the summed probability of all paths that collapse to it. CTC has been applied to speech, handwriting, gesture and sign-language recognition, audio tagging, summarization, simultaneous translation, machine translation, and phonetic transcription. Its defining properties are monotonic alignment, conditional independence of path symbols given the input, and efficient forward–backward computation.

1. Mathematical formulation and alignment marginalization

Let the input sequence be $\mathbf{x}=(x_1,\ldots,x_T)$ and the target sequence be $\mathbf{y}=(y_1,\ldots,y_U)$, generally with $U\leq T$. The model produces, at each input position, a probability distribution over an output vocabulary $\mathcal{Y}$ augmented with a blank symbol $\varnothing$:

$\mathcal{Y}'=\mathcal{Y}\cup\{\varnothing\}.$

A CTC path is a length-$T$ sequence

$$\boldsymbol{\pi}=(\pi_1,\ldots,\pi_T),\qquad \pi_t\in\mathcal{Y}'.$$

The collapse function $\mathcal{B}$ first merges consecutive repetitions of the same nonblank symbol and then removes blanks. Thus,

$$\mathcal{B}(\varnothing,A,A,\varnothing,B,B)=(A,B),$$

whereas

$\mathbf{y}=(y_1,\ldots,y_U)$0

The blank is therefore necessary for representing consecutive occurrences of the same target label. The set of paths compatible with a target is

$\mathbf{y}=(y_1,\ldots,y_U)$1

Under the CTC conditional-independence assumption, the path probability factorizes over time:

$\mathbf{y}=(y_1,\ldots,y_U)$2

where $\mathbf{y}=(y_1,\ldots,y_U)$3 is the network posterior for the symbol at time $\mathbf{y}=(y_1,\ldots,y_U)$4. Although the factors are independent at the output level, the encoder producing them may use the complete input sequence through an RNN, LSTM, bidirectional LSTM, CNN, self-attention network, or Conformer.

The target probability is obtained by marginalizing all compatible paths:

$\mathbf{y}=(y_1,\ldots,y_U)$5

The training objective is the negative log-likelihood:

$\mathbf{y}=(y_1,\ldots,y_U)$6

For a training set, the objective is summed or averaged over examples. CTC consequently replaces externally supplied segmentation with latent alignment marginalization: training observes the input sequence and complete target sequence, but not which frames emitted individual labels.

CTC assumes monotonicity. Target labels must occur in the same relative order as the input positions, although multiple frames may correspond to one label and arbitrary stretches of input may be assigned to blanks. This property is appropriate for speech, handwriting, continuous gestures, and other order-preserving sequence problems, but it restricts tasks requiring unconstrained reordering.

2. Forward–backward computation and optimization

Direct enumeration of $\mathbf{y}=(y_1,\ldots,y_U)$7 is exponential in sequence length. CTC instead inserts blanks at the beginning, end, and between target labels:

$\mathbf{y}=(y_1,\ldots,y_U)$8

whose length is $\mathbf{y}=(y_1,\ldots,y_U)$9. A forward variable $U\leq T$0 records the total probability of paths that consume the first $U\leq T$1 input positions and end at extended-target position $U\leq T$2. Under a standard indexing convention,

$U\leq T$3

The two-position skip is legal when the destination is nonblank and differs from the symbol two positions earlier:

$U\leq T$4

This restriction prevents two identical target labels from being merged without an intervening blank. The final target likelihood is obtained from the final blank and final-label states:

$U\leq T$5

A backward variable $U\leq T$6 computes corresponding suffix probabilities. The forward–backward products provide posterior occupancy values,

$U\leq T$7

which are aggregated over extended-target states corresponding to the same symbol. These posteriors supply gradients with respect to frame-level logits. Implementations commonly use log-domain computation or numerical scaling to avoid underflow.

The standard CTC dynamic program has approximately $U\leq T$8 complexity per example, rather than exponential complexity over paths. It is used both in conventional neural CTC systems and in hybrid models in which CTC is placed on top of an LDCRF, a segmental CRF encoder, a CNN, a self-attention encoder, or a variational latent representation.

CTC’s conditional independence is both a restriction and a computational advantage. It does not directly model dependencies among successive output labels, such as spelling, syntax, or word-level language regularities. External LLMs can therefore be integrated during decoding. Alternatively, output-history information can be incorporated into the encoder or CTC representation without replacing the CTC objective.

3. Decoding and linguistic integration

The simplest decoding procedure is greedy best-path decoding. At each time step, the most probable symbol is selected:

$U\leq T$9

and the resulting path is collapsed:

$\mathcal{Y}$0

Greedy decoding is computationally inexpensive and requires no LLM. It is not generally equivalent to maximum-probability sequence decoding, because the single most probable path need not collapse to the sequence that receives the greatest total probability from all paths.

Prefix beam search retains several high-probability partial hypotheses and merges paths that share collapsed prefixes. It can reduce search error relative to greedy decoding. In CTC acoustic modeling, language-model-enhanced decoding can combine CTC scores with character- or word-level linguistic information.

Weighted finite-state transducer (WFST) decoding uses a token transducer for CTC blank and repetition behavior, a lexicon transducer mapping characters or phonemes to words, and a grammar transducer representing word sequences through an $\mathcal{Y}$1-gram LLM. This approach supports fixed-vocabulary recognition and integrates spelling, pronunciation, and linguistic constraints efficiently. In one comparison, WFST decoding was strongest, particularly with phoneme-based acoustic units, reaching $\mathcal{Y}$2 WER on Eval2000, whereas greedy character decoding reached $\mathcal{Y}$3 and character beam decoding reached $\mathcal{Y}$4 (Zenkel et al., 2017).

Character-level recurrent LLMs provide an open-vocabulary alternative. They can generate words absent from a fixed lexicon, although they require beam search and depend on language-model weighting, insertion bonuses, and beam size. In the reported comparison, character beam decoding achieved $\mathcal{Y}$5 Eval2000 WER, compared with $\mathcal{Y}$6 for character WFST decoding and $\mathcal{Y}$7 for phoneme WFST decoding (Zenkel et al., 2017).

CTC can also be combined with autoregressive attention decoding. Hybrid CTC/attention systems use a shared encoder, a CTC branch, and an autoregressive decoder. During training, losses may be interpolated; during decoding, CTC and attention scores may be combined synchronously. In translation, output-synchronous joint decoding is generally more accurate, while input-synchronous decoding can be faster and provides a natural input-based termination condition. The CTC branch supplies monotonic alignment and input-synchronous structure, while the attention branch supplies target-side dependence and flexible reordering (Yan et al., 2022).

4. Encoders and architectural variants

CTC is an objective rather than an encoder architecture. It has therefore been paired with a wide range of sequence representations.

Recurrent encoders remain common. A shared bidirectional LSTM encoder can feed both CTC and a segmental CRF. Joint optimization of the two objectives improved both decoding branches on TIMIT; for example, CTC evaluation PER with fMLLR features improved from $\mathcal{Y}$8 to $\mathcal{Y}$9, while SRNN/SCRF improved from $\varnothing$0 to $\varnothing$1 (Lu et al., 2017). CTC pretraining of the shared encoder accelerated convergence of the joint model.

All-convolutional encoders replace recurrence with one-dimensional temporal convolutions, batch normalization, ReLU activations, max-pooling, and residual blocks. On Switchboard, CNN CTC encoders trained substantially faster and decoded faster than LSTM encoders, although their WER was worse. A representative five-layer LSTM required approximately $\varnothing$2 wall-clock hours per epoch, compared with approximately $\varnothing$3–$\varnothing$4 hours for tested CNN configurations. The best CNN reached $\varnothing$5 Eval2000 WER with a 9-gram LLM, compared with $\varnothing$6 for the LSTM (Krishna et al., 2017).

Self-attention encoders provide global contextual access and parallel temporal computation. SAN-CTC uses a deep self-attentional encoder with temporal downsampling, positional representations, residual connections, layer normalization, and position-wise feed-forward layers. On WSJ, it achieved $\varnothing$7 CER on eval92 with label smoothing and $\varnothing$8 WER with an extended 3-gram LLM. On LibriSpeech test-clean, it achieved $\varnothing$9 CER and $\mathcal{Y}'=\mathcal{Y}\cup\{\varnothing\}.$0 WER with a 4-gram LLM (Salazar et al., 2019).

Attention can also be integrated inside a CTC network without adding an autoregressive decoder. Time-convolution features, content attention, hybrid location-aware attention, an implicit recurrent language-model representation, and component-wise attention improved greedy character CTC on a 3,400-hour Cortana task. The complete system produced approximately a $\mathcal{Y}'=\mathcal{Y}\cup\{\varnothing\}.$1 relative WER reduction over vanilla CTC while retaining CTC training and greedy decoding (Das et al., 2018).

CTC has also been applied to convolutional-recurrent audio tagging, raw-frame sign-language recognition, multi-cue sign-language recognition, and latent-dynamic graphical models. In sign-language recognition, CTC allowed sentence-level gloss prediction without pre-segmenting individual signs. On RWTH-PHOENIX-Weather, a raw-frame LRCN achieved $\mathcal{Y}'=\mathcal{Y}\cup\{\varnothing\}.$2 test WER, while a Multi-Cue network using hand shape, movement, and location achieved $\mathcal{Y}'=\mathcal{Y}\cup\{\varnothing\}.$3 (Akandeh, 2022).

5. Applications and extensions

CTC was initially associated with speech recognition but has been adapted to tasks in which ordered labels are available without precise temporal boundaries.

In gesture recognition, CTC was placed on top of a latent-dynamic conditional random field. The resulting CTC-LDCRF retained latent states, discriminative conditional modeling, transition dynamics, and framewise label probabilities while training from sequence-level labels. On the ArmGesture-Continuous task, pretraining with segmented subsequences followed by unsegmented CTC-LDCRF training achieved $\mathcal{Y}'=\mathcal{Y}\cup\{\varnothing\}.$4 frame accuracy, compared with $\mathcal{Y}'=\mathcal{Y}\cup\{\varnothing\}.$5 for a segmented LDCRF, $\mathcal{Y}'=\mathcal{Y}\cup\{\varnothing\}.$6 for an HMM, and $\mathcal{Y}'=\mathcal{Y}\cup\{\varnothing\}.$7 for a segmented CRF (Atashin et al., 2016).

For order-preserving abstractive summarization, CTC maps document positions to output labels and blanks. Blank predictions represent input material interpreted as less important or noisy, while collapse preserves the order of emitted words or characters. On Chinese Gigaword, character-input CTC achieved ROUGE-1 $\mathcal{Y}'=\mathcal{Y}\cup\{\varnothing\}.$8, ROUGE-2 $\mathcal{Y}'=\mathcal{Y}\cup\{\varnothing\}.$9, ROUGE-3 $T$0, and ROUGE-L $T$1, exceeding the reported ordinary and attentive sequence-to-sequence baselines (Lu et al., 2017). Its advantage was concentrated in examples whose reference headlines preserved document order.

Sequentially labelled audio tagging uses ordered event labels without onset and offset annotations. A CRNN-CTC model trained on such labels achieved an average AUC of $T$2, compared with $T$3 for a max-pooled CRNN and $T$4 for an average-pooled CRNN (Hou et al., 2018). The task differs from conventional sound-event detection because the labels specify event order but not exact durations or frame-level boundaries.

CTC has also been used to model adaptive read/write timing in simultaneous translation. A special $T$5 token represents reading another source token rather than emitting target text. The CTC-like objective marginalizes over multiple wait placements that produce the same reference translation, while a delay penalty controls the latency–quality trade-off (Chousa et al., 2019).

In phonetic transcription, graph-based CTC replaces a single phoneme sequence with a graph of alternative pronunciations. Graph Temporal Classification marginalizes both frame-level alignments and pronunciation alternatives. On English TIMIT, 2-best GTC reduced PER from $T$6 for 1-best CTC to $T$7; on Belgian Dutch, 3-best GTC reduced PER from $T$8 to $T$9 (Grafé et al., 5 Sep 2025).

Other extensions alter the supervision or probabilistic structure. Acoustic-landmark CTC inserts phone-boundary or manner-transition symbols during pretraining and then fine-tunes with phone-only CTC. Mixed Label 2 achieved $$\boldsymbol{\pi}=(\pi_1,\ldots,\pi_T),\qquad \pi_t\in\mathcal{Y}'.$$0 PER on 61-phone TIMIT compared with $$\boldsymbol{\pi}=(\pi_1,\ldots,\pi_T),\qquad \pi_t\in\mathcal{Y}'.$$1 for the baseline, an $$\boldsymbol{\pi}=(\pi_1,\ldots,\pi_T),\qquad \pi_t\in\mathcal{Y}'.$$2 relative reduction (He et al., 2018). Manner-based CTC uses a separately trained manner recognizer to modify character CTC paths and produced modest WER reductions on AN4, LibriSpeech, and TEDLIUM-2 (R et al., 2018).

6. Limitations, optimization issues, and recent developments

A central limitation is CTC’s monotonicity assumption. It is well matched to speech, handwriting, gesture sequences, and order-preserving summarization, but it does not directly represent arbitrary reordering. Translation systems therefore require an encoder capable of latent reordering before applying a final monotonic CTC alignment. CTC-only translation remains weaker than attention-only translation; improvements arise when CTC is used jointly with an autoregressive attention decoder (Yan et al., 2022).

The conditional-independence assumption limits direct modeling of output-label dependencies. External LLMs, attention branches, implicit recurrent histories, latent variables, or graph-based supervision can compensate, but they do not remove the formal CTC factorization.

CTC commonly produces blank-dominated and sharply peaked posteriors. Nonblank symbols may appear as short spikes, which are useful for classification and recognition but do not directly supply segment interiors or reliable onset and offset boundaries. A temporal classification and segmentation topology addresses this issue by introducing explicit background and foreground states. It retains CTC-style alignment-free training but expands the target topology to approximately $$\boldsymbol{\pi}=(\pi_1,\ldots,\pi_T),\qquad \pi_t\in\mathcal{Y}'.$$3 states, increasing computation and requiring additional structural assumptions (Zhao, 2019).

Entropy regularization addresses concentration on narrow path distributions. Fixed maximum-entropy CTC subtracts a constant-weight path-entropy term from the CTC loss. AdaMER-CTC adapts the entropy coefficient through a target-entropy constraint. On LibriSpeech with a Conformer-small encoder, standard CTC achieved $$\boldsymbol{\pi}=(\pi_1,\ldots,\pi_T),\qquad \pi_t\in\mathcal{Y}'.$$4 test-all WER, fixed EnCTC achieved $$\boldsymbol{\pi}=(\pi_1,\ldots,\pi_T),\qquad \pi_t\in\mathcal{Y}'.$$5, and AdaMER-CTC achieved $$\boldsymbol{\pi}=(\pi_1,\ldots,\pi_T),\qquad \pi_t\in\mathcal{Y}'.$$6 (Eom et al., 2024).

Variational CTC replaces deterministic frame representations with distributions over latent sequences. One formulation assumes conditionally independent latent variables; another uses a Markovian latent prior. Both retain a CTC prediction term and add a KL regularizer. The proposed formulation is theoretical and does not report datasets, baselines, or numerical experiments (Nan et al., 2023). LV-CTC develops a related latent-variable approach for non-autoregressive ASR, conditioning CTC emissions on a latent sequence intended to encode token dependencies. With iterative refinement and self-distillation, it achieved $$\boldsymbol{\pi}=(\pi_1,\ldots,\pi_T),\qquad \pi_t\in\mathcal{Y}'.$$7 test-clean/test-other WER on LibriSpeech and $$\boldsymbol{\pi}=(\pi_1,\ldots,\pi_T),\qquad \pi_t\in\mathcal{Y}'.$$8 on TED-LIUM2, while retaining non-autoregressive decoding (Fujita et al., 2024).

LCS-CTC constrains CTC using a similarity-aware local alignment module. A frame–phoneme cost matrix and modified longest-common-subsequence procedure identify high-confidence alignment zones. These zones receive local cross-entropy supervision, while masked CTC preserves flexibility elsewhere. On WavLM-L, LCS-CTC reduced LibriSpeech test-other WER-related phoneme error from the reported CTC baseline of $$\boldsymbol{\pi}=(\pi_1,\ldots,\pi_T),\qquad \pi_t\in\mathcal{Y}'.$$9 to $\mathcal{B}$0, reduced boundary loss from $\mathcal{B}$1 ms to $\mathcal{B}$2 ms, and reduced articulatory reconstruction loss from $\mathcal{B}$3 to $\mathcal{B}$4 (Ye et al., 5 Aug 2025).

CTC implementations must also manage temporal downsampling carefully. The output sequence must remain long enough to realize the target, particularly when repeated labels require intervening blanks. Padding requires explicit input and target lengths. Practical systems additionally face decoder search error, language-model and lexicon maintenance, beam-size selection, blank semantics, and possible mismatches between sequence-level training and frame-level evaluation.

CTCModel packages the objective into separate Keras training, prediction, and evaluation branches. The training branch uses TensorFlow’s ctc_batch_cost, the prediction branch uses ctc_decode, and the evaluation branch uses edit_distance to provide loss, label error rate, and sequence error rate (Soullard et al., 2019). More recent graph-based formulations generalize the linear target trellis to weighted finite-state acceptors, allowing CTC-like marginalization over alternative pronunciations and other structured supervision.

CTC is therefore best understood not as a single neural architecture but as a family of alignment-based sequence objectives. Its invariant core is the marginalization of monotonic blank-and-label paths. The principal research directions extend that core by enriching encoders, integrating linguistic or articulatory information, constraining implausible alignments, modeling uncertainty, representing alternative targets as graphs, or combining CTC with autoregressive objectives.