Compact-CTC: Efficient Speech & Sequence Models

• Compact-CTC is a framework that adapts the CTC method for efficient neural speech and sequence processing by reducing computational and memory overhead.
• It leverages phonetic alignment to dynamically compress input sequences and constructs compact WFST decoding graphs, reducing complexity from O(N²) to O(3N).
• It also implements adaptive frame-level token pruning during beam search to accelerate decoding without compromising recognition or translation accuracy.

Compact-CTC defines a class of adaptations to the Connectionist Temporal Classification (CTC) framework targeted at reducing computational, memory, and representational costs in neural speech and sequence models. These adaptations concern (1) dynamic sequence-level compression during encoding, (2) decoding algorithmic compactness via adaptive token pruning, and (3) compact topology construction for decoding graphs used in weighted finite-state transducer (WFST) or lattice-based processing. The unifying principle is the use of phonetic, sequence, and posterior information from CTC alignments to minimize redundant computation and storage without sacrificing recognition or translation accuracy.

1. Dynamic Compression via CTC Alignment in Direct Speech Translation

Compact-CTC enables sequence-level input compression by leveraging CTC-based phonetic alignment predictions. In the canonical pipeline, input Mel-filterbank features are first downsampled with two 2D convolutional layers, effecting a 4× reduction in frame rate. The compressed frames are then passed through a stack of $N_E$ Transformer encoder layers. At an intermediate layer $N_{\text{CTC}}$, a linear-softmax head produces per-frame phone (or token) posterior distributions $P(a_t|X)$.

Consecutive frames predicted to share the same phone class (argmax) are grouped, and aggregation is performed through one of several strategies—unweighted average, confidence-weighted average, or softmax-dominated weighting—collapsing the grouped frames into a single vector. This process yields a variable-length, phonetic-content-adaptive representation, serving as the input to the remaining encoder layers and the cross-attention decoder. The entire model is trained end-to-end with a multi-task loss:

$$\lambda = \mathcal{L}_{\text{CTC}}(E_{N_\text{CTC}}) + \mathcal{L}_{\text{CE}}(D_{N_D}),$$

where $\mathcal{L}_{\text{CTC}}$ supervises the auxiliary CTC head and $\mathcal{L}_{\text{CE}}$ is the label-smoothed cross-entropy for translation (Gaido et al., 2021).

2. WFST Topology Optimizations and the “Compact-CTC” Graph

Standard CTC decoding with WFSTs incurs quadratic complexity in vocabulary size due to direct transitions between all symbol pairs. Compact-CTC, as introduced in (Laptev et al., 2021), eliminates this redundancy with a novel WFST topology. Instead of direct $u\to v$ transitions, transitions pass through a shared blank state via $\varepsilon$-labeled arcs:

• Each non-blank state $q_u$ has an optional self-loop $(u:u)$;
• From $N_{\text{CTC}}$0, an $N_{\text{CTC}}$1-arc $N_{\text{CTC}}$2 leads to $N_{\text{CTC}}$3;
• From $N_{\text{CTC}}$4, all possible $N_{\text{CTC}}$5 arcs target $N_{\text{CTC}}$6 for $N_{\text{CTC}}$7;
• $N_{\text{CTC}}$8 also features a $N_{\text{CTC}}$9 self-loop.

The number of arcs is reduced from $P(a_t|X)$0 (standard CTC) to $P(a_t|X)$1, resulting in 1.24–1.31× smaller TLG decoding graphs and roughly double the feasible batch size during LF-MMI training on large-vocabulary tasks. Recognition performance remains unchanged; e.g., compact-CTC under MMI on LibriSpeech achieves test WER deviations of $P(a_t|X)$2 from standard CTC (Laptev et al., 2021).

3. Efficient Decoding via Frame-Level Token Pruning

Compact-CTC decoding further encompasses algorithmic improvements to standard beam search. Frame-Level Token Pruning (FLToP CTC) applies both top-N static pruning and a dynamic, frame-relative threshold criterion during beam expansion. For each frame $P(a_t|X)$3, only tokens $P(a_t|X)$4 with $P(a_t|X)$5 (where $P(a_t|X)$6 is a tunable threshold and $P(a_t|X)$7 is the highest probability) are considered for hypothesis extension.

Empirical findings on LibriSpeech with a wav2vec2-large backbone include:

• Reducing top-N expansions from $P(a_t|X)$8 to $P(a_t|X)$9 yields a 3.94× speedup with no WER degradation.
• With $$\lambda = \mathcal{L}_{\text{CTC}}(E_{N_\text{CTC}}) + \mathcal{L}_{\text{CE}}(D_{N_D}),$$0 and $$\lambda = \mathcal{L}_{\text{CTC}}(E_{N_\text{CTC}}) + \mathcal{L}_{\text{CE}}(D_{N_D}),$$1, runtime improves 10.5× and memory use 2.78× over baseline, with WER held constant at 3.852%.
• More aggressive pruning (smaller $$\lambda = \mathcal{L}_{\text{CTC}}(E_{N_\text{CTC}}) + \mathcal{L}_{\text{CE}}(D_{N_D}),$$2) yields diminishing speed gains and stable or slightly improved WER (Shree et al., 10 Oct 2025).

This method introduces no structural changes to the decoder other than modifying the per-frame expansion loop; memory footprint and hardware footprint are minimized, and the technique integrates cleanly with both CPU and GPU implementations.

4. Empirical Impact and Quantitative Benchmarking

Across both speech translation and ASR, Compact-CTC substantially reduces resource requirements:

Task	Model/Config	Compact-CTC Effect	Accuracy Impact	Reference
Direct ST (en$$\lambda = \mathcal{L}_{\text{CTC}}(E_{N_\text{CTC}}) + \mathcal{L}_{\text{CE}}(D_{N_D}),$$3it)	8L PH AVG	0.89× RAM, +1.1 BLEU	BLEU: 22.1$$\lambda = \mathcal{L}_{\text{CTC}}(E_{N_\text{CTC}}) + \mathcal{L}_{\text{CE}}(D_{N_D}),$$423.2	(Gaido et al., 2021)
Direct ST (en$$\lambda = \mathcal{L}_{\text{CTC}}(E_{N_\text{CTC}}) + \mathcal{L}_{\text{CE}}(D_{N_D}),$$5de)	8L PH AVG	0.80× RAM, +1.4 BLEU	BLEU: 20.4$$\lambda = \mathcal{L}_{\text{CTC}}(E_{N_\text{CTC}}) + \mathcal{L}_{\text{CE}}(D_{N_D}),$$621.8
ASR (WFST)	Compact topology	1.5× smaller graphs, 2× batch	$$\lambda = \mathcal{L}_{\text{CTC}}(E_{N_\text{CTC}}) + \mathcal{L}_{\text{CE}}(D_{N_D}),$$70.1% WER difference	(Laptev et al., 2021)
ASR (decoding)	FLToP ($$\lambda = \mathcal{L}_{\text{CTC}}(E_{N_\text{CTC}}) + \mathcal{L}_{\text{CE}}(D_{N_D}),$$8, $$\lambda = \mathcal{L}_{\text{CTC}}(E_{N_\text{CTC}}) + \mathcal{L}_{\text{CE}}(D_{N_D}),$$9)	10.5× faster, 2.78× less RAM	WER: 3.864$\mathcal{L}_{\text{CTC}}$03.852	(Shree et al., 10 Oct 2025)

Sequence compression slightly harms ASR WER (increase by 0.3–0.5) but systematically improves ST BLEU. WFST-based Compact-CTC shows negligible degradation for ASR. The FLToP decoding improvements are consistent across devices and vocabularies, indicating the robustness of the compact approach in low-resource settings.

5. Design Considerations, Limitations, and Trade-offs

Compression effectiveness is contingent on placement within the encoder; collapsing too early (e.g., post-subsampling) degrades performance, as discriminative phonetic features have not yet emerged. The simple averaging aggregation outperforms weighted and softmax-based schemes, suggesting that uniform representations best preserve downstream information flow.

Methods relying on off-the-shelf phone aligners (e.g., Gentle) are sensitive to alignment quality. Compression is more beneficial for tasks tolerant to frame detail loss (ST) than phoneme-sensitive ones (ASR). WFST training with compact-CTC requires careful handling of $\mathcal{L}_{\text{CTC}}$1-arcs; practical heuristics include doubling network output length to insert dummy alignment states. LF-MMI training creates large denominator lattices at initialization, manageable via aggressive pruning and batch-size scaling.

6. Comparisons and Directions for Further Research

Prior approaches to phone-informed compression involved multi-stage/cascaded systems and external phone recognizers, resulting in additional latency and potential error propagation. Compact-CTC achieves end-to-end compressive modeling within a single architecture. Relative to fixed downsampling or uniform stacking, dynamic compact-CTC compression preserves information in phonetically rich segments and aggressively compresses silence or redundancy.

Future research directions identified include:

• Extending Compact-CTC scale to larger multilingual corpora.
• Hybrid/combinatorial compression strategies, including dynamic placement of collapse layers and mixed compress/no-compress paths.
• Integration of explicit duration modeling for improved segment boundary detection.
• Real-time factor and streaming latency evaluation.
• Merging Compact-CTC with low-rank/sparse attention for further resource savings (Gaido et al., 2021).
• Adaptive and hardware-specialized pruning strategies guided by acoustic and device signals (Shree et al., 10 Oct 2025).

7. Summary

Compact-CTC unifies several advances for resource-efficient sequence modeling in speech processing: dynamic encoding compression based on CTC phonetic alignments (Gaido et al., 2021), topology compaction in WFST-based decoders (Laptev et al., 2021), and adaptive frame-level token pruning during CTC beam decoding (Shree et al., 10 Oct 2025). These techniques collectively yield significant reductions in memory, compute, and graph size without compromising translation or recognition accuracy, and provide explicit axes for accuracy-efficiency trade-offs that can be tuned per deployment scenario.