SiamCTC: Learning Speech Representations through Monotonic Temporal Alignment

Abstract: Self-supervised speech representation learning has made significant progress through Siamese networks, which leverage different views of the same input. However, existing methods often require frame-wise alignment between these views, overlooking the broader linguistic context invariance across different speaking styles. We introduce SiamCTC, a framework that integrates Siamese networks with Connectionist Temporal Classification (CTC) to learn speech representations without strict frame-level correspondence. By employing CTC loss to establish flexible, monotonic alignments between differing temporal realizations of the same content, SiamCTC accommodates speed perturbations and other temporal augmentations. This design relaxes frame-wise constraints while preserving temporal coherence and enhancing robustness to speaking-rate variations in downstream tasks. Our experiments demonstrate that SiamCTC leads to more adaptable speech representations, particularly at diverse speaking rates.

• The paper introduces SiamCTC, which fuses Siamese networks with CTC-based monotonic alignment to overcome strict frame-wise constraints in speech SSL.
• It employs a composite loss combining CTC, KL divergence, and Temporal InfoNCE to maintain global sequence coherence while enhancing local contrast.
• Experiments demonstrate significant improvements in phoneme error rate and robustness to rate variability, outperforming conventional baselines.

SiamCTC: Learning Speech Representations through Monotonic Temporal Alignment

Introduction

The study presents SiamCTC, a self-supervised framework enhancing speech representation learning by fusing the representational invariance of Siamese networks with the monotonic temporal alignment capabilities of Connectionist Temporal Classification (CTC). Conventional SSL methods for speech, such as contrastive and predictive frameworks, typically enforce strict frame-wise alignment between augmented views, constraining their robustness to natural variations like speaking rate. SiamCTC addresses these limitations by leveraging CTC’s flexible, monotonic alignment, coupled with tailored auxiliary objectives to safeguard local context and discourage representational collapse.

Figure 1: The SiamCTC framework processes two temporally perturbed views in parallel via a shared encoder, aligning them with CTC loss and contrastive regularization.

Methods

Siamese Temporal Representation with Flexible Alignment

SiamCTC generates two temporally perturbed versions of each utterance—the original and an augmented sequence (e.g., speed-perturbed)—and passes both through a shared encoder to produce latent representations. This dual-view approach extends the paradigm of vision-based Siamese SSL (e.g., SimCLR, MoCo, BYOL) to speech while relaxing the stringent frame correspondence requirements that hinder previous speech SSL frameworks.

A key innovation is the use of CTC loss to optimize the monotonic mapping between the original and augmented representation sequences. Instead of relying on discrete phonetic targets or frame-to-frame correspondence, SiamCTC treats one representation as a “pseudo-label” and learns to align the other, permitting rate and style variations while preserving linguistic sequential structure. Additional alignment is encouraged via Kullback-Leibler divergence between hard alignments (using Viterbi or DTW paths) and the model’s soft attention-based alignment.

Temporal InfoNCE (TINCE) loss regularizes the representations to avoid collapse by maximizing local context: adjacent latent frames are encouraged to remain distinct, and negative samples are drawn from temporally distant positions. The composite loss combines CTC, KL, and InfoNCE objectives, enforcing both global sequence coherence across views and local temporal distinctiveness.

Experimental Setup

SiamCTC is built atop existing SSL encoders, specifically HuBERT and WavLM pretrained on LibriSpeech. The models are further trained with the SiamCTC objective on the train-clean-100 subset for computational efficiency, and evaluated on the test-clean split. The framework adopts aggressive temporal augmentations, primarily speed perturbations, to assess invariance and robustness. Baselines include naive HuBERT/WavLM, as well as SPIN and LASER, which also attempt to address speaker and rate invariance.

Results

Supervised Task Performance

SiamCTC yields clear improvements over both baselines and recent Siamese extensions. In phoneme recognition, HuBERT+SiamCTC reduces PER from 5.41% to 4.32%, outperforming SPIN and LASER. Similarly, WavLM+SiamCTC achieves the lowest WER (5.73%) among its cohort and achieves a substantial PER reduction (4.84% to 3.96%).

Robustness to Temporal Variability

A central claim is confirmed by experiments on speed-perturbed evaluation sets. While HuBERT’s PER increases steeply as speech rate diverges from the training condition, SiamCTC’s performance remains stable under both speed-ups and slowdowns, with error rates increasing much less sharply.

Figure 2: Phoneme Error Rate on LibriSpeech test-clean for varying speed factors, showing SiamCTC’s rate robustness compared to HuBERT.

Ablation Analysis

Ablation studies dissect loss contributions. CTC loss alone improves over the baseline but is prone to mode collapse. Augmenting with TINCE loss leads to substantial further improvement in PER, and the combination of CTC, TINCE, and KL divergence produces the best performance. These findings validate the necessity of both monotonic temporal alignment and local contrastive regularization.

Implications and Discussion

SiamCTC’s relaxation of the frame-wise alignment constraint enables effective exploitation of a broader range of augmentations, supporting greater robustness to temporal distortions inherent in real-world speech. The framework endows downstream ASR and phoneme recognition systems with capacity to maintain content-invariant representations under variable rates, which is critical for deployment across speakers, languages, and conditions.

From a theoretical viewpoint, the integration of a monotonic (“soft, latent labeling”) alignment objective with the invariance-driven design of Siamese networks advances SSL towards learning abstract sequence structure, not mere fine-scale artifact matching. This approach is extensible to other sequence-to-sequence domains—e.g., translation, multimodal fusion—where monotonic alignment is plausible but explicit framewise supervision is infeasible.

Potential further directions include training SiamCTC from scratch without relying on HuBERT/WavLM initialization, expanding augmentation diversity (e.g., incorporating masking strategies), and extending to discrete unit prediction via vector quantization for spoken language modeling or cross-lingual scenarios.

Conclusion

SiamCTC introduces a principled composition of monotonic alignment, local temporal contrast, and alignment consistency into the Siamese SSL paradigm for speech. Empirical results demonstrate strong improvements in recognition accuracy and robustness under temporal perturbation, substantiating the utility of CTC-based monotonic alignment as a flexible alternative to frame-level matching. The method has both practical and theoretical impact, providing a foundation for future advances in robust, content-preserving self-supervised sequence representation learning.