SiamCTC: Flexible CTC for Speech SSL

• SiamCTC is a self-supervised speech representation framework that uses a Siamese network and CTC-based dynamic programming to align temporally distorted views.
• It employs a flexible, monotonic alignment approach combined with temporal InfoNCE to preserve local sequence structure and handle speed perturbations.
• Empirical evaluations on LibriSpeech benchmarks show improved phoneme and word error rates, demonstrating robustness against varied speaking rates.

Searching arXiv for papers on SiamCTC and related speech SSL methods. SiamCTC is a self-supervised speech representation learning framework that combines a Siamese architecture with Connectionist Temporal Classification (CTC) to learn temporally coherent, content-focused embeddings that are robust to speaking-rate and other temporal variations. It was introduced to address a central limitation of prior Siamese and contrastive speech SSL methods: their frequent reliance on strict frame-wise correspondence between two views of the same utterance, which breaks under speed perturbation or other time-warping transformations. By replacing frame-level matching with a flexible, monotonic alignment over continuous hidden states, SiamCTC targets linguistic content invariance while preserving temporal coherence and local sequence structure (Eom et al., 1 Jun 2026).

1. Problem setting and conceptual motivation

SiamCTC is formulated around two different temporal views of the same utterance, typically an original waveform and a perturbed version produced by speed or pitch modification. The intended invariance is not generic augmentation robustness in the abstract, but specifically invariance to temporal realizations of the same phonetic or lexical content. In the formulation described for SiamCTC, the same phonetic or word sequence spoken faster or slower should induce similar representation trajectories even when the two sequences differ in length and local timing (Eom et al., 1 Jun 2026).

The framework is motivated by the limitations of prior Siamese and contrastive speech SSL methods, including SPIN, R-SPIN, C-Siam, and DinoSR, which generally assume strict frame-wise correspondence between views. Under strong temporal augmentations, that assumption fails because the $t$-th frame in one view no longer corresponds to the $t$-th frame in the other. SiamCTC therefore relaxes frame-wise constraints while keeping the order structure that is characteristic of speech. This suggests a design principle in which temporal invariance is treated as an alignment problem rather than as an ordinary pointwise similarity objective (Eom et al., 1 Jun 2026).

The method also targets local context preservation. The goal is not only to align content globally across different speaking rates, but also to ensure that neighboring frames remain distinguishable and coherent. This requirement is important because a purely flexible alignment objective can otherwise be satisfied by degenerate representations that collapse local structure.

2. Architecture and objective formulation

The model uses a shared encoder $f_\theta$, instantiated as a pre-trained self-supervised speech encoder such as HuBERT-base or WavLM-base, as provided by S3PRL. For an input waveform $X$ and a perturbed view $\tilde{X}$, the encoder produces

$$Z = f_\theta(X) \in \mathbb{R}^{T \times 256}, \quad \tilde{Z} = f_\theta(\tilde{X}) \in \mathbb{R}^{\tilde{T} \times 256}.$$

The second view is created using speed perturbation factors $\{0.8, 0.9, 1.0, 1.1, 1.2\}$ and pitch perturbation factors $\{-2,-1,0,1,2\}$. A linear prediction layer $\phi$ is applied on the perturbed branch, and a separate linear prediction head $\psi$ is used for the temporal contrastive component (Eom et al., 1 Jun 2026).

The central operation is to treat the original representation sequence $t$0 as a target sequence of pseudo-labels and the perturbed representation $t$1 as the prediction sequence. For each pair of perturbed-time index $t$2 and target position $t$3, SiamCTC computes an attention score $t$4, conceptually via a scaled dot-product or cosine similarity. These scores form a logit matrix that CTC interprets as per-time-step distributions over target positions, together with a dedicated blank column. The CTC loss then marginalizes over monotonic alignment paths: $t$5 Because standard CTC requires input length $t$6 target length, SiamCTC transposes the logits when the perturbed sequence is shorter, as in speed-up cases (Eom et al., 1 Jun 2026).

CTC alone is insufficient, because many-to-one alignments can admit partially collapsed solutions. SiamCTC therefore adds a temporal InfoNCE loss, denoted TINCE, defined over each sequence separately. The positive pair is $t$7, the adjacent frame, while negatives are sampled at least $t$8 steps away, with $t$9 negatives per anchor and cosine similarity at temperature $f_\theta$0. TINCE encourages adjacent frames to remain similar but distinct and distant frames to remain more dissimilar, thereby preserving local temporal structure (Eom et al., 1 Jun 2026).

A third term, $f_\theta$1, compares a soft alignment distribution derived from normalized attention logits with a hard alignment path obtained from Viterbi decoding over CTC alignment probabilities or from DTW between $f_\theta$2 and $f_\theta$3. The full objective is

$f_\theta$4

with $f_\theta$5 and $f_\theta$6 in the reported experiments (Eom et al., 1 Jun 2026).

3. Monotonic temporal alignment as the defining mechanism

The defining technical feature of SiamCTC is its use of CTC as a differentiable alignment mechanism over continuous latent sequences rather than over discrete transcript labels. For two views of lengths $f_\theta$7 and $f_\theta$8, CTC considers all alignment paths that map time indices in the perturbed view to positions in the original view, plus blanks, under a non-decreasing constraint. Multiple perturbed frames can align to the same original frame, and some frames can align to blank. This is the sense in which the alignment is flexible but monotonic (Eom et al., 1 Jun 2026).

That monotonicity matters because it preserves the temporal order of linguistic events. SiamCTC does not attempt arbitrary sequence matching; it assumes the two views realize the same content in the same order but at different rates. Segment boundaries are therefore elastic, but the path cannot jump backward in time. In slowed speech, where $f_\theta$9, CTC naturally maps multiple time steps to the same target position. In speeded-up speech, where $X$0, the transposed-logit procedure ensures that CTC’s length constraint still holds (Eom et al., 1 Jun 2026).

The attention score matrix functions as the interface between Siamese representation learning and CTC dynamic programming. Forward–backward computation marginalizes over all valid monotonic paths, while Viterbi or DTW can be used to extract a single best hard path for the consistency term. A plausible implication is that SiamCTC occupies an intermediate position between strict frame matching and unconstrained sequence comparison: it retains order-aware dynamic programming while avoiding the rigidity of one-to-one alignment.

4. Training regime, datasets, and induced invariances

The reported configuration uses HuBERT-base and WavLM-base encoders pre-trained on LibriSpeech 960h. SiamCTC fine-tuning pre-training is then conducted on LibriSpeech train-clean-100, with evaluation on LibriSpeech test-clean. Optimization uses AdamW with a peak learning rate of $X$1, linear warmup over the first 1,000 updates, linear decay thereafter, a total of 5,000 updates, total batch size 8, and gradient clipping with max norm 1.0. The alignment head uses a temperature of 2.0 for alignment logits and a dedicated blank column; the negative log-probability of blank is fixed at $X$2 when used in TINCE (Eom et al., 1 Jun 2026).

The induced invariances are explicitly tied to speaking-rate and tempo variation. SiamCTC aligns different speed factors of the same utterance and forces them toward similar hidden trajectories while maintaining temporally structured embeddings. The result is a representation that is globally aligned across temporal deformations yet locally structured at frame or segment level. The authors report that removing TINCE harms phoneme recognition significantly, which they interpret as evidence that TINCE preserves fine-grained content information (Eom et al., 1 Jun 2026).

The framework also encourages invariance to pitch perturbation and minor acoustic variations introduced through augmentation, and reduced sensitivity to speaker style variations related to rate and tempo. This suggests that SiamCTC is especially suited to downstream settings in which speaking-rate mismatch between pretraining and deployment is a significant source of degradation.

5. Empirical results and ablation structure

Evaluation follows SUPERB-style benchmarks on phoneme recognition, measured by Phoneme Error Rate, and automatic speech recognition, measured by Word Error Rate, using LibriSpeech test-clean. For HuBERT-based models, the baseline yields PER $X$3 and WER $X$4; HuBERT + SPIN yields PER $X$5 and WER $X$6; HuBERT + LASER yields PER $X$7 and WER $X$8; and HuBERT + SiamCTC yields PER $X$9 and WER $\tilde{X}$0, giving the best PER among HuBERT variants while remaining slightly behind LASER on WER. For WavLM-based models, the baseline yields PER $\tilde{X}$1 and WER $\tilde{X}$2; WavLM + SPIN yields PER $\tilde{X}$3 and WER $\tilde{X}$4; WavLM + LASER yields PER $\tilde{X}$5 and WER $\tilde{X}$6; and WavLM + SiamCTC yields PER $\tilde{X}$7 and WER $\tilde{X}$8, which is the best PER and best WER among WavLM variants (Eom et al., 1 Jun 2026).

The ablation study isolates the three loss components. On HuBERT phoneme recognition, CTC only gives PER $\tilde{X}$9; CTC + KLD gives $$Z = f_\theta(X) \in \mathbb{R}^{T \times 256}, \quad \tilde{Z} = f_\theta(\tilde{X}) \in \mathbb{R}^{\tilde{T} \times 256}.$$0; CTC + TINCE gives $$Z = f_\theta(X) \in \mathbb{R}^{T \times 256}, \quad \tilde{Z} = f_\theta(\tilde{X}) \in \mathbb{R}^{\tilde{T} \times 256}.$$1; and CTC + KLD + TINCE gives $$Z = f_\theta(X) \in \mathbb{R}^{T \times 256}, \quad \tilde{Z} = f_\theta(\tilde{X}) \in \mathbb{R}^{\tilde{T} \times 256}.$$2. The reported interpretation is that CTC alone is not enough, TINCE is crucial for strong performance, and KLD has a smaller but consistent effect when combined with TINCE (Eom et al., 1 Jun 2026).

Robustness to speaking-rate variation is evaluated directly by testing on speed factors $$Z = f_\theta(X) \in \mathbb{R}^{T \times 256}, \quad \tilde{Z} = f_\theta(\tilde{X}) \in \mathbb{R}^{\tilde{T} \times 256}.$$3. At speed $$Z = f_\theta(X) \in \mathbb{R}^{T \times 256}, \quad \tilde{Z} = f_\theta(\tilde{X}) \in \mathbb{R}^{\tilde{T} \times 256}.$$4, HuBERT has PER approximately $$Z = f_\theta(X) \in \mathbb{R}^{T \times 256}, \quad \tilde{Z} = f_\theta(\tilde{X}) \in \mathbb{R}^{\tilde{T} \times 256}.$$5 and SiamCTC approximately $$Z = f_\theta(X) \in \mathbb{R}^{T \times 256}, \quad \tilde{Z} = f_\theta(\tilde{X}) \in \mathbb{R}^{\tilde{T} \times 256}.$$6. At $$Z = f_\theta(X) \in \mathbb{R}^{T \times 256}, \quad \tilde{Z} = f_\theta(\tilde{X}) \in \mathbb{R}^{\tilde{T} \times 256}.$$7, HuBERT peaks at $$Z = f_\theta(X) \in \mathbb{R}^{T \times 256}, \quad \tilde{Z} = f_\theta(\tilde{X}) \in \mathbb{R}^{\tilde{T} \times 256}.$$8, whereas SiamCTC remains below $$Z = f_\theta(X) \in \mathbb{R}^{T \times 256}, \quad \tilde{Z} = f_\theta(\tilde{X}) \in \mathbb{R}^{\tilde{T} \times 256}.$$9. At $\{0.8, 0.9, 1.0, 1.1, 1.2\}$0, HuBERT is approximately $\{0.8, 0.9, 1.0, 1.1, 1.2\}$1, whereas SiamCTC is approximately $\{0.8, 0.9, 1.0, 1.1, 1.2\}$2. These results support the claim that SiamCTC significantly reduces error under speed perturbation and learns content representations that are more stable across rate variation (Eom et al., 1 Jun 2026).

6. Relation to adjacent CTC-based Siamese methods, limitations, and extensions

SiamCTC belongs to a broader pattern in which CTC is used to relax hidden alignment decisions, but its task and mechanism are distinct from other CTC-based Siamese or quasi-Siamese models. In simultaneous NMT, for example, CTC has been used to marginalize over unknown placements of a special <wait> token so that the model can adaptively decide when to read more source context and when to emit target tokens; there CTC addresses timing ambiguity in online translation rather than latent alignment between two continuous views of the same utterance (Chousa et al., 2019). In CTC-triggered Siamese ASR, by contrast, two dropout-perturbed copies of the same input are passed through a joint CTC-attention recognizer, and cosine similarity is enforced only at CTC spike frames; that method improves robustness and uniformity in end-to-end ASR but does not use CTC to align two variable-rate continuous representation sequences (Gao et al., 2022).

Within speech SSL, SiamCTC is positioned against wav2vec 2.0, HuBERT, WavLM, and Data2vec as an additional self-supervised stage rather than a replacement pretraining paradigm. It is also contrasted with CPC-style future prediction and with Siamese approaches such as SPIN, R-SPIN, C-Siam, DinoSR, and LASER. The conceptual distinction emphasized for SiamCTC is that CTC serves as a learned monotonic alignment layer over latent pseudo-labels, permitting multiple valid paths and accommodating stronger temporal augmentations than frame-indexed objectives typically allow (Eom et al., 1 Jun 2026).

The framework has explicit assumptions and limitations. It requires paired views of the same utterance and therefore access to the raw waveform and augmentation pipeline. The alignment attention involves quadratic time and space in sequence length through the attention matrix, together with CTC dynamic programming, so it can be computationally heavy. Performance is reported as sensitive to alignment temperature, augmentation choice and strength, and the negative sampling strategy in TINCE. The reported experiments use pre-trained encoders and only 100 hours of additional training, so behavior in fully from-scratch large-scale settings is left open (Eom et al., 1 Jun 2026).

Future directions mentioned for SiamCTC include training from scratch instead of only fine-tuning HuBERT or WavLM, combining CTC alignment with masking-based objectives, incorporating vector quantization to align discrete speech units, and developing more robust hyperparameter strategies, especially for alignment temperature and augmentation strength. Likely application areas beyond the directly evaluated benchmarks include low-resource ASR, speaker normalization, cross-lingual transfer, and query-by-example speech search. These are presented as plausible extensions rather than established results (Eom et al., 1 Jun 2026).