- The paper introduces a novel approach that fuses self-supervised speech representations with learned dynamic programming to accurately detect word boundaries.
- It demonstrates significant improvements, including over 16% accuracy boost on TIMIT compared to traditional aligners, validated on multiple datasets.
- The method scales to over 1,100 languages and shows robust zero-shot performance on unseen languages through an encoder-decoder design.
Multilingual Word-Level Forced Alignment with Self-Supervised Representations and Learned Dynamic Programming
Overview
The paper "Multilingual Word-Level Forced Alignment with Self-Supervised Representations and Learned Dynamic Programming" (2606.10675) presents a new approach for accurate forced alignment of word boundaries in speech across multiple languages. The method uniquely combines representations from two self-supervised models: Massively Multilingual Speech (MMS) and UnSupSeg, a phoneme boundary detector. The architecture leverages an alignment encoder, which fuses these complementary representations to estimate word-boundary probabilities, and a dynamically learned alignment decoder that applies a customized dynamic programming strategy. Empirical evaluations demonstrate consistent improvements over established aligners such as the Montreal Forced Aligner (MFA), CTC-based MMS alignment, and WhisperX—even on languages not seen during training.
Technical Contributions
Representation Fusion for Alignment
The proposed system employs two pretrained models to generate alignment-relevant features:
- UnSupSeg: Provides frame-level acoustic representations tailored for detecting fine-grained phoneme boundaries, trained with self-supervised contrastive learning. These representations are particularly effective for capturing local acoustic transitions.
- MMS-CTC: Utilizes CTC-based word alignment probabilities from the MMS model, which supports more than 1,100 languages. Each feature encodes the likelihood that a given frame corresponds to a specific word boundary as inferred by large-scale multilingual pretraining.
The representations from both models are concatenated and normalized before being input into the alignment encoder. This fusion capitalizes on the complementary strengths of both granular phoneme segmentation and robust multilingual word-level modeling.
Alignment Encoder
The alignment encoder is responsible for producing refined, frame-level word boundary probabilities. The study evaluates three architectural backbones: VGG, Transformer, and Conformer. The encoder is optimized as a binary classifier over frames, using focal loss to mitigate severe class imbalance (as word boundaries are sparse relative to non-boundaries).
Notably, the Conformer architecture demonstrates the best trade-off between context modeling and computational efficiency. The encoder operates at a high-resolution temporal scale (10 ms frames) and supports long context windows, crucial for accurate placement of word boundaries.
Learned Dynamic Programming Decoder
The alignment decoder is a learned dynamic programming module parameterized by weights on feature functions. It takes the encoder's output, the original representations, and the word sequence to infer the optimal alignment path. Feature functions include:
- Localized boundary discontinuity (via Euclidean distance on UnSupSeg vectors)
- Encoder-derived boundary likelihood at each candidate position
- Aggregated encoder output within word spans (with negative contribution for misalignment)
- Letter emission probabilities from MMS across the time interval corresponding to each word
Feature weights are trained iteratively, following principles from structured prediction. The decoder imposes minimum duration constraints and employs a maximization strategy over all feasible alignments.
Experimental Results and Analysis
Results on English Datasets
Evaluations are conducted on TIMIT and Buckeye, two manually annotated English corpora. The results show that the proposed Multilingual Word Aligner (MWA) outperforms all baselines, particularly MFA and MMS-CTC alignment, across strict (10 ms) and relaxed (100 ms) tolerance thresholds. At a 10 ms threshold, MWA improves absolute accuracy by over 16% on TIMIT and nearly 10% on Buckeye when compared to MFA. MWA also demonstrates substantial gains over WhisperX and Nvidia-Canary-1B models.
Zero-Shot Transfer to Unseen Languages
A central claim is that MWA generalizes robustly without retraining to languages outside the training set (Hebrew, Dutch, German), owing to its reliance on MMS and language-agnostic UnSupSeg features. Results show that, for Hebrew and German, MWA achieves accuracy comparable to or exceeding MFA at strict thresholds and always outperforms MMS-CTC alignment. Notably, on Dutch, where traditional MFA alignment is weak due to sparse G2P coverage, MWA achieves alignment accuracy more than double that of MFA (29% vs. 11.6% at 50 ms).
Ablation and Architecture Comparisons
Among encoder architectures, VGG and Conformer are superior to Transformer, indicating the necessity of local convolutional feature extraction for boundary detection. Fine-tuning and decoupled optimization, necessitated by the non-differentiable decoder, further boost overall accuracy.
Implications and Future Directions
Theoretical Impacts
The work challenges the dominance of traditional HMM-GMM forced aligners (exemplified by MFA) by demonstrating that self-supervised representations, when appropriately fused and decoded, can yield superior word-level alignments. Importantly, the model does not require explicit phoneme-based lexicons or G2P conversions during inference, enhancing portability across languages and domains.
Practical Significance
Scalability to over 1,100 languages is enabled by the MMS backbone, with demonstrated zero-shot transfer to unseen languages. This capability has direct benefits for under-resourced languages and for linguistic research in settings lacking annotated phoneme resources. Furthermore, the methodological framework—featuring encoder-decoder modularity, decoupled optimization, and learnable dynamic programming—can be extended to related alignment tasks, including segmentation and labeling in other modalities.
Prospective Research
Potential directions include joint or end-to-end training strategies to bridge the current reliance on separate encoder and decoder optimization, more sophisticated confidence aggregation from CTC outputs, and exploration of additional self-supervised tasks for enhanced multilingual generalization. Extending the approach to syllable- or phoneme-level forced alignment with minimal supervision also appears tractable.
Conclusion
The paper introduces a principled and empirically validated method for multilingual word-level forced alignment, outperforming established aligners on both monolingual and cross-lingual tasks. The synergy of self-supervised phoneme segmentation, robust multilingual CTC alignment, and a parameterized dynamic programming decoder offers a path toward accurate, scalable, and language-agnostic forced alignment in speech processing (2606.10675).
