Fully Differentiable Neural Forced Alignment via Soft Dynamic Programming

Abstract: Recent advances in sequence modeling have significantly improved ASR systems, bringing them close to human-level recognition accuracy and enhancing robustness across diverse acoustic conditions and languages. In contrast, Forced Alignment has not experienced comparable progress, and traditional HMM-GMM frameworks remain widely adopted and highly competitive. To address this gap, we propose an end-to-end, fully differentiable neural architecture specifically designed for phoneme alignment. The model consists of an encoder that processes the input signal and a decoder that produces alignment decisions. The encoder is structured into two complementary branches: one dedicated to phoneme identity verification and the other to phoneme boundary detection. The decoder is implemented as a trainable module based on differentiable soft dynamic programming. The entire system is optimized end-to-end using a novel contrastive loss that encourages clear separation between steady-state phoneme regions and transition boundaries. The proposed approach outperforms the current state of the art in phoneme alignment on hand-annotated English benchmarks, achieves strong word-level generalization results, and demonstrates generalization on unseen languages.

• The paper introduces FALCON, a fully differentiable neural forced alignment system that integrates contrastive acoustic representation, linguistic contextualization, and soft dynamic programming.
• The model achieves state-of-the-art phoneme and word-level alignment accuracy across languages and datasets, outperforming traditional HMM-GMM and recent neural approaches.
• Ablation studies highlight the critical role of the MNCE objective and differentiable DP module in enhancing segmentation precision compared to standard methods.

Fully Differentiable Neural Forced Alignment via Soft Dynamic Programming

Introduction

The paper "Fully Differentiable Neural Forced Alignment via Soft Dynamic Programming" (2606.25460) proposes a novel end-to-end neural framework for phoneme-level forced alignment (FA). Addressing the persistent gap between robust automatic speech recognition (ASR) pipelines and the temporal precision required for phoneme segmentation, the approach unifies contrastive acoustic representation, linguistic contextualization, and a differentiable alignment criterion via soft dynamic programming (Soft-DP). Rigorous empirical evaluations demonstrate that this method, termed FALCON, achieves state-of-the-art phoneme and word-level alignment accuracy across datasets and languages, establishing a new neural benchmark for the forced alignment problem.

Methodology

System Architecture

The model consists of three principal subsystems:

1. Representation Encoder: Learns boundary-sensitive acoustic representations using a modified noise contrastive estimation (MNCE) objective to distinguish intra-phoneme from boundary frames.
2. Context Encoder: Aggregates segmental context, producing frame-wise phoneme class posteriors conditioned on the phoneme transcript using a BiLSTM stack.
3. Differentiable Decoder (Soft-DP): Performs alignment via soft dynamic programming, directly optimizing boundary selection in a fully differentiable manner.

The end-to-end pipeline is visualized below.

Figure 1: Overall workflow: waveform → representation encoder (latent features) → context encoder (phoneme posteriors) → decoder (Soft-DP for alignments).

Representation Encoder and Contrastive Learning

The encoder $f_\theta$ transforms the speech signal into a sequence of high-resolution feature vectors $Z$, optimized to enhance spectral contrasts at phoneme transitions. The MNCE objective maximizes similarity within stable phoneme regions while minimizing it near phoneme boundaries by dynamic sampling of positive (intra-phoneme) and negative (boundary-region) pairs:

Figure 2: Positive/negative sample selection within phoneme segments, facilitating contrastive learning focused on alignment-relevant cues.

An explicit segmentation of the latent space emerges, mirroring true phoneme boundaries.

Figure 3: (a) Input spectrogram; (b) encoder output $z$. Ground-truth boundaries (red) align closely with features' transitions.

Context Encoder

The context encoder $g_\psi$, implemented as a deep BiLSTM, maps latent acoustic representations to frame-wise posterior probability distributions over the phoneme inventory. This incorporates transcript constraints and imposes linguistic consistency on the segmentation process.

Figure 4: Frame-wise phoneme posterior map; color encoding indicates class likelihood per frame, grouped by phoneme identity.

Decoder: Soft Dynamic Programming

Alignment is performed via a differentiable DP module. The decoder computes local alignment scores using:

• $\phi_1$: Frame-wise cosine dissimilarity and its derivative (acoustic transitions).
• $\phi_2$: Segmental phoneme label confidence (linguistic consistency).

The derivative of frame-wise cosine similarity exhibits clear peaks at phoneme transitions, serving as robust boundary cues.

Figure 5: Cosine similarity (red) and its derivative (pink) over $z$, with peaks indicating likely boundaries (green: model boundaries, red: ground truth).

The DP table is constructed over all candidate segment boundaries, where weight smoothing via a temperature hyperparameter $\gamma$ enables gradient flow. Boundary hypotheses are integrated with Soft-DP, yielding expected boundary times rather than discrete indices.

Figure 6: Soft-DP matrix: accumulated scores over possible boundary placements, yielding smooth expected alignment paths.

Experimental Results

Phoneme-Level Alignment

FALCON achieves superior accuracy over classical HMM-GMM aligners (Montreal Forced Aligner, MFA) and recent neural systems (MMS, WhisperX, Nvidia-Canary-1B) on English (TIMIT, Buckeye) phoneme alignment at practical tolerances (25–100 ms), despite a modest drop at the strictest threshold due to the 10 ms frame resolution. Notably, FALCON is the only neural aligner reporting consistent phoneme-level results.

Cross-Lingual and Word-Level Generalization

FALCON demonstrates strong zero-shot generalization to Dutch, German, and Hebrew. Unlike MFA and other systems, FALCON requires no language-specific models or phoneme inventories at inference. On previously unobserved languages, the model consistently outperforms classical and neural baselines both for phoneme- and word-level alignment, even without retraining or adaptation. Word-level boundary detection is accomplished by aggregating phoneme segmentations, and matches or surpasses state-of-the-art word-level aligners at all but the strictest (10 ms) thresholds.

Ablation Study

Substituting InfoNCE (standard contrastive loss) for MNCE leads to a sizeable decrease in alignment accuracy, establishing the importance of the alignment-focused contrastive objective. Furthermore, replacing Soft-DP with non-differentiable decision modules (DP or naive peak-picking) results in a marked reduction in segmentation precision, underpinning the essential role of joint end-to-end differentiation.

Theoretical and Practical Implications

1. Direct Optimization of Boundary Precision: Unlike CTC-based systems or ASR-aligned methods, FALCON directly backpropagates alignment error through to the acoustic and contextual modules, surmounting limitations associated with recognition-byproduct timestamps and marginalization over alignments.
2. Universality and Robustness: The learned segmentation cues are not language- or inventory-specific, implying that the model infers universal phonetic transition signals from the waveform. This property is especially impactful for developing FA systems in low-resource or under-documented languages.
3. Modular Integration: The architecture separates acoustic discrimination, contextual inference, and temporal decoding, permitting future reconfiguration, e.g., by incorporating more expressive encoders, attention-based context models, or alternative loss functions.
4. Efficiency and Accessibility: Although training complexity is elevated (due to Soft-DP within the learning loop), inference runtimes are on par with highly-optimized HMM-GMM systems, enabling practical deployment.

Future Directions

Key avenues for further research include finer-resolution acoustic feature modeling (e.g., hierarchical encoders), language-agnostic phoneme inventory mapping, joint conformer-based or self-supervised pretraining integration, and more scalable/efficient differentiable DP approximations. Joint training on hybrid phoneme-grapheme objectives could bridge the gap towards unified end-to-end segmentation and recognition. The demonstrated strong transfer to word segmentation, without explicit supervision, suggests exploration into richer linguistic alignment tasks.

Conclusion

This work delivers an authoritative contribution to neural forced alignment, validating a fully differentiable, contrastive, context-aware approach optimized for temporal boundary detection. Empirical results confirm that joint end-to-end neural optimization with alignment-centric contrastive representations and differentiable soft-DP decisively improves performance over traditional FA systems and neural ASR-aligned models. The architecture's strong transferability to new languages and segmentation granularities, in the absence of in-domain data or retraining, signifies its practical utility for speech technology in low-resource and cross-lingual scenarios.