NeMo Forced Aligner (NFA) Overview

• NeMo Forced Aligner (NFA) is a neural forced-alignment system that employs an auxiliary CTC-based model to align audio frames with transcription tokens reliably.
• It is integrated as a dedicated post-processing step in ASR and AST pipelines, separating the transcription process from precise timestamp extraction.
• The system delivers high-precision word- and segment-level timestamps, offering robust performance in multilingual ASR while addressing non-monotonic challenges in AST.

The NeMo Forced Aligner (NFA) is a stand-alone neural forced-alignment system, integral to state-of-the-art multilingual automatic speech recognition (ASR) and automatic speech translation (AST) pipelines, particularly within the Canary-1B-v2 and related model families. NFA employs a hybrid approach, leveraging an auxiliary connectionist temporal classification (CTC) ASR model to reliably align transcription tokens with audio frames, producing word- and segment-level timestamps crucial for formal ASR scoring, content retrieval, and downstream adaptive tasks. Its architecture and methodology reflect a division of labor: primary attention-based encoder–decoder models focus on sequence transduction, while NFA is tasked specifically with monotonic, high-precision alignment using robust CTC decoding and dynamic programming. This modular approach addresses the challenges of non-monotonic attention distributions inherent in neural sequence-to-sequence models and supports efficient, accurate timestamping for both monolingual and cross-lingual speech tasks.

1. CTC-Based Forced Alignment Methodology

Fundamental to NFA is the use of an auxiliary CTC-based ASR model, such as Parakeet-TDT-0.6B-v3 (600M parameters), to generate alignments between reference or predicted transcriptions and the input acoustic signal. Forced alignment is framed as an optimization problem over all valid monotonic token–frame alignments:

$$\hat{A} = \arg\max_{A\in\mathcal{A}(y)} \prod_{t=1}^{T} p(a_t|x)$$

where $T$ is the number of time frames, $a_t$ is the aligned token at frame $t$ (including blanks), $\mathcal{A}(y)$ is the set of valid alignments for token sequence $y$, and $p(a_t|x)$ are the CTC log-probabilities for frame $t$ given the audio $x$.

In practice, Viterbi decoding is applied to the sequence of CTC outputs to efficiently compute the most probable alignment path, ensuring temporal monotonicity and reproducibility. The precondition that the auxiliary CTC model uses the same byte pair encoding (BPE) tokenizer as the main encoder–decoder enables direct alignment of the ASR system's vocabulary.

2. Integration in Canary-1B-v2 and Workflow Design

Within the Canary-1B-v2 system (Sekoyan et al., 17 Sep 2025), NFA operates as a dedicated post-processing step after the primary attention-based transcription is generated. The principal workflow is:

1. Inference with Primary Model: The encoder–decoder architecture transcribes the audio.
2. Parallel Audio Routing: The audio is simultaneously routed through the auxiliary CTC model.
3. Alignment Input: The NFA module receives the transcript and the corresponding CTC output matrix (per-frame log-probabilities).
4. Alignment Computation: Using dynamic programming (Viterbi), NFA computes frame-wise alignments for each token in the transcript.
5. Timestamp Extraction: Start and end times are derived for words or segments, which are then aggregated for final output.

This separation of the alignment task from sequence generation allows the decoder to focus on transcription and translation accuracy without needing to predict explicit alignment tokens, a requirement in token-insertion-based methods.

3. Timestamp Granularity and Segmentation Considerations

NFA supports both word-level timing (precise boundaries for each token) and segment-level aggregation (for longer contiguous units), adapting to the demands of ASR and AST:

• ASR: The alignment is inherently monotonic and segment-to-segment, producing accurate word- or phrase-level timestamps.
• AST: Due to the non-monotonic mappings between source and target utterances (introduced by translation reordering), word-level timestamps may be unreliable. Therefore, NFA is used to generate segment-level timestamps for translated output, marking sequence boundaries rather than individual words.

A plausible implication is that, for speech translation where cross-lingual phrase reordering is significant, reliance on NFA for word-level alignment may add temporal ambiguity; thus, segment granularity is preferred.

4. Performance Advantages and Limitations

The use of NFA yields several critical advantages:

• Alignment Robustness: By avoiding reliance on soft cross-attention distributions, which can be non-monotonic and noisy, NFA produces more reliable and temporally stable boundaries (Sekoyan et al., 17 Sep 2025).
• Decoupled Training: The main model's objectives are isolated from the statistically independent task of frame-timed alignment, simplifying optimization and data balancing.
• Inference Efficiency: Although NFA introduces a lightweight auxiliary computation at inference, forced alignment is highly parallelizable and does not impact decoder latency.

For AST, a limitation is that the CTC model, having been trained solely for transcription in the source language, cannot model cross-lingual structure. Segment-level timestamping mitigates this, but fine-grained word alignment in translations remains challenging.

5. Comparative Analysis with Alternative Alignment Strategies

Alternative alignment and timestamping methods include:

Method	Strengths	Weaknesses
Cross-attention DTW	Works directly on attention maps; can be post-hoc applied	Sensitive to non-monotonicities and requires smoothing
Token-prediction	Integrates timestamp tokens (<\	timestamp
NFA (CTC-based)	Robust monotonic alignment, modular design, efficient post-processing	Auxiliary model needed; segment granularity for translation

Relative to cross-attention approaches, NFA avoids the sensitivity to jumps and the need for dynamic time warping (DTW) smoothing. Compared to token-insertion architectures (e.g., those inspired by Whisper), NFA reduces model output complexity and avoids trade-off scenarios between transcription accuracy and timestamping (Hu et al., 21 May 2025).

6. Quantitative Performance and Reporting

Models leveraging NFA as the teacher aligner achieve high-quality timestamp generation. Representative results for timestamp prediction in ASR are:

• Precision between 80% and 90%, e.g., 93.7% on LibriSpeech test-other
• Recall averaging 90.2% across datasets; timestamp errors (start/end) in the 20–120 ms range across four languages
• For AST, timestamp errors are larger, averaging around 200 ms; a trade-off exists, with observed performance regression in BLEU (−4.4) and COMET (−2.6) when timestamping is enabled (Hu et al., 21 May 2025)

This suggests that NFA-enabled pipelines can deliver fine-grained, high-fidelity timing in ASR, and robust, if coarser, segment timestamps for AST.

7. Challenges, Trade-offs, and Future Outlook

Principal challenges center on the adaptation of CTC-based alignment to translation tasks and the limitations of word-level alignment in non-monotonic, cross-lingual settings. The explicit separation of alignment (NFA) and transcription/translation (main model) offers maintainability and extensibility; extensions that leverage multilingual or contrastive pretraining for greater language generality (e.g., as in IPA-ALIGNER (Zhu et al., 2023)) represent an area of ongoing research and differentiation.

The NeMo Forced Aligner is currently positioned as an efficient, reliable forced-alignment module, supporting large-scale multilingual ASR and segment-timed AST applications, offering a practical balance between accuracy, architectural modularity, and computational efficiency.