Swan Forced Aligner

• Swan Forced Aligner is a transcript-conditioned, word-level forced alignment system that recovers precise word boundaries and pause structure for expressive TTS.
• It employs an interleaved word/blank topology with dedicated anchor tokens to overcome subword segmentation issues and accurately model gaps.
• Empirical evaluations on Chinese and LibriSpeech datasets show competitive accuracy in aligning speech with transcripts compared to modern forced alignment systems.

Searching arXiv for papers on Swan Forced Aligner and related forced alignment systems. Swan Forced Aligner is a transcript-conditioned, word-level forced alignment model introduced within the SwanData-Speech and SwanVoice pipeline for aligning a speech waveform with its transcript while recovering both word boundary timestamps and pause structure (Li et al., 29 May 2026). Unlike forced alignment methods that treat timing primarily as a by-product of ASR or punctuation restoration, Swan Forced Aligner is designed to anchor lexical words explicitly and decode over an interleaved word/blank topology so that it can model where each word occurs and where silence or gaps occur between words (Li et al., 29 May 2026). In the SwanVoice system, this pause-aware alignment is used for corpus construction, transcription and punctuation correction, and downstream pause-aware text supervision for long-form monologue and dialogue text-to-speech (Li et al., 29 May 2026).

1. Definition and position within forced alignment

Swan Forced Aligner belongs to the family of forced alignment systems that take a known transcript and an audio signal and infer temporal boundaries for linguistic units. In the general literature, forced alignment is distinguished from ASR by the fact that the text is assumed known and the model is required to fit that text to the signal rather than predict the text itself (Rousso et al., 2024). Traditional systems such as Montreal Forced Aligner are typically based on Kaldi-style GMM-HMM pipelines with explicit phone-state temporal structure and Viterbi decoding (Rousso et al., 2024), whereas more recent systems have explored neural end-to-end alignment, CTC-style alignment, ensemble uncertainty estimation, and multilingual timestamp prediction (Li et al., 2022, Kelley, 2 Jun 2025, Shi et al., 29 Jan 2026).

Within this landscape, Swan Forced Aligner is presented not as a generic phoneme aligner or an ASR-derived timestamp heuristic, but as a pause-aware word-level aligner for data preparation in expressive TTS (Li et al., 29 May 2026). The model is specifically optimized for settings where punctuation alone is an unreliable proxy for acoustic pausing. The SwanVoice paper states that modern ASR punctuation is optimized for readability and semantic plausibility rather than actual acoustic pause structure, so punctuation may correlate only weakly with real pauses, hesitations, and phrase boundaries (Li et al., 29 May 2026). This motivates an aligner that grounds text directly in acoustics and exposes pause/gap structure explicitly.

A plausible implication is that Swan Forced Aligner is best understood as a specialized alignment component for speech synthesis corpora rather than as a universal replacement for all audio-based forced aligners. This interpretation is consistent with the paper’s emphasis on pause-aware corpus annotation and downstream generation quality rather than on general phonetic annotation alone (Li et al., 29 May 2026).

2. Role in SwanData-Speech and SwanVoice

Swan Forced Aligner is embedded in the SwanData-Speech pipeline, which constructs monologue and dialogue corpora from in-the-wild audio for training SwanVoice (Li et al., 29 May 2026). The paper situates it in a sequence of processing stages consisting of speech enhancement or separation, diarization, ASR transcription, punctuation refinement, forced alignment, pause-aware punctuation rewriting, and quality filtering (Li et al., 29 May 2026).

Its operational role is explicit. A pretrained forced aligner first aligns the audio with the transcription and assigns a timestamp to each character, after which pauses are defined by the time gap between consecutive characters (Li et al., 29 May 2026). These pause measurements are then used to rewrite punctuation into pause-aware supervision. The paper gives concrete thresholds:

• pauses shorter than 0.08 s are ignored;
• pauses between 0.08 s and 0.18 s insert <|sp|>;
• pauses between 0.18 s and 0.45 s insert a comma;
• pauses longer than 0.45 s use ., !, or ? depending on original punctuation, with . as default;
• punctuation with no observed pause is removed;
• pauses without punctuation cause punctuation insertion (Li et al., 29 May 2026).

This makes the aligner more than a timestamp generator. It is a mechanism for transforming acoustically observed temporal structure into symbolic text supervision. In the SwanVoice framework, that supervision is then used for raw-text conditioning with pause-aware symbols, which the paper identifies as important for long-form expressiveness, dialogue coherence, and pause control (Li et al., 29 May 2026).

The paper further argues that annotation errors inherited from imperfect alignment degrade downstream generation models (Li et al., 29 May 2026). This places Swan Forced Aligner in a foundational position: poor alignment affects pause-aware punctuation, dialogue segmentation, speaker-turn consistency, and controllability. This suggests that, in SwanVoice, alignment quality is treated as a prerequisite for high-quality generative modeling rather than as a secondary preprocessing convenience.

3. Input representation and word-anchored formulation

The formal alignment problem is defined with input speech waveform $x$ and transcript $y=(y_1,\dots,y_N)$ (Li et al., 29 May 2026). The transcript is tokenized into lexical words $g=(g_1,\dots,g_M)$ and then into subword tokens. The model predicts word-level timing intervals

$\{(s_i, e_i)\}_{i=1}^M$

where $s_i$ and $e_i$ are the start and end times of word $i$ (Li et al., 29 May 2026). The paper also states that it provides decoded alignment paths, per-word confidence scores, and pause/gap structure between words (Li et al., 29 May 2026).

A central design problem is that tokenizer segmentation may split a lexical word into multiple subword pieces, whereas the downstream supervision required by SwanData-Speech is word-level. Swan Forced Aligner resolves this by inserting a dedicated anchor token <|wbd|> after each lexical word:

$$\tilde{y} = (t_{1,1}, \dots, t_{1,n_1}, <|wbd|>, \dots, t_{M,1}, \dots, t_{M,n_M}, <|wbd|>)$$

and extracting word-level hidden states at these anchor positions:

$W = (w_1,\dots,w_M)$

(Li et al., 29 May 2026).

This word-anchor mechanism is one of the model’s defining features. It lets the aligner represent each lexical word with a single anchor state even when the underlying tokenizer is subword-based. In contrast, many alignment frameworks operate directly on phoneme or token sequences. For example, Qwen3-ForcedAligner reformulates forced alignment as timestamp-slot filling over a transcript augmented with [time] tokens (Shi et al., 29 Jan 2026), while NeuFA learns alignment from shared ASR/TTS attention and then extracts boundaries from attention patterns (Li et al., 2022). Swan Forced Aligner instead retains a structured monotonic decoding framework but shifts the state representation to explicit lexical-word anchors and blank states (Li et al., 29 May 2026).

A plausible implication is that the <|wbd|> mechanism makes the model particularly suitable for TTS-oriented supervision, where lexical timing and pause structure are more directly useful than fine-grained subword timings.

4. Structured topology and pause-aware blank modeling

The aligner uses an interleaved state space

$$\mathcal{S} = (b_0, w_1, b_1, w_2, \dots, w_M, b_M)$$

where $y=(y_1,\dots,y_N)$0 denotes the word state for the $y=(y_1,\dots,y_N)$1-th word and $y=(y_1,\dots,y_N)$2 denotes a blank or gap state before, between, or after words (Li et al., 29 May 2026). This explicit word–blank topology is the core of its pause-aware design.

The blank states are modeled heterogeneously rather than as a single universal blank embedding. The paper states that $y=(y_1,\dots,y_N)$3 and $y=(y_1,\dots,y_N)$4 are separate learnable initial and final blank states, while internal blank states depend on neighboring words:

$y=(y_1,\dots,y_N)$5

(Li et al., 29 May 2026). The intent is to condition internal blank states on the adjacent word pair so that the model can distinguish short coarticulatory gaps, longer pauses, and phrase boundaries.

This explicit blank modeling aligns Swan Forced Aligner with a broader line of work that treats silence and gaps as first-class alignment objects rather than incidental intervals between labels. BFA, for example, introduces explicit modeling of inter-phoneme gaps and silences in a CTC-based phoneme aligner and predicts both onset and offset boundaries (Rehman et al., 27 Sep 2025). Swan Forced Aligner applies a similar principle at the word level in a structured monotonic decoder (Li et al., 29 May 2026).

The valid alignment path is

$y=(y_1,\dots,y_N)$6

with monotonic transitions of three types: stay, adv1, and adv2 (Li et al., 29 May 2026). The paper states:

• stay: remain in the current state;
• adv1: advance by one state in the topology;
• adv2: skip over an intermediate blank when transitioning into a word state (Li et al., 29 May 2026).

This transition system defines a constrained path space over the interleaved word/blank graph. Compared with purely token-level timestamp predictors or unconstrained attention extraction, this preserves explicit structural priors about monotonicity and gap placement.

5. Acoustic-text backbone, scoring, and decoding

The acoustic side begins with a pretrained acoustic encoder:

$y=(y_1,\dots,y_N)$7

followed by projection and refinement:

$y=(y_1,\dots,y_N)$8

to produce frame-level acoustic features $y=(y_1,\dots,y_N)$9 (Li et al., 29 May 2026).

The text side encodes the tokenized transcript as

$g=(g_1,\dots,g_M)$0

after which the word-anchor states at <|wbd|> positions are extracted to form $g=(g_1,\dots,g_M)$1 (Li et al., 29 May 2026). The paper states that the backbone allows text-conditioned acoustic encoding and audio-conditioned text encoding, indicating that cross-modal interaction is incorporated before decoding (Li et al., 29 May 2026).

For frame $g=(g_1,\dots,g_M)$2 and state $g=(g_1,\dots,g_M)$3, the unary score is

$g=(g_1,\dots,g_M)$4

where $g=(g_1,\dots,g_M)$5 is cosine similarity or dot-product similarity (Li et al., 29 May 2026). Transition preferences are parameterized as

$g=(g_1,\dots,g_M)$6

with learned neural heads conditioned on the destination state (Li et al., 29 May 2026).

A notable implementation choice is score canonicalization. Unary and transition scores are normalized:

$g=(g_1,\dots,g_M)$7

and then rescaled with separate learnable gains:

$g=(g_1,\dots,g_M)$8

(Li et al., 29 May 2026). The paper states that this decoupled scaling improves decoding robustness across machines and environments because Viterbi paths can otherwise change under small numerical differences (Li et al., 29 May 2026). That emphasis on numerical stability is relatively unusual in forced-alignment descriptions and indicates that deployment consistency was a design concern.

The score of a path $g=(g_1,\dots,g_M)$9 is

$\{(s_i, e_i)\}_{i=1}^M$0

where $\{(s_i, e_i)\}_{i=1}^M$1 is the transition type used to enter $\{(s_i, e_i)\}_{i=1}^M$2 (Li et al., 29 May 2026). Default inference uses Viterbi decoding:

$\{(s_i, e_i)\}_{i=1}^M$3

with $\{(s_i, e_i)\}_{i=1}^M$4 the set of valid monotonic paths (Li et al., 29 May 2026). The paper also supports an optional posterior-based decoding mode using forward–backward inference, described as more robust under locally ambiguous evidence (Li et al., 29 May 2026).

In methodological terms, Swan Forced Aligner therefore remains close to the classical alignment-as-decoding paradigm, but it modernizes the state representation, blank modeling, and numerical stabilization. This contrasts with end-to-end neural alternatives such as NeuFA’s shared attention framework (Li et al., 2022), Qwen3-ForcedAligner’s NAR timestamp-slot filling (Shi et al., 29 Jan 2026), or FALCON’s soft dynamic programming with differentiable decoding (Rousso et al., 24 Jun 2026).

6. Training objectives and confidence estimation

The SwanVoice appendix specifies several training losses for Swan Forced Aligner (Li et al., 29 May 2026). The first is a frame-level cross-entropy loss:

$\{(s_i, e_i)\}_{i=1}^M$5

where $\{(s_i, e_i)\}_{i=1}^M$6 is the set of valid acoustic frames, $\{(s_i, e_i)\}_{i=1}^M$7 is the target state, $\{(s_i, e_i)\}_{i=1}^M$8 is the predicted state distribution, and $\{(s_i, e_i)\}_{i=1}^M$9 is an optional confidence weight (Li et al., 29 May 2026).

The second is a CRF loss over structured paths:

$s_i$0

which encourages the gold path to score higher than all valid monotonic paths (Li et al., 29 May 2026).

The third is duration supervision:

$s_i$1

regularizing both word-state occupancy and blank-state occupancy. The paper states that the duration terms combine absolute-error and log-duration penalties (Li et al., 29 May 2026). The fourth is a monotonicity regularization term $s_i$2, which penalizes decreases in expected word index over time (Li et al., 29 May 2026).

The final objective is

$s_i$3

with all major loss terms combined with unit weight unless otherwise specified (Li et al., 29 May 2026).

This training setup combines local classification supervision, global structured-path discrimination, duration regularization, and monotonicity control. Relative to traditional GMM-HMM aligners, which rely on generative acoustic likelihoods and state-topology constraints (Rousso et al., 2024), Swan Forced Aligner incorporates richer discriminative supervision over the path structure itself. Relative to fully differentiable systems such as FALCON, which place soft dynamic programming inside the training loop (Rousso et al., 24 Jun 2026), Swan Forced Aligner remains structured but not fully differentiable end-to-end in the same sense.

After decoding, the model computes word confidence by aggregating emission probabilities over the frames assigned to that word; the paper specifies the confidence as the average word-state probability over the decoded word span (Li et al., 29 May 2026). This is relevant for filtering noisy alignments during corpus construction and resonates with a broader trend toward representing alignment uncertainty explicitly. For instance, ensemble-based methods have proposed confidence intervals for boundary placements using multiple acoustic models rather than single-point estimates (Kelley, 2 Jun 2025). Swan Forced Aligner does not report confidence intervals, but it does expose per-word confidence scores (Li et al., 29 May 2026).

7. Empirical evidence, comparison, and significance

The direct experimental evidence reported for Swan Forced Aligner appears in the SwanVoice appendix (Li et al., 29 May 2026). Training uses an 80K-hour Chinese-English alignment subset from internal resources spanning audiobooks, podcasts, conversational speech, meetings, and live streams (Li et al., 29 May 2026). Evaluation uses human-timestamped datasets:

• GTSinger-Speech-ZH
• LibriSpeech-Clean
• LibriSpeech-Others (Li et al., 29 May 2026)

The evaluation metric is Accumulated Averaging Shift (AAS):

$s_i$4

with lower values better (Li et al., 29 May 2026). This differs slightly from the AAS formulation reported in Qwen3-ForcedAligner, where the metric is defined over timestamp-slot predictions more generally (Shi et al., 29 Jan 2026), but both reflect average absolute temporal deviation.

The SwanVoice paper reports the following AAS values for Swan Forced Aligner (Li et al., 29 May 2026):

Evaluation set	Swan Forced Aligner AAS
GTSinger-Speech-ZH	45.19 ms
LibriSpeech-Clean	27.67 ms
LibriSpeech-Others	29.92 ms

The paper states that these results constitute the best open-source AAS on Chinese and LibriSpeech-Clean, that on LibriSpeech-Others the model is within 0.18 ms of Qwen3 Forced Aligner, and that it is about 10 ms behind the proprietary LattifAI Aligner (Li et al., 29 May 2026). The comparison baselines include Monotonic-Aligner, NeMo Forced Aligner, WhisperX, Qwen3 Forced Aligner, and LattifAI Aligner (Li et al., 29 May 2026).

These results place Swan Forced Aligner among recent neural forced alignment systems that compete directly with modern alternatives rather than only with classical HMM-based aligners. By contrast, broader comparative work has shown that traditional MFA-style systems still outperform adaptations of modern ASR systems such as WhisperX and MMS on boundary-accurate forced alignment in English benchmarks (Rousso et al., 2024). The SwanVoice results suggest that task-specific neural alignment can nevertheless achieve competitive timestamp accuracy when designed explicitly for alignment and evaluated on relevant multilingual data (Li et al., 29 May 2026).

Several boundaries should be kept clear. The paper does not provide a formal ablation isolating the contribution of Swan Forced Aligner to downstream SwanVoice synthesis quality (Li et al., 29 May 2026). It strongly implies that better pause annotations and corpus construction benefit dialogue prosody and expressiveness, but this remains an implication rather than a directly quantified causal result. The paper also does not present Swan Forced Aligner as a universal phoneme-level or language-agnostic forced aligner in the way some other systems aim to be (Shi et al., 29 Jan 2026, Rousso et al., 24 Jun 2026). Its demonstrated role is narrower and highly consequential: it is an enabling alignment module for pause-aware corpus construction in SwanData-Speech and thus for SwanVoice’s long-form expressive TTS pipeline (Li et al., 29 May 2026).

Taken together, Swan Forced Aligner represents a specialized synthesis-oriented development in forced alignment: a structured, transcript-conditioned, word-level aligner with explicit blank-state modeling, pause-aware supervision, and competitive AAS-based timestamp performance on Chinese and English evaluation sets (Li et al., 29 May 2026).