Unconstrained Dysfluency Modeling (UDM)

• UDM is a research framework that models dysfluent speech by jointly addressing transcription, alignment, and detection at both word and phonetic levels.
• The approach uses unconstrained forced alignment with architectures like WavLM and conformers, evolving into hierarchical models and scalable extensions with articulatory gestures.
• The framework employs both rule-based template matching and learned alignment techniques to detect phenomena such as repetition, prolongation, and irregular pauses in speech.

Searching arXiv for core UDM papers and follow-on work. Unconstrained Dysfluency Modeling (UDM) is a line of research that frames dysfluent speech analysis as a joint problem of dysfluency-aware transcription, alignment, and detection, rather than as isolated utterance-level classification or conventional ASR. In the formulation introduced by Lian et al., UDM targets time-accurate and silence-aware modeling at both the word level and phonetic level, with the goal of recovering what was actually spoken, how it diverged from the intended or reference sequence, and when the divergence occurred (Lian et al., 2023). Subsequent work has expanded this agenda into hierarchical alignment (Lian et al., 2024), scalable articulatory and subsequence-alignment frameworks (Lian et al., 2024, Lian et al., 2024), multimodal large-language-model decoding (Wagner et al., 2024, Altinok, 23 Jun 2025), temporal detection (Zhou et al., 2024), weakly supervised segmentation (Ghosh et al., 4 Aug 2025), and decoder-centric zero-shot transcription (Guo et al., 22 May 2025). Across these variants, the common theme is that dysfluency is treated as structured deviation in speech production rather than as a narrow label attached to an utterance.

1. Conceptual definition and scope

UDM was introduced to address a mismatch between real dysfluent speech and the assumptions of standard speech processing pipelines. The original formulation argues that dysfluent speech modeling requires outputs that are time-accurate, silence-aware, and available at both word-level and phonetic-level (Lian et al., 2023). This view departs from conventional ASR, which tends to normalize dysfluent speech into fluent lexical output, and from dysfluency classifiers that predict only whether a segment contains stuttering or another event type.

In the original UDM paper, dysfluent speech is treated as speech containing abnormalities such as repetition, prolongation, and irregular pauses, including cases in stuttering and aphasia (Lian et al., 2023). H-UDM later broadens the working definition to “any form of speech characterized by abnormal patterns such as repetition, replacement, and irregular pauses,” and defines speech disfluency modeling as detecting all types of disfluencies at both the word and phoneme levels while providing a timestamp for each type (Lian et al., 2024). This suggests a broader ontology than stuttering-only detection, though the realized label inventories in many systems remain fixed and relatively small.

The term “unconstrained” has a specific technical meaning in this literature. In the original UDM framework, it refers primarily to the fact that the alignment module can operate without text input at inference time, even though it is trained with weak text supervision, and that alignment is not forced to remain strictly monotonic with a reference transcript (Lian et al., 2023). A plausible implication is that UDM is “unconstrained” relative to conventional forced alignment and transcript-faithful ASR, not unconstrained in the sense of open-ended ontology or free-form event discovery. Many later systems inherit this partial unconstrainedness while retaining fixed dysfluency classes, simulated training data, or reference-conditioned decoding (Zhou et al., 2024, Guo et al., 22 May 2025).

2. Original UDM architecture

The original UDM framework consists of a Transcription Module and a Detection Module (Lian et al., 2023). Its core transcription component is the Unconstrained Forced Aligner (UFA), which takes speech as input and produces a phonetic alignment that is text-independent at inference time. UFA uses a WavLM encoder followed by a 4-layer conformer, with separate heads for phoneme classification and boundary prediction (Lian et al., 2023). The conformer settings are given explicitly: layer 1 has hidden size 1024, 4 heads, filter size 5, dropout 0.1; layers 2 and 3 have hidden size 1024, 8 heads, filter size 3, dropout 0.1; layer 4 has hidden size 1024, 4 heads, filter size 3, dropout 0.1 (Lian et al., 2023).

Training supervision for UFA is weakly derived from Montreal Forced Aligner (MFA) outputs. The phoneme classifier uses softmax cross-entropy, while the boundary predictor uses logistic regression, producing values between 0 and 1 for non-boundary versus boundary (Lian et al., 2023). The phoneme inventory comprises 39 CMU monophones plus 1 silence label, with lexical stress removed (Lian et al., 2023). Silence is therefore modeled as a first-class unit, which is central to UDM’s treatment of pauses and blocks.

Inference relies on boundary-aware dynamic alignment search, described as an extension of Viterbi decoding (Lian et al., 2023). With phoneme logits and boundary predictions, the decoder incorporates a bigram phoneme LLM and modulates transitions using boundary probabilities so that short phones are less likely to be absorbed into surrounding silence. The paper presents this as a solution to a characteristic dysfluency problem: standard monotonic or silence-dominant decoding can collapse repeated or weakly articulated phones into silence, erasing precisely the events of interest (Lian et al., 2023).

A second core object is 2D alignment, which represents the relationship between the reference phoneme sequence and the predicted forced-alignment phoneme sequence. UDM computes a similarity matrix between reference and produced phoneme embeddings and then derives alignment patterns through Viterbi search and DTW comparisons (Lian et al., 2023). Dysfluency detection is not learned end-to-end from dense labels; instead, categories are inferred by template matching over alignment geometry. At the phonetic level, UDM detects missing/deletion, replacement, repetition, insertion, and irregular pause by comparing unconstrained alignment with a monotonic DTW reference (Lian et al., 2023). At the word level, it detects missing, insertion, replacement, and repetition using analogous alignment logic.

The original UDM also includes a Text Refresher module, motivated by the observation that Whisper-large tends to output overly fluent text for dysfluent speech. It compares Whisper’s word transcription, converted to phonemes via the CMU dictionary, with UFA’s phonetic output using DTW; suspected insertion and deletion errors are then corrected to yield an “imperfect” word transcript that preserves dysfluency better than standard ASR (Lian et al., 2023). This introduces a recurrent UDM theme: lexical transcription must be counterbalanced by phonetic evidence to prevent normalization.

3. Hierarchical and scalable extensions

A major extension is Hierarchical UDM (H-UDM), which adds recursive segmentation and re-alignment on top of original UDM (Lian et al., 2024). H-UDM retains UFA, 2D alignment, and template-based detection, but introduces two changes. First, it adds an auxiliary CTC constraint during training to improve robustness of the otherwise non-monotonic alignment process (Lian et al., 2024). Second, it proposes the Unconstrained Recursive Forced Aligner (URFA): zero-order UFA processes the whole utterance, then a smoothed monotonic 2D alignment is used to infer word boundaries, after which the utterance is re-aligned recursively at the segment level (Lian et al., 2024). The recursive orders extend to first-, second-, and third-order URFA.

This hierarchical design is intended to reduce variance in whole-utterance alignment. H-UDM uses cosine similarity between unconstrained 2D alignment and DTW-based 2D alignment to merge sufficiently similar regions and derive a smoothed segmentation map (Lian et al., 2024). The smoothed monotonic map is used only for segmentation, while the original non-monotonic alignment remains the source for detection. This suggests a compromise between local structure discovery and preservation of dysfluency evidence.

Empirically, H-UDM reports consistent gains from both the CTC regularizer and recursion. On phonetic disfluency detection, F1 improves from 62.4 for UFA-VCTK to 67.2 with VCTK$^{++}$, CTC, and 3rd-order recursion; matching score (MS) improves from 55.2 to 62.8 (Lian et al., 2024). On word-level dysfluency detection, F1 improves from 64.0 for Whisper-Large to 73.1 with Text Refresher, VCTK$^{++}$, CTC, and 3rd-order recursion (Lian et al., 2024). The same paper reports word segmentation MS improving from 42.1 for WhisperX to 81.4 for 3rd-order H-UDM (Lian et al., 2024). These results matter because they show that the original unconstrained alignment framework can be made more stable without abandoning its non-monotonic character.

The other major extension is SSDM: Scalable Speech Dysfluency Modeling (Lian et al., 2024). SSDM explicitly targets the scalability limitations of UDM/H-UDM, arguing that 2D alignment fails to improve as training data increases (Lian et al., 2024). SSDM replaces earlier final representations with articulatory gestures as a scalable forced-alignment substrate, introduces a Connectionist Subsequence Aligner (CSA), builds the large simulated corpus Libri-Dys, and adds an LLM-based end-to-end output module (Lian et al., 2024).

CSA is one of the most important architectural departures in the UDM lineage. Rather than non-monotonic 2D alignment over speech and reference, SSDM assumes that each reference token aligns to a consecutive subsequence of the dysfluent phonetic sequence, which is sufficient to represent insertion, deletion, repetition, block, and replacement (Lian et al., 2024). This is framed as a differentiable approximation to local subsequence alignment inspired by longest common subsequence. The paper’s claim is that this preserves dysfluency semantics while scaling better than prior 2D non-monotonic alignment.

SSDM 2.0 further extends this direction by introducing Neural Articulatory Flow (NAF), a Full-stack Connectionist Subsequence Aligner (FCSA), a mispronunciation prompt pipeline, a consistency-learning module, and the co-dysfluency corpus Libri-Co-Dys (Lian et al., 2024). NAF converts articulatory trajectories into sparse semi-implicit gestural representations, then injects intelligibility via conditional flow matching from WavLM features (Lian et al., 2024). FCSA expands CSA into a six-stack alignment system covering fluent monotonic alignment, copy-style repetition/insertion/block, skip-style omissions, non-monotonic replacements, and passive insertion/removal states (Lian et al., 2024). Relative to UDM, this marks a shift from hand-crafted alignment templates toward a differentiable subsequence-alignment model with broader dysfluency coverage.

4. Dysfluency representation and modeling strategies

UDM and its descendants do not share a single representational formalism. Instead, several strategies coexist.

The original UDM/H-UDM strategy is alignment-derived event inference. Dysfluencies are implicit in the mismatch geometry between reference and produced phoneme sequences, and categories are obtained by rule-based interpretation of alignment patterns (Lian et al., 2023, Lian et al., 2024). This makes the method weakly supervised and relatively interpretable, but it also constrains it to predefined templates.

A second strategy is detector-style region prediction. Stutter-Solver formulates dysfluency modeling as a 1D object-detection problem over a soft speech-text alignment matrix derived from VITS (Zhou et al., 2024). Each utterance is partitioned into $S=64$ temporal cells, and each cell predicts dysfluency confidence, start/end boundaries, and class scores for five categories: repetition, block, missing, replacement, and prolongation (Zhou et al., 2024). This is detection-oriented UDM: more open than hand-built templates, but still limited to a fixed label inventory and dependent on synthetic corpora.

A third strategy is token-based transcription. Time and Tokens tokenizes dysfluencies and treats detection as an ASR-style sequence problem (Zhou et al., 2024). Word-level tokens include [REP], [DEL], [INS], and [PAU]; phoneme-level tokens include [REP], [DEL], [SUB], and [PRO] (Zhou et al., 2024). This formulation turns dysfluency into part of the symbolic output sequence rather than a separate event layer. Smooth Operators extends this idea with a multimodal LLM decoder that generates fully annotated dysfluency-rich transcripts with timestamps from imperfect hints and audio (Altinok, 23 Jun 2025). The model, Smooth-LLaMa, takes 80-dimensional log Mel spectrograms, text hints such as clean transcripts, MFA-aligned words/phonemes, or wav2vec-phoneme outputs, and generates transcripts containing explicit dysfluency tokens and timing (Altinok, 23 Jun 2025). This suggests a path from UDM-style alignment toward conditional transcript reconstruction.

A fourth strategy is multimodal label generation without open transcript reconstruction. LLMs for Dysfluency Detection in Stuttered Speech casts clip-level multi-label detection as autoregressive generation of label tokens conditioned on ASR hypotheses and wav2vec 2.0 embeddings (Wagner et al., 2024). The model consumes a mean-pooled acoustic prefix and lexical input, then generates a short label sequence over a fixed set of event types such as block, prolongation, sound repetition, word repetition, interjection, and, for KSoF, modified speech (Wagner et al., 2024). The prediction space is textual, but the semantics remain bounded multi-label classification over 3-second clips.

A fifth strategy is segmentation with weak supervision. StutterCut formulates dysfluency segmentation as graph partitioning over overlapping speech windows using normalized cut and a weak-label pseudo-oracle (Ghosh et al., 4 Aug 2025). It produces binary dysfluent/non-dysfluent segmentation from utterance-level labels, without requiring transcript alignment. This suggests a practical UDM interpretation centered on localization without dense supervision.

Taken together, these approaches show that UDM is better understood as a research program than a single algorithm. The common axes are: reference-aware vs transcript-free, rule-based vs learned alignment, event detection vs tokenized transcription, and strong-label vs weak-label supervision.

5. Data resources and simulation pipelines

UDM research has depended heavily on simulated corpora because large real dysfluency datasets with time-accurate annotation are scarce. The original UDM introduced VCTK$^{++}$, a simulated dysfluent dataset derived from the 44-hour multi-speaker VCTK corpus (Lian et al., 2023). It injects repetitions, prolongations, and blocks into waveform segments using MFA alignments and WSOLA-based or silence-based editing (Lian et al., 2023). H-UDM reuses VCTK$^{++}$ for training and shows that it materially improves downstream phonetic transcription and detection (Lian et al., 2024).

SSDM introduces Libri-Dys, described as the first large-scale simulated dysfluency corpus, built by applying text-space dysfluency simulation to the full LibriTTS corpus and synthesizing audio with StyleTTS2 (Lian et al., 2024). Libri-Dys has 3983.44 hours and models repetition, missing, block, replacement, and prolongation (Lian et al., 2024). The paper reports overall MOS 4.15 ± 0.93, much higher than the MOS reported for VCTK++ (2.14 ± 0.64) (Lian et al., 2024). SSDM 2.0 adds Libri-Co-Dys, described as the largest co-dysfluency corpus, with 6023.24 hours and an average of 2.51 dysfluencies per utterance (Lian et al., 2024).

Stutter-Solver contributes three synthetic corpora geared toward time-based detection: VCTK-Pro, VCTK-Art, and AISHELL3-Pro (Zhou et al., 2024). VCTK-Pro is built with VITS-based text/phonological dysfluency injection and totals 1125.1 hours, including a dedicated co-dysfluency subset (Zhou et al., 2024). VCTK-Art uses articulatory editing with Articulatory Encodec and totals 392.19 hours (Zhou et al., 2024). AISHELL3-Pro is a Mandarin counterpart totaling 511.37 hours (Zhou et al., 2024). These corpora matter because they shift UDM-style detection toward multilingual and time-localized modeling.

Time and Tokens introduces VCTK-Token, a VITS-based simulated dataset with pairs of dysfluent speech and annotated text, totaling 584.05 hours across repetition, missing/deletion, block/pause, replace/substitution, prolongation, and insertion (Zhou et al., 2024). LLM-Dys later extends simulation further with LLM-generated dysfluent text and controllable TTS synthesis, covering 11 dysfluency categories across word and phoneme levels and totaling 12,790 hours (Zhang et al., 28 May 2025). This suggests a trend from hand-built rule simulation toward context-aware LLM-based generation.

For real speech, commonly used evaluation resources include Buckeye, SEP-28k-Extended, FluencyBank, KSoF, UCLASS, Aphasia Speech, and nfvPPA (Lian et al., 2023, Wagner et al., 2024, Zhou et al., 2024, Lian et al., 2024). Their annotation styles vary sharply: clip-level multi-label event tagging in SEP-28k and FluencyBank (Wagner et al., 2024), frame- or span-level labels in more recent segmentation work (Ghosh et al., 4 Aug 2025), and phoneme-level manual annotation in nfvPPA (Guo et al., 22 May 2025). This heterogeneity is a major reason UDM remains methodologically fragmented.

6. Empirical findings and recurrent limitations

Across the literature, several empirical patterns recur. First, multimodal conditioning helps, but often only within bounded label spaces. The LLM-based clip classifier in (Wagner et al., 2024) shows that combining acoustic and lexical information improves multi-label stuttering detection, with the strongest gains on blocks. On SEP, block F1 increases from 0.32 in the baseline to roughly 0.58 in the best LLM condition; on FluencyBank, block F1 rises from 0.36 to 0.58 (Wagner et al., 2024). However, word repetition remains hard, and German KSoF results are mixed (Wagner et al., 2024). This suggests that multimodal UDM is promising, but ASR normalization and English-centric LLM backbones remain limitations.

Second, alignment-aware or detector-style models outperform template-only baselines on synthetic and some real corpora, but the synthetic-to-real gap remains large. Stutter-Solver improves over H-UDM across synthetic datasets and on public benchmarks such as UCLASS and SEP-28K, with Time F1 on SEP-28K increasing from 0.699 for H-UDM to 0.813 with VCTK-Pro training (Zhou et al., 2024). Yet real aphasia-speech accuracy remains much lower than performance on simulated corpora, which the paper explicitly identifies as the main caveat (Zhou et al., 2024).

Third, audio-native modeling is often necessary when transcripts are unreliable. The paper on automatic disfluency detection from untranscribed speech shows that transcript-dependent approaches degrade heavily with ASR errors, while acoustic-only models such as WavLM outperform transcript-based systems on several frame-level disfluency classes (Romana et al., 2023). This supports a core UDM intuition: transcription should be treated as an auxiliary source or target, not the only substrate.

Fourth, weakly supervised localization is feasible on real speech. StutterCut shows that a graph-based segmentation framework guided by a weak-label pseudo-oracle can outperform direct segmentation baselines on both real and synthetic data, reaching overall 69.3 ± 4.9 t-F1 on FluencyBank++ and 87.2 ± 3.1 on VCTK-TTS (Ghosh et al., 4 Aug 2025). The ablations further show that uncertainty-aware weighting is especially important on real speech (Ghosh et al., 4 Aug 2025). This suggests that UDM need not require dense frame labels to achieve useful temporal localization.

Fifth, decoder-based zero-shot methods can be highly competitive in constrained settings. Dysfluent-WFST builds a dysfluency-aware WFST on top of pretrained phoneme encoders and achieves strong phonetic transcription improvements without dysfluency-specific training. On simulated insertion data, WavLMCTC-WFST reduces PER from 19.80% for greedy decoding to 2.80%; on real nfvPPA, it reduces PER from 61.49% to 44.75% (Guo et al., 22 May 2025). Repetition detection reaches 100 accuracy at the count level in simulated conditions (Guo et al., 22 May 2025). This suggests that some UDM subproblems may be addressed more effectively through explicit decoding than through heavier learned architectures.

The literature also exposes stable limitations. The most common are:

• Fixed label ontologies: many methods still operate over five or six dysfluency categories, even when motivated as unconstrained (Wagner et al., 2024, Zhou et al., 2024, Guo et al., 22 May 2025).
• Dependence on simulated corpora: strong performance often derives from synthetic data, while real pathological or spontaneous speech remains much harder (Zhou et al., 2024, Lian et al., 2024).
• Reference dependence: several strong systems still require a reference text or intended sequence, especially UDM, H-UDM, SSDM, SSDM 2.0, and WFST-based decoders (Lian et al., 2023, Lian et al., 2024, Lian et al., 2024, Lian et al., 2024, Guo et al., 22 May 2025).
• Difficulty with deletion, insertion, and word repetition: these remain harder than repetition of acoustically salient units or blocks in some corpora (Wagner et al., 2024, Lian et al., 2024, Guo et al., 22 May 2025).
• Limited multilingual evidence: multilingual support exists in isolated cases, especially English/German and English/Mandarin, but consistent cross-lingual UDM remains unsolved (Wagner et al., 2024, Zhou et al., 2024).
• No unified benchmark: corpora differ in segmentation, annotation granularity, taxonomy, and domain, making direct comparison difficult (Zhou et al., 2024).

A plausible implication is that “unconstrained” currently describes a direction of travel rather than a solved modeling regime.

7. Position within related research areas

UDM intersects with several neighboring domains. It overlaps with forced alignment when recovering time-accurate phonetic structure from speech and reference text, but differs in allowing strong mismatches and dysfluency events (Kouzelis et al., 2023, Lian et al., 2023). It overlaps with mispronunciation detection and speech rich transcription, especially in SSDM 2.0, where non-fluencies are treated as part of a broader discrepancy between intended lexical content and produced acoustic realization (Lian et al., 2024). It overlaps with multimodal spoken language understanding, particularly where LLMs consume acoustic prefixes and textual hints (Wagner et al., 2024, Altinok, 23 Jun 2025). It also relates to temporal action detection through YOLO-inspired region prediction for dysfluency spans (Zhou et al., 2024).

The field also contains a tension between text-dependent and text-independent approaches. Original UDM claims unconstrained inference because UFA itself does not require text input (Lian et al., 2023), but detection still relies on comparison with reference structure. WFST-based decoders and SSDM-style systems are more clearly reference-conditioned (Guo et al., 22 May 2025, Lian et al., 2024). By contrast, acoustic-only and weakly supervised segmentation approaches move toward transcript-free localization (Romana et al., 2023, Ghosh et al., 4 Aug 2025). This suggests that future UDM may bifurcate into two tracks: reference-aware pronunciation/deviation modeling and transcript-free event discovery or segmentation.

Another recurrent tension is between interpretable explicit structure and end-to-end flexibility. UDM, H-UDM, and Dysfluent-WFST rely on explicit alignment geometry or graph decoding (Lian et al., 2023, Lian et al., 2024, Guo et al., 22 May 2025). SSDM 2.0 introduces stronger learned components but still grounds them in alignments and prompts (Lian et al., 2024). LLM-based systems offer flexible multimodal generation but often collapse back into closed-set labeling or simulated settings (Wagner et al., 2024, Altinok, 23 Jun 2025). This suggests that clinical and deployment-oriented UDM may continue to favor alignment-centric interpretability, while broader speech-rich transcription may favor larger generative models.

In summary, UDM denotes a family of methods that seek to model dysfluent speech as structured, time-localized deviation in production. The original framework established unconstrained phonetic alignment and template-based detection (Lian et al., 2023). H-UDM added recursive hierarchy and monotonicity-assisted training (Lian et al., 2024). SSDM and SSDM 2.0 pursued scalability through articulatory representations, subsequence alignment, and rich transcription (Lian et al., 2024, Lian et al., 2024). Parallel lines explored multimodal label generation (Wagner et al., 2024), tokenized dysfluency transcription (Zhou et al., 2024, Altinok, 23 Jun 2025), object-style detection (Zhou et al., 2024), weakly supervised segmentation (Ghosh et al., 4 Aug 2025), and zero-shot WFST decoding (Guo et al., 22 May 2025). The field remains constrained by synthetic data, heterogeneous benchmarks, and fixed taxonomies, but it has established a durable principle: dysfluency should be modeled as part of speech structure itself, not as an afterthought appended to fluent transcription.