StutterFormer: End-to-End Speech Models

• StutterFormer is a term for two end-to-end models that address stuttering with distinct designs for detection and correction.
• The detection-oriented model uses a Conformer-BiLSTM framework with multi-task learning to classify five stuttering phenomena, achieving notable F1 score improvements.
• The correction-oriented model employs a dual-stream Transformer to directly convert stuttered speech into fluent speech while jointly generating a corrected transcript, reducing WER and improving BERTScore.

Searching arXiv for the specified StutterFormer papers and closely related entries to ground the article in current arXiv records. StutterFormer is a name applied in recent arXiv literature to two distinct end-to-end architectures for stuttering-related speech processing. In "An End-To-End Stuttering Detection Method Based On Conformer And BILSTM" (Liu et al., 2024), StutterFormer denotes a Conformer-BiLSTM system for stuttering event detection, optimized through multi-task learning over five disfluency labels. In "StutterZero and StutterFormer: End-to-End Speech Conversion for Stuttering Transcription and Correction" (Xu, 21 Oct 2025), StutterFormer denotes a dual-stream Transformer that directly converts stuttered speech into fluent speech while jointly predicting a corrected transcript. The shared label reflects a common emphasis on end-to-end modeling and multitask supervision, but the two systems address different targets, operate on different output spaces, and embody different architectural commitments.

1. Terminological scope and problem setting

In current arXiv usage, "StutterFormer" does not identify a single canonical model; it names two architectures developed for different tasks within stuttering research. One is a detector of stuttering phenomena, and the other is a waveform-to-waveform correction-and-transcription model (Liu et al., 2024, Xu, 21 Oct 2025).

Usage of "StutterFormer"	Core architecture	Primary objective
Detection-oriented StutterFormer	Conformer + BiLSTM + task-specific classification heads	Stuttering event detection
Correction-oriented StutterFormer	Dual-stream Transformer with shared acoustic-linguistic representations	Direct stutter correction and transcription

The detection-oriented model is framed around clinical assessment support. Its stated purpose is to help speech-language pathologists assess both the type and severity of stuttering by detecting five stuttering phenomena: word/phrase repetition, block, prolongation, sound repetition, and interjection. The correction-oriented model is framed around direct speech conversion. It is described as the first Transformer-based architecture for direct stutter correction and transcription, designed to remove repetitions, blocks, and prolongations, reconstruct a fluent spectrogram, and simultaneously output a corrected transcript.

This bifurcated usage is important for interpretation. A reference to StutterFormer in the context of SLT 2024 challenge results concerns event detection on the AS-70 dataset, whereas a reference to StutterFormer in the context of FluencyBank, WER, and BERTScore concerns speech conversion and transcription. This suggests that the term functions more as a local project name than as a stable architectural taxonomy.

2. Detection-oriented StutterFormer: architecture and representation

The StutterFormer described in (Liu et al., 2024) follows the topology: input $\rightarrow$ 80-dim filter-bank features with SpecAugment $\rightarrow$ convolutional subsampling $\rightarrow$ $N$ Conformer blocks $\rightarrow$ Bi-directional LSTM layers $\rightarrow$ task-specific classification heads. The input representation consists of 80-dim Mel-filterbank features with frame size $25 \,\mathrm{ms}$ and hop $10 \,\mathrm{ms}$, followed by per-utterance mean-variance normalization and SpecAugment on time and frequency bands. The model uses no voice activity detection or manual segmentation.

Its convolutional subsampling stage comprises two 2D-convolution layers with stride $2$, reducing time resolution by factor $4$, followed by projection to model dimension $\rightarrow$0. Each Conformer block contains a Feed-Forward Module with dimension expansion factor $\rightarrow$1 and dropout $\rightarrow$2, a Multi-Head Self-Attention module with $\rightarrow$3 heads and model dimension $\rightarrow$4, a convolution module with $\rightarrow$5 projection, depth-wise convolution with kernel $\rightarrow$6, batch norm, Swish, and pointwise convolution, and a second Feed-Forward Module identical to the first. Each sub-layer is followed by LayerNorm, dropout, and residual connection. In the reported experiments, $\rightarrow$7 and $\rightarrow$8.

The recurrent backend consists of two stacked BiLSTM layers. Each direction has hidden size $\rightarrow$9; the concatenated output size is $\rightarrow$0, then linearly projected down to $\rightarrow$1. The stated rationale is to capture long-range bidirectional context beyond the Conformer’s local/global modeling.

The model is formulated as a multi-task classifier over five labels: word/phrase repetition ("[]"), block ("/b"), prolongation ("/p"), sound repetition ("/r"), and interjection ("/i"). The shared trunk is the Conformer plus BiLSTM stack, while the task-specific component is a final linear layer producing $\rightarrow$2 logits per task for positive versus negative classification. The paper’s rationale for multi-task learning is that the five stuttering phenomena are related as disfluencies while exhibiting different prevalence and acoustic signatures; under data scarcity and class imbalance, the shared inductive bias is expected to improve generalization for low-resource sub-tasks.

3. Detection objective, optimization, and empirical findings

For the detection-oriented model, the per-sample, per-task objective is binary cross-entropy with logits. For task $\rightarrow$3, raw logit $\rightarrow$4, ground-truth $\rightarrow$5, and $\rightarrow$6, the loss is

$\rightarrow$7

With $\rightarrow$8 samples and $\rightarrow$9 tasks, the total loss is

$N$0

The authors also explored focal loss and class-weighted BCE, but reported that unweighted BCEWithLogitsLoss was sufficient under their multi-task learning scheme. Regularization is provided by dropout within the Conformer and BiLSTM, with $N$1 (Liu et al., 2024).

Training uses the WeNet library, Adam with learning rate $N$2, betas $N$3, and $N$4. The learning-rate schedule is constant with early stopping of patience $N$5 epochs on dev loss. Transfer learning initializes the Conformer from an AISHELL-1 ASR pre-trained model with $N$6 layers. The validation split follows the SLT2024 challenge protocol at speaker level: $N$7 train, $N$8 dev, and $N$9 test.

Evaluation uses Micro-F1 per task, averaged into F1-final. On the AS-70 test set, the baseline attained approximately $\rightarrow$0 mean F1, the reported "OURS" system attained approximately $\rightarrow$1, and the reported "OURS*" attained approximately $\rightarrow$2. At the task level, the baseline scores were $\rightarrow$3 on /p, $\rightarrow$4 on /b, $\rightarrow$5 on /r, $\rightarrow$6 on /i, and $\rightarrow$7 on [], while "OURS*" yielded $\rightarrow$8, $\rightarrow$9, $\rightarrow$0, $\rightarrow$1, and $\rightarrow$2, respectively. The abstract reports a $\rightarrow$3 mean F1 improvement over the baseline in the SLT 2024 Stuttering Speech Challenge and first place, and a $\rightarrow$4 improvement under the fuller exploitation strategy.

The ablation studies are central to the interpretation of this model. ASR pre-training improved mean F1 by $\rightarrow$5 relative to random initialization for a $\rightarrow$6-layer Conformer. Varying the number of Conformer layers from $\rightarrow$7 showed best performance at $\rightarrow$8, with $\rightarrow$9 layers beginning to overfit. Replacing a unidirectional LSTM with a BiLSTM yielded a reported $25 \,\mathrm{ms}$0--$25 \,\mathrm{ms}$1 F1 gain. Depth sensitivity was task-dependent: the optimal depth was $25 \,\mathrm{ms}$2 BiLSTM layers for /p and /i, $25 \,\mathrm{ms}$3 for /b and /r, and $25 \,\mathrm{ms}$4 for []. Multi-task learning versus separate single-task models produced mixed effects, and a $25 \,\mathrm{ms}$5-task configuration combining /p, /i, and [] outperformed $25 \,\mathrm{ms}$6-task MTL on those tasks by $25 \,\mathrm{ms}$7--$25 \,\mathrm{ms}$8 F1. The final ensemble strategy used the $25 \,\mathrm{ms}$9-task MTL model for /p and [], the $10 \,\mathrm{ms}$0-task MTL model for /i, and single-task models for /b and /r. Statistical significance was not explicitly reported, although the paper notes consistent gains across five tasks.

4. Correction-oriented StutterFormer: dual-stream Transformer design

The StutterFormer introduced in (Xu, 21 Oct 2025) is a waveform-to-waveform system for direct stutter correction with joint transcription. Raw speech at $10 \,\mathrm{ms}$1 is converted to an $10 \,\mathrm{ms}$2-channel log-Mel spectrogram $10 \,\mathrm{ms}$3 using STFT with $10 \,\mathrm{ms}$4 window length and $10 \,\mathrm{ms}$5 frame shift, followed by Mel-filterbank projection. A linear layer maps each $10 \,\mathrm{ms}$6-dimensional frame to $10 \,\mathrm{ms}$7, and sinusoidal positional encodings are added:

$10 \,\mathrm{ms}$8

The encoder stack contains $10 \,\mathrm{ms}$9 identical Transformer layers. Each layer uses multi-head self-attention with $2$0 heads and per-head dimension $2$1, followed by Add & LayerNorm and a position-wise feed-forward network $2$2. The encoder output is a context tensor $2$3.

A defining feature is the dual-stream decoder arrangement. The acoustic stream reconstructs a fluent spectrogram. Its decoder includes a prenet with two fully-connected layers that compress the previous predicted Mel frame $2$4 to a $2$5-dimensional vector, masked self-attention over past frames, cross-attention to the encoder context $2$6, an FFN with residual connections and layer normalization, and a post-net comprising a $2$7-layer 1D convolutional stack to refine $2$8. Griffin-Lim reconstructs the time-domain waveform from $2$9. The linguistic stream predicts a corrected transcript from previous ground-truth or predicted tokens, using masked self-attention, cross-attention to $4$0, FFN, residuals, and a final linear-plus-softmax layer over the BPE or grapheme vocabulary.

The attention mechanism is specified as

$4$1

with multi-head attention concatenating $4$2 heads using learned projection matrices. The combined training objective couples speech reconstruction and transcription. If $4$3 is the ground-truth Mel spectrogram and $4$4 the predicted one, then

$4$5

with $4$6 in practice, and the ASR component is a token-level cross-entropy loss. The overall objective is

$4$7

with $4$8--$4$9 to emphasize audio quality while still forcing correct transcripts.

5. Training regime and quantitative evaluation for speech conversion

The training corpus for the correction-oriented model is synthesized from SEP-28K and LibriStutter, yielding approximately $\rightarrow$00 hours of paired stuttered-to-fluent data. SEP-28K is cleaned for "Music" and "NoSpeech," resampled to balance stutter types, transcribed via a fine-tuned Whisper-Small ASR, and re-synthesized to fluent audio via MeloTTS. Data are split $\rightarrow$01 into train/validation/test, with $\rightarrow$02-fold cross-validation and disjoint speakers. The batch size is $\rightarrow$03 spectrogram sequences per GPU. Optimization uses Adam with learning rate $\rightarrow$04, $\rightarrow$05, weight decay $\rightarrow$06, and $\rightarrow$07, together with Cosine Annealing Warm Restarts, initial period $\rightarrow$08 epochs and multiplicative factor $\rightarrow$09. Training runs for $\rightarrow$10 epochs, using mixed precision and gradient checkpointing to fit in $\rightarrow$11 GB VRAM. Regularization includes weight decay, layer normalization in every sublayer, dropout $\rightarrow$12 after attention and FFN, and data augmentation implicit in TTS variation (Xu, 21 Oct 2025).

Evaluation is performed on $\rightarrow$13 manually downloaded stuttered clips from FluencyBank Voices-AWS, with unseen speakers. For fairness, an untouched Whisper-Small ASR transcribes each model’s output. The reported quantitative results are:

Model	WER (mean $\rightarrow$14 sd)	BERTScore Precision
Whisper-Medium (baseline)	$\rightarrow$15	$\rightarrow$16
StutterZero	$\rightarrow$17	$\rightarrow$18
StutterFormer	$\rightarrow$19	$\rightarrow$20

The abstract states that StutterFormer achieved a $\rightarrow$21 decrease in WER and a $\rightarrow$22 improvement in BERTScore relative to Whisper-Medium. The detailed description also reports that, relative to Whisper-Medium, StutterFormer reduces WER by approximately $\rightarrow$23 on combined validation and improves semantic similarity by approximately $\rightarrow$24, and that over StutterZero on FluencyBank, WER drops another approximately $\rightarrow$25 while BERTScore rises by $\rightarrow$26. Wilcoxon tests with $\rightarrow$27 are reported to confirm statistical significance at $\rightarrow$28.

The paper also attributes a specific role to joint supervision. An ablation is reported in which removing the transcript decoder raises WER from $\rightarrow$29, supporting the claim that linguistic supervision aids acoustic reconstruction.

6. Comparative interpretation, limitations, and research directions

The two StutterFormer systems are linked by a common design logic: both are end-to-end and both use multitask structure to exploit related supervision. In the detection-oriented model, the shared trunk is the Conformer plus BiLSTM stack with five task-specific binary heads; in the correction-oriented model, the shared representation is the encoder context tensor consumed by parallel acoustic and linguistic decoders (Liu et al., 2024, Xu, 21 Oct 2025).

Their differences are nonetheless fundamental. The detection-oriented StutterFormer is a stuttering event detector operating on utterance-level acoustic features without voice activity detection or manual segmentation. Its outputs are logits for five stuttering phenomena, and its central metric is mean Micro-F1 across tasks. The correction-oriented StutterFormer is a waveform-to-waveform model that reconstructs fluent speech and emits text; its central metrics are WER and BERTScore Precision. One is designed to support type and severity assessment, while the other targets direct stutter correction and transcription.

Several limitations are explicitly identified. For the detection model, data imbalance remains challenging, especially for rare sub-types such as blocks, and the lack of explicit phonetic-level or language-model integration may limit phoneme-level precision. For the correction model, the Transformer encoder-decoder architecture incurs $\rightarrow$30 attention costs per layer relative to $\rightarrow$31 recurrent steps, although the paper argues that GPU parallelism benefits training and inference. The correction paper also notes that parameter count grows modestly, approximately $\rightarrow$32--$\rightarrow$33 over StutterZero, in exchange for greater accuracy.

The proposed future directions are likewise distinct. For the detection-oriented system, the paper identifies deeper theoretical analysis of task correlations, fusion with semantic or ASR outputs such as wav2vec 2.0 phoneme probabilities, and cross-corpus generalization to English or other stuttering corpora. For the correction-oriented system, practical applications are enumerated in speech therapy, accessibility, inclusive human-computer interaction, and mobile or embedded deployment through quantization and pruning. A plausible implication is that "StutterFormer" is best understood not as a fixed architecture but as a research line centered on task-coupled neural processing of stuttered speech, spanning both disfluency recognition and direct fluency restoration.