Stuttering Classification and Segmentation with Attention-Based Multiple Instance Learning

Abstract: Stuttering detection and classification using deep learning methods has the potential to improve the process of stuttering severity assessment. Most stuttering classification datasets provide clip-level labels, making them unsuitable for fine-grained frame-level classification needed to determine the duration of individual stuttering dysfluencies. To overcome this challenge, we present a multiple instance neural network architecture based on fine-tuned wav2vec 2.0, WavLM and Whisper encoders. We apply instance- and embedding-based multiple instance learning approaches to train models on a clip-level dataset for both clip-level and frame-level stuttering classification tasks. Our results show a 23% improvement in frame-level F1 score and between 2% and 9% in clip-level F1 score, demonstrating the ability of our models to utilize clip-level data for frame-level segmentation.

• The paper presents an attention-based MIL framework that segments and classifies stuttering from weak clip-level labels using advanced speech encoders.
• It introduces both instance-based and embedding-based models, with the latter achieving superior precision and recall, including a 23% improvement in frame-level segmentation.
• The study demonstrates clinical relevance for granular stuttering detection and outlines future directions for dynamic context and multimodal integration.

Attention-Based Multiple Instance Learning for Stuttering Classification and Segmentation

Problem Formulation and Motivation

Stuttering, a complex speech fluency disorder, presents challenges for both automated detection and clinical assessment, especially in segmenting and classifying diverse dysfluency types at both clip and frame granularity. Contemporary datasets predominately offer clip-level labels, lacking the fine-grained annotations required by clinical instruments such as SSI-4 or SES. This paper formalizes stuttering detection as a weakly supervised multiple instance learning (MIL) task, leveraging clip-level labeled data to infer frame-level dysfluency events. The standard MIL assumption closely aligns with the requirement that a clip is labeled as dysfluent if any contained frame exhibits stuttering. Thus, frames represent instances and clips are bags of instances, enabling efficient utilization of available annotated data for granular segmentation analysis.

Model Architecture

Two variants of multiple-instance neural networks (MINNs) were constructed: an instance-based model utilizing max pooling and an embedding-based model with an attention pooling operator. Both models are scaffolded atop pretrained foundation encoders—wav2vec 2.0, WavLM, and Whisper—selected for comparable parameterization and embedding size. The encoder outputs are pooled across multiple layers using the HConv interface, then passed through a 4-layer bidirectional LSTM for temporal smoothing and subsequently, a multi-layer projector.

For the instance-based model, frame-level predictions are aggregated via a max pooling MIL function, providing interpretability and direct frame segmentation. For the embedding-based model, attention pooling produces adaptive bag embeddings, with attention weights facilitating interpretability and improved bag-level classification performance. Attention pooling is implemented via a two-layer fully connected mechanism with $\tanh$ and softmax activations.

Figure 1: Diagrammatic representation of the instance-based MINN model for stuttering classification.

Loss function design incorporates binary cross-entropy with per-label and annotator agreement-based balancing to address class imbalance and annotation uncertainty.

Experimental Results

Training was conducted on SEP-28k-E, encompassing 28k clips with six stuttering-related labels, and evaluated cross-dataset on FluencyBank and CASA frame-level annotations. Optimization procedures involved alternating encoder freezing/unfreezing, Adam optimizer, and systematic thresholding.

Strong numerical results were observed:

• Clip-level multi-label classification: Whisper and WavLM embedding-based models achieved F1 scores of 0.35–0.53 across individual dysfluency types, outperforming several baselines, especially for blocks, sound repetitions, word repetitions, and interjections.
• Single-label cross-dataset classification: Whisper + attn. pool achieved an F1 of 0.90 on FluencyBank, exhibiting a marked improvement over the previous SOTA (0.85).
• Frame-level segmentation: The Whisper + attn. pool model established a new SOTA on CASA gold standard data, with an F1 score of 0.70—a 23% absolute improvement over prior methods. Notably, the embedding-based models consistently outperformed instance-based models in both recall and precision.

Figure 2: Spectrogram and transcription of a stuttered speech sample illustrating input and model operation for frame-level segmentation.

Technical Analysis

The embedding-based MIL approach demonstrates superior performance for both clip-level and frame-level labeling, circumventing the limitations faced by instance-based methods (e.g., poor recall and contextual dependency). The architecture's reliance on foundation encoders significantly influences efficacy, with wav2vec 2.0 showing comparatively weaker performance, highlighting the importance of encoder selection and fine-tuning.

A critical innovation is the use of unnormalized attention weights for frame-level inference, mitigating duration-induced bias from softmax normalization. Attention plots reveal enhanced recall, vital for diagnostic tasks. However, consistent performance degradation is noted for long-lasting blocks, underscoring the restriction imposed by fixed context windows. There exists potential for further improvement via expanded temporal context or integration of frame-level pretraining.

Implications and Future Directions

The work presents clinically relevant advances by reconciling practical clip-level annotation with the necessity for precise frame-level dysfluency localization. Embedding-based MIL provides both performance and interpretability, supporting integration in clinical workflows for severity assessment and guiding downstream interventions. The embedding-based pooling mechanism, when combined with modern speech encoders, enables zero-shot frame-level inference directly from clip-level supervision.

Future directions entail:

• Evaluating and refining multi-label segmentation on frame-level annotated datasets.
• Architectures leveraging dynamic context windows for improving block detection.
• Incorporation of cross-modal features, e.g., physiological or visual cues.

The paradigm can generalize to other speech disorder detection domains, offering robust weakly supervised techniques for fine-grained behavior segmentation from coarse labels.

Conclusion

This paper introduces and rigorously evaluates weakly-supervised MIL-based architectures for stuttering detection and segmentation, demonstrating strong empirical results and practical clinical utility. Attention-based embedding pooling, applied here for the first time to multi-label stuttering classification, yields significant improvements over both instance-based and prior SOTA models for frame-level segmentation. The approach enables efficient reuse of existing clip-level annotated datasets for constructing clinically actionable frame-level classifiers, with implications for automated severity assessment and broader application in speech disorder analysis.