FluencyBank++: Annotated Stuttering Corpus

• FluencyBank++ is a comprehensive, multi-modal stuttering corpus with detailed clinical labels for primary disfluency types, secondary behaviors, and tension scores.
• It integrates frame-level annotations and rigorous expert consensus protocols to benchmark automatic dysfluency detection and segmentation systems.
• The dataset underpins state-of-the-art modeling with standardized metrics, enabling effective evaluation of stuttering severity and clinical insights.

FluencyBank++ is an enhanced, multi-modal, clinically annotated corpus of stuttered speech, providing granular, frame-level, and event-level labels for primary disfluency types, secondary behaviors, and associated tension. It extends the original FluencyBank database with state-of-the-art clinical annotation protocols, consensus gold-standard test sets, and audiovisual integration. FluencyBank++ constitutes a standardized benchmark for both modeling and evaluating systems for automatic stuttering detection, segmentation, and severity assessment, and serves as the principal dataset for recent advances in weakly supervised dysfluency segmentation and clinically grounded machine learning evaluation (Ghosh et al., 4 Aug 2025, Valente et al., 31 May 2025).

1. Origins, Scope, and Extensions of FluencyBank++

FluencyBank++ originates from the public-domain FluencyBank corpus, a collection of readings and interviews from adults who stutter (AWS). The enhanced dataset expands on the original by:

• Dataset Expansion: Expert speech-language pathologists (SLPs) re-annotated 30 reading samples (1 h 10 m, mean 2 m 11 s per sample) and 36 interview samples (6 h 15 m, mean 10 m 41 s per sample), encompassing all original data.
• Annotation Tiers: Three new tiers aligned with clinical practice:
  1. Primary Disfluency Types using the Lidcombe Behavioral Data Language (LBDL): Syllable Repetition (SR), Incomplete Syllable Repetition (ISR), Multi-Syllable Unit Repetition (MUR), Sound Prolongation (P), Block (B).
  2. Secondary Behaviors: Verbal (V), Facial Grimace (FG), Head Movement (HM), Extremity Movement (ME).
  3. Tension Scores: Physical tension rated on a 0–3 ordinal scale (no tension to severe tension).

All annotations are time-aligned in ELAN and span both audio and video modalities, allowing for sophisticated event localization and behavioral categorization (Valente et al., 31 May 2025). For segmentation studies, the FluencyBank++ protocol also involved extending 3-second original speech clips up to 5 seconds and annotating four types (interjection, repetition, block, prolongation) at frame-level (0.1 s granularity) for high-fidelity onset/offset boundaries (Ghosh et al., 4 Aug 2025).

2. Clinical Annotation Protocols and Validation

FluencyBank++ employs clinically rigorous, multi-stage annotation protocols:

• Annotator Expertise: Three SLPs (2–40 years AWS experience) received structured training based on standardized frameworks (LBDL for primary disfluency, SSI-4 for secondary behaviors, and Boey et al. for tension).

• Annotation Process: Events are marked in audio and video, with start/end boundaries determined by clinical rules (e.g., onset = first sign of repetition/prolongation; offset = return to fluent speech).

• Review and Consensus: Several rounds of majority voting and consensus-building ensure label reliability. In later phases (refined for segmentation tasks), frames with at least two votes across three SLPs finalize dysfluency boundaries.

• Inter-Annotator Agreement (IAA): Reliability is quantified across categories:

  • Temporal spans (overlapping events) show Krippendorff’s α=0.68.
  • Primary types: α≈0.5–0.8; secondary behaviors: α≈0.3–0.6 (annotator-level).
  • Tension labels show lower agreement (α=0.18).

A held-out “gold” test set was constructed from data showing highest annotator disagreement, followed by consensus meetings and cross-validated with a dedicated Disagreement tier (Valente et al., 31 May 2025, Ghosh et al., 4 Aug 2025).

3. Dataset Structure, Scope, and Label Taxonomy

The FluencyBank++ corpus comprises:

• Over 7 hours of re-annotated adult stuttering data: 1,654 reading events, 4,037 interview events.
• Each event labeled on three axes: a single primary disfluency class, zero or more secondary behaviors, and one tension score.
• For segmentation benchmarking: 3,017 five-second clips annotated at frame level for four dysfluency types. Table below summarizes segmentation clip distribution and durations.

Dysfluency Type	# Clips	Duration Range (s)
Interjection	1,130	0.12–1.88
Repetition	921	0.20–4.99
Block	530	0.23–4.20
Prolongation	436	0.41–3.95

The gold test set contains 732 events across types (SR, ISR, MUR, P, B), with detailed label distributions for each type and secondary behavior.

This taxonomy enables evaluation of both segment-level (boundary and type detection) and event-level (co-occurrence and tension scoring) tasks, facilitating fine-grained clinical model assessment.

4. Benchmarking Protocols and Model Evaluation

Standardized experimental practices on FluencyBank++ include:

• Segmentation Metrics: time-F1 (t-F1) and time-recall (t-recall) for overlap-based segment evaluation (IoU>0.5) and onset error (absolute offset between predicted and true onsets).
• Modeling Baselines:
  • Audio: wav2vec 2.0 backbone, two linear layers, classifier head, weighted binary cross-entropy.
  • Video: VIVIT with partially unfrozen layers, linear classifier.
  • Multi-modal: concatenation of audio and video feature embeddings, fully connected post-processing.
• Performance:
  • Annotator majority vote achieves macro-F1 of 0.76 (audio-based) and up to 0.83 for specific secondary behaviors (FG) (Valente et al., 31 May 2025).
  • Baseline models: Audio-only is optimal for primary events (e.g., Block: F1=0.68); multi-modal models achieve best overall F1 for detection of any disfluency event (F1=0.95).

For segmentation, StutterCut achieves the highest t-F1 (69.3% on FluencyBank++; 87.2% on synthetic VCTK-TTS) and the lowest onset error (0.3 s and 0.2 s, respectively), substantially outperforming YOLO-Stutter, Harvill et al., and weakly supervised Whisper-based classifiers (Ghosh et al., 4 Aug 2025).

5. Impact and Implications for Stuttering Research

FluencyBank++ constitutes the first open-access, multi-modal, granular clinical annotation resource that encompasses the full spectrum of stuttering phenomena—segmental type, behavioral context, and physiological tension.

Key implications:

• Model Development: FluencyBank++ enables fine-grained, clinically interpretable end-to-end models for stuttering and dysfluency detection, segmentation, and severity assessment.
• Benchmarking: The resource underpins rigorous, fair comparison of both conventional and neural modeling techniques, supporting frame/segment/event-level evaluation.
• Clinical Alignment: Integration of expert consensus and clinical decision rules makes FluencyBank++ a valid testbed for translational research and clinical deployment.
• Challenges and Open Problems: Persistent subjectivity (tension rating), cross-lingual confounders (accent vs. stuttering), and comorbid behaviors highlight the continued need for robust annotation and generalization strategies.

A plausible implication is that the multi-dimensional, consensus-grounded nature of FluencyBank++ will contribute to improved real-world reliability of automatic stuttering assessment and therapeutic progress-tracking (Valente et al., 31 May 2025, Ghosh et al., 4 Aug 2025).

6. Availability and Future Directions

All FluencyBank++ annotations, time-aligned ELAN files, aggregation scripts, and baseline code are to be released at https://github.com/mbzuai-nlp/CASA.git for public access. The dataset establishes a foundation for ongoing research on:

• Enhanced segmental and event-level stuttering detection (e.g., via weakly supervised or uncertainty-aware models)
• Multi-label and multi-modal fusion techniques for clinical tasks
• Development and evaluation of tension-scoring frameworks
• Cross-lingual and cross-population generalization studies
• Automatic progress monitoring and personalized therapy in clinical settings

These directions aim to address outstanding challenges in annotation subjectivity, clinical validity, and model robustness in diverse speech contexts (Valente et al., 31 May 2025, Ghosh et al., 4 Aug 2025).