Augmenting Dysarthric Speech Severity Assessment with MOS Supervision

Abstract: Dysarthria is a speech disorder marked by reduced intelligibility and communicative effectiveness. Automatic utterance-level assessment of dysarthric speech can support scalable speech monitoring and therapy-related analysis. Yet training such systems is bottlenecked by the scarcity of clinically annotated dysarthric speech. This work proposes to augment dysarthric speech assessment using data from speech synthesis evaluations, specifically human-annotated utterances with Mean Opinion Score (MOS) labels from the QualiSpeech corpus. Experiments show that fine-tuning on speech synthesis assessment data consistently improves performance on both intelligibility and naturalness prediction, while joint training yields gains primarily on naturalness. These results suggest that synthesis artifacts and dysarthric speech share perceptual commonalities, and speech synthesis evaluation corpora offer a practical augmentation source that reduces reliance on scarce clinical annotations.

• The paper demonstrates that sequential fine-tuning with MOS supervision significantly reduces MSE and improves LCC in dysarthric speech assessments across SSL models.
• The methodology integrates TTS synthesis evaluation data with clinical annotations, employing joint training and fine-tuning paradigms to align perceptual scales.
• Experimental results reveal that larger SSL models benefit more from the augmentation, especially enhancing naturalness prediction compared to intelligibility.

Augmenting Dysarthric Speech Severity Assessment with MOS Supervision

Problem Formulation and Corpus Characterization

Dysarthria, a neuro-motor speech disorder, manifests as impaired intelligibility and naturalness, significantly complicating effective communication. Conventional utterance-level assessment protocols rely on expert clinical annotation, an approach that is time-consuming and demands highly specialized personnel—limiting scalability. The Speech Accessibility Project (SAP) delivers a large, open-domain corpus consisting of over 190,000 utterances from diverse pathologies, annotated granularly across multiple perceptual dimensions, with a particular focus on intelligibility and naturalness. Nevertheless, the distribution is uneven: intelligibility annotations are heavily concentrated at minimal impairment levels, while naturalness exhibits greater class balance, suggesting that variability in perceived speech quality dominates among dysarthric speakers.

Figure 1: Distribution of SAP severity levels for intelligibility.

Recent efforts have utilized self-supervised learning (SSL) encoders (e.g., wav2vec 2.0, HuBERT) and generative/data augmentation strategies. However, prior augmentation approaches lack perceptual validation, as synthetic data are not accompanied by clinically relevant severity labels. This work addresses this gap by leveraging MOS-labeled, human-scored utterances from the QualiSpeech corpus, sourced from TTS synthesis evaluations, as a perceptual supervision signal for dysarthric speech assessment.

Model Architecture and Training Paradigms

The paper proposes two transfer learning strategies incorporating MOS supervision into SSL-based dysarthria assessment models:

• Joint Training (JT): QualiSpeech and SAP utterances are mixed at a 1:1 ratio and jointly optimized using a shared encoder/regression head. QualiSpeech MOS and SAP severity scores are linearly mapped to a unified scale.
• Fine-Tuning (FT): Models are first pretrained on QualiSpeech MOS prediction, then sequentially fine-tuned on SAP for dysarthric severity prediction. This avoids direct optimization interference from misaligned labels and domains.

  Figure 2: Illustration of the joint training and fine-tuning paradigms integrating QualiSpeech MOS supervision.

The backbone utilizes SSL encoders (wav2vec 2.0 Base/Large/+, HuBERT Base/Large), extracting frame-level representations aggregated via mean pooling into utterance-level embeddings, fed to a simple FNN regression head. Evaluation employs MSE, LCC, and SRCC, capturing both absolute accuracy and relative perceptual ranking alignment.

Experimental Results and Quantitative Analysis

Extensive experiments reveal several pronounced findings:

• Fine-tuning with QualiSpeech augmentation consistently produces substantial improvements in SAP assessment accuracy. For intelligibility prediction, FT yields a maximum 43.7% MSE reduction for wav2vec 2.0 Large+ over the in-domain baseline, and a 19.6% LCC gain for wav2vec 2.0 Base.
• Naturalness prediction exhibits stronger transfer effects than intelligibility, as perceptual alignment between TTS synthesis artifacts and dysarthric naturalness is more direct. Using QualiSpeech naturalness under FT yields lowest MSE and highest LCC across all encoders, with up to 40.9% MSE reduction.
• Joint Training provides robust gains primarily on naturalness, but is consistently outperformed by FT on intelligibility. Negative transfer from label misalignment under JT degrades intelligibility discrimination, since MOS overall quality does not directly correlate with clinical phonemic clarity.
• Larger SSL models (315M params) benefit more from augmentation, especially when perceptual supervision matches the SAP target dimension. However, smaller backbones (wav2vec 2.0 Base) generalize better on intelligibility under in-domain setups.

Theoretical and Practical Implications

The demonstrated effectiveness of MOS-supervised augmentation suggests that synthesis artifacts and dysarthric manifestations share acoustic and perceptual commonalities, enabling transfer learning between TTS assessment and clinical speech pathology. This establishes a practical avenue to mitigate clinical annotation scarcity by leveraging large-scale, richly labeled synthesis corpora.

From a theoretical standpoint, the results highlight the importance of cross-domain label alignment. Sequential fine-tuning preserves discriminative features critical for clinical tasks, whereas naive joint optimization with misaligned supervision may induce negative transfer. Future work could focus on domain-invariant representation learning or adaptive label mapping strategies to further enhance transferability.

Practically, adopting MOS supervision can unlock scalability in automated dysarthria assessment, supporting longitudinal clinical tracking, therapy analysis, and fast prototyping for downstream ASR and disordered speech reconstruction systems. Moreover, the findings encourage the systematic exploration of other perceptually related domains (e.g., voice conversion, speech enhancement) for augmenting pathological speech assessment tasks.

Conclusion

This work proposes and validates MOS-based augmentation for dysarthric speech severity assessment, demonstrating strong quantitative improvements—particularly with sequential fine-tuning and dimension-matched supervision. The observed performance advantages confirm perceptual commonalities between speech synthesis artifacts and dysarthria, underscore the utility of large-scale TTS evaluation corpora, and motivate future research in domain-adaptive transfer learning for clinical speech technologies.