Improving End-to-End Speech Recognition for Dysarthric Speech through In-Domain Data Augmentation

Abstract: Dysarthric speech recognition is crucial for facilitating effective communication among individuals with dysarthria. However, accurately recognizing dysarthric speech poses significant challenges due to varying severity levels and limited data availability. In this paper, we explore data augmentation techniques for dysarthric automatic speech recognition (ASR) systems by fine-tuning the End-to-End pre-trained Wav2Vec2 model, with a specific focus on severity levels. To address the challenges of data scarcity and the need for extensive data in fine-tuning pre-trained ASR systems for dysarthric speech, we investigate four prominent data augmentation methods: Speaking-Rate Modification (SRM), Pitch Modification (PM), Formant Modification (FM), and vocal tract Length Perturbation (VTLP), tailored to different aspects of dysarthria. The study uses individually fine-tuned Wav2Vec2 models for each severity class as baseline systems. Additionally, we conducted severity-specific fine-tuning of the ASR model using augmented data. Results demonstrate distinct efficacy patterns for each augmentation technique across severity levels. The best WERs were achieved with SRM ($s$=0.8) for \textit{low} (9.02\%) and \textit{medium} (38.11\%) severities, and with PM ($τ$=0.8) for \textit{high} severity (55.15\%), reflecting relative improvements of 30.02\%, 16.64\%, and 15.47\%, respectively. These results confirm the effectiveness of the augmentation methods in improving dysarthric ASR performance.

• The paper demonstrates that severity-specific data augmentation, especially speaking-rate and pitch modifications, significantly reduces word error rates in dysarthric ASR.
• The methodology tailors augmentation parameters to dysarthria severity levels, achieving up to 30% relative improvement in WER across test cases.
• Integrating these strategies during Wav2Vec2 fine-tuning enhances model generalization and provides a robust framework for pathology-specific speech recognition.

In-Domain Data Augmentation for End-to-End Dysarthric Speech Recognition

Introduction

Automatic Speech Recognition (ASR) for dysarthric speech is hindered by substantial inter- and intra-speaker variability, non-standard articulation, and scarcity of in-domain training data. Addressing these challenges requires techniques that enhance data diversity and model robustness. The paper "Improving End-to-End Speech Recognition for Dysarthric Speech through In-Domain Data Augmentation" (2606.19797) systematically investigates the use of four established speech data augmentation methods—Speaking-Rate Modification (SRM), Pitch Modification (PM), Formant Modification (FM), and Vocal Tract Length Perturbation (VTLP)—applied during fine-tuning of the pre-trained Wav2Vec2 model. Uniquely, the study tailors augmentation factors to specific dysarthria severity levels, enabling analysis of both within-severity performance and cross-severity generalization.

Data Augmentation Techniques

The augmentation techniques deployed serve to increase the effective diversity of dysarthric training data, targeting aspects of speech known to be affected by dysarthria:

• Speaking-Rate Modification (SRM): Alters temporal dynamics via Time Scale Modification based on the RTISI-LA algorithm. By varying scaling factor $s$ across $0.5 \leq s \leq 2$, the method exposes the model to disfluencies and non-standard speech timing.
• Pitch Modification (PM): Engages frequency scaling, adjusting pitch without affecting duration, via STFT-based resynthesis controlled by factor $\tau$.
• Formant Modification (FM): Employs Linear Predictive Coding with pole warping by parameter $\alpha$ to manipulate formant structure, reflecting altered resonance patterns typical in severe dysarthria.
• Vocal Tract Length Perturbation (VTLP): Warps the frequency axis by modification factor $\beta$, simulating inter-speaker anatomical variation and bolstering robustness to spectral shifts.

End-to-End Fine-Tuning Framework

A large pre-trained Wav2Vec2 model, utilizing 60K hours of Libri-Light speech for pre-training and 960 hours from LibriSpeech for supervised fine-tuning, was further adapted on the TORGO dysarthric speech corpus. Data augmentations were performed only on the training set for each experiment. The ASR system was assessed using word error rate (WER), with severity-specific training and evaluation. Severity classes followed established Franchey Dysarthria Assessment (FDA) ratings for low, medium, and high severity.

Figure 1: Block diagram of the proposed framework for employing data augmentation during the fine-tuning of a pre-trained Wav2Vec2 model.

The framework in (Figure 1) encapsulates model initialization, data augmentation, training, and evaluation, reflecting a modular approach to the integration of augmentation strategies with pre-trained self-supervised models.

Experimental Analysis

Baseline and Data Augmentation Impact

Baseline fine-tuning without augmentation established a cross-severity WER of 41.28%, with pronounced performance drops when training and testing severities were mismatched (e.g., model trained on high severity, tested on low: WER = 12.89%; trained on medium, tested on high: WER = 65.24%).

Data augmentation was systematically applied to each severity class with parameter sweeps to identify optimal modification factors for SRM, PM, FM, and VTLP. The strongest results were as follows:

• SRM ($s=0.8$): Achieved best performance on low (9.02% WER, 30.02% relative improvement) and medium severity (38.11% WER, 16.64% improvement)
• PM ($\tau=0.8$): Delivered best results on high severity (55.15% WER, 15.47% improvement)

  Figure 2: Results for the augmentation of high severity data during training/fine-tuning, tested on low and medium severity levels, across all four augmentation methods.

(Figure 2) illustrates that the efficacy of each augmentation method is both severity- and factor-dependent. Notably, SRM with lower $s$ values provided the most substantial WER reduction on less severe test sets, confirming the necessity of targeted augmentation tuning. PM, FM, and VTLP offered incremental improvements, but none surpassed the gains from SRM and PM for optimal test severity pairing.

Figure 3: Results for the augmentation of medium severity data during training/fine-tuning, tested on low and high severity levels, detailing the impact of each augmentation method and factor value.

(Figure 3) highlights that, for high severity test cases, PM and SRM delivered the greatest improvements when trained on medium severity and augmented appropriately. FM and VTLP augmentation showed less pronounced, but still positive, effects depending on the exact $\alpha$ and $\beta$ choices.

Severity-Aware Model Generalization

Analysis demonstrated that training on high-severity augmented data improved generalization to both low and medium severities, and vice versa. The findings underscore the criticality of both the augmentation type and the modification parameter for maximizing cross-severity recognition performance.

Discussion and Implications

The results support several key claims:

• Severity-specific augmentation factors are essential: Optimal WER is only achieved when the parameterization of data augmentation is matched to the target severity, suggesting that uniform augmentation pipelines are suboptimal for dysarthric ASR.
• Speaking-rate and pitch modification are most beneficial: SRM and PM delivered the highest WER reductions for low/medium and high severities, respectively. This aligns with the phonetic manifestations of dysarthria, where rate and prosody are most frequently distorted.
• Augmentation improves cross-severity robustness: Models fine-tuned with high-severity augmented data display improved decoding for less severe speech, suggesting a regularization effect and increased generalizability.

From a practical perspective, this research establishes that methodical in-domain data augmentation, informed by severity annotation, can partially compensate for the scarcity and heterogeneity inherent to pathological speech corpora. The approach is directly applicable within current ASR model development pipelines, especially those leveraging SSL representations.

Theoretically, the work demonstrates that end-to-end ASR models retain sufficient plasticity during fine-tuning for augmentation-induced benefit, advocating for further exploration of task-specific augmentation design. The documented dependence of optimal factors on severity implies that automatic severity estimation and adaptive augmentation selection could drive further performance gains.

Future Directions

Practical deployment of these findings remains hampered by the lack of automatic speaker severity assessment in real-world systems. Future research should focus on SSL-based severity detection, leveraging unsupervised representations to categorize speaker characteristics prior to specialization of ASR models. Integrating severity detection as an upstream module offers potential for dynamic, severity-aware ASR systems capable of improved personalization and accuracy.

Additionally, comparative study against adversarial augmentation [e.g., (Freymuth et al., 2024)] and hybrid augmentation-adaptation strategies across larger pathology datasets could clarify further the boundary of benefit in data-limited, disordered-speech settings.

Conclusion

This study establishes that severity-aware in-domain data augmentation, particularly SRM and PM, enables substantial WER reductions when fine-tuning Wav2Vec2-based ASR models for dysarthric speech. Severity-specific augmentation parameters are required for optimal benefit, and augmenting training data from higher severity classes enhances generalizability to lower severity test cases. These results provide a robust benchmark for future work integrating severity detection and adaptive augmentation pipelines in pathological speech recognition.