- The paper shows that fine-tuning Whisper models with synthetic speech reduces WER by 15-65% across Hindi, Kannada, and Telugu, highlighting the method’s efficacy.
- The study demonstrates that advanced voice cloning and moderate speaker diversity (10+ profiles) notably enhance ASR performance, though returns diminish beyond a certain diversity threshold.
- The analysis reveals that the choice of TTS backend and script quality, including human-curated versus LLM-generated texts, critically influences ASR accuracy.
Analysis of Synthetic Speech Data for ASR Fine-tuning in Indic Languages
Problem Context and Motivation
The collection of diverse, high-quality speech corpora remains a significant bottleneck for robust ASR model development in many low-resource languages, including the majority of Indic languages. The linguistic plurality and limited standardized datasets exacerbate these challenges. This study systematically investigates the potential of synthetic speech for fine-tuning ASR models in Hindi, Kannada, and Telugu—spanning the Indo-Aryan and Dravidian language families. The work evaluates synthetic data as both a supplement and, critically, a possible substitute for real speech, using several benchmarks and fine-grained ablation studies.
Synthetic Data Generation Protocols
Synthetic speech was generated using TTS models with advanced voice cloning, notably the Coqui TTS framework based on VITS, fine-tuned on the SYSPIN dataset for non-Hindi languages to ensure naturalness. Textual prompts came from a curated set, including human-produced (RESPIN, Indic Voices, Kathbath) and LLM-generated (Gemini 2.5 and 3) scripts. To examine the effect of speaker diversity, synthetic utterances were rendered using up to 10,000 distinct speaker profiles, leveraging Vaani's speaker pool. Multiple TTS systems (Coqui-XTTS-v2, IndicParlor TTS, and IndriTTS) were compared to decouple synthesis model effects.
Experimental Setup
The Whisper Small ASR model served as the baseline, with fine-tuning implemented under controlled strategies: (i) authentic real speech (Vaani), (ii) exclusively synthetic speech, (iii) combined real and synthetic speech, and (iv) various combinations involving RESPIN. For evaluation, models were tested on diverse in-domain and out-of-domain datasets (Gram Vaani, FLEURS, MUCS, Common Voice, Kathbath, RESPIN), accounting for variation in accents, noisiness, and recording environments. Experimentation also included ablations on speaker diversity, script source, and ASR model size (from Whisper Tiny to Large).
Key Findings
Effectiveness of Synthetic Data
Fine-tuning Whisper models with a mixture of synthetic and authentic data consistently reduced WER across all languages studied. The reported WER improvements from data augmentation with synthetic speech (on top of the Vaani corpus) were 15.06% (Hindi), 9.27% (Kannada), and 13.19% (Telugu). Using synthetic data exclusively reduced WER by 62.68% (Hindi), 48.85% (Kannada), and 65.21% (Telugu) relative to unadapted models, clearly establishing synthetic speech as especially effective in low-resource scenarios.
However, the absolute performance of synthetic-only models remains behind that achieved by further supplementing with additional real speech, underscoring that synthetic speech, while highly complementary, cannot yet fully replace the advantages conferred by naturally diverse human recordings.
Impact of Voice Cloning and Speaker Diversity
Voice cloning substantially enhances the utility of synthetic data for ASR fine-tuning. Gains in ASR robustness and WER were disproportionately realized by increasing the number of unique speaker profiles in synthetic data from 1 to 10; beyond this, WER plateaued, showing little additional reduction. For models with strong pre-training and broad initial speaker coverage (such as Whisper), moderate speaker diversity (10+ distinct speakers) in synthetic data is sufficient for maximizing speaker adaptation effects; exponential increases in diversity yield diminishing returns.
Script Quality and TTS Model Factors
The choice of script source for synthetic speech had a measurable but secondary impact on WER, with both human-curated and LLM-generated scripts supporting meaningful ASR improvements. Quality, naturalness, and representativeness of transcription texts continue to affect downstream performance, implicating script diversity and correctness as important factors. Across TTS backends, IndriTTS yielded the greatest mean WER reduction (52.49%), followed by Coqui-XTTS-v2 and IndicParlor, highlighting the practical importance of TTS model selection.
Whisper Model Scalability
Fine-tuning improvements persisted across Whisper variants. Higher-capacity models (Medium, Large) benefited more from fine-tuning with synthetic data, demonstrating enhanced capacity to leverage diverse and potentially noisy synthetic data for error reduction.
Practical and Theoretical Implications
The empirical results support the integration of synthetic speech data in ASR pipelines for Indic languages, particularly for rapid domain or speaker adaptation in settings with minimal real recordings. Current TTS systems, especially those with robust multilingual voice cloning, can supply effective augmentation data. However, the fact that real speech remains superior points to critical gaps—perhaps in prosodic variation, spontaneous speech phenomena, and environmental realism—that current synthesis models do not capture.
LLM-generated scripts extend the diversity and scale of synthetic speech resources, but do not supplant the need for human curation to maintain transcription quality. Speaker diversity should be prioritized to about 10–100 unique synthetic speakers; increasing beyond this confers little additional ASR benefit, at least for well-pretrained models.
The findings suggest that future improvements in synthetic data quality should target not just acoustic naturalness but also prosodic variability, dialectal coverage, and spontaneous speech features. For TTS research, system robustness to domain and language expansion, nuanced emotion, and environmental conditions will likely further narrow the gap with real data-driven ASR outcomes.
Conclusion
This study provides a comprehensive evaluation of synthetic speech data as a tool for ASR fine-tuning in low-resource Indic languages. Synthetic speech—especially when generated via advanced voice cloning from sufficiently diverse scripts and speakers—offers significant ASR performance improvements, though it does not fully substitute for real speech data. Script quality, TTS engine choice, and moderate speaker diversity are dominant factors for maximizing gains. The continued superiority of authentic speech indicates opportunities for advancing TTS realism and diversity. Ongoing innovation in synthetic speech generation will be critical for scaling ASR to the long tail of under-resourced languages.