Efficient ASR Training with Conversations that Never Happened

Abstract: Conversational ASR for lower-resource languages and niche domains is limited by the scarcity of domain-matched multi-speaker training data. We propose an augmentation pipeline that generates scenario-level dialogues with participant metadata, maps speaker attributes to TTS voice profiles, and assembles synthesized utterances into speaker-aware simulated conversations. We evaluated five LLM families under single-generator, fixed-budget mixture, and scale-up settings using the same FastConformer-Large training recipe for each one. We ran comprehensive evaluations on the Hungarian BEA-Dialogue benchmark corpus, with the method itself being applicable to any language given the resources for each component. The results show that synthetic conversations consistently improve speech recognition performance, but generator choice and data composition strongly affect the gains. Our largest training configuration, using only 67 hours of real conversations and 636 hours of simulated data, achieves better performance on the evaluation benchmark than a zero-shot model trained on 2700 hours of Hungarian speech. These findings indicate that LLM-generated conversational data synthesized with TTS is a practical complement to real conversational corpora for speech model training.

• The paper introduces a unified three-stage pipeline leveraging LLMs and TTS to generate synthetic, speaker-aware conversations for ASR training.
• It demonstrates significant performance gains (e.g., cpWER reductions from ~17.75% to 15.40%) by optimizing generator mixtures and data scale.
• The findings challenge conventional reliance on large annotated corpora, emphasizing dialogue coherence and metadata conditioning for effective ASR augmentation.

Efficient ASR Training with LLM-Generated Synthetic Conversations: An Expert Review

Introduction and Motivation

Automatic speech recognition (ASR) performance remains highly contingent on the availability of large, domain-matched, multi-speaker datasets—a condition rarely met for low-resource languages or specialized conversational domains. Traditional signal-level augmentation (e.g., SpecAugment, speed perturbation) or utterance-level TTS-based methods fail to capture the complex interplay between speaker identity, discourse structure, and contextual diversity found in natural conversations. This paper, "Efficient ASR Training with Conversations that Never Happened" (2606.03957), introduces a unified pipeline leveraging LLMs and neural TTS for simulating highly controlled, diverse, and speaker-aware synthetic conversations. The efficacy of this approach is rigorously benchmarked on Hungarian conversational ASR, opening new avenues for data-efficient recognition systems in resource-constrained domains.

Pipeline Architecture and Key Methodological Insights

The proposed three-stage augmentation pipeline consists of:

1. LLM-based Scenario and Dialogue Generation: For each synthetic conversation, the pipeline generates a scenario with explicit participant metadata (age, gender, occupation, conversational role). This metadata drives multi-turn dialogue generation, yielding not only diverse lexical content but also simulating authentic discourse phenomena, such as autobiographical or experience-driven interactions.
2. Metadata-Conditioned TTS Synthesis: Each utterance is rendered using neural TTS (specifically xTTS-v2), with speaker voices selected from a reference bank to match the generated speaker metadata. This preserves the alignment between synthetic linguistic content and the underlying speaker profile distributions.
3. Speaker-Aware Conversation Simulation: Synthesized utterances are assembled into multi-speaker conversational waveforms. Timing parameters (pauses, overlaps) are modeled via statistically grounded simulation, emulating natural turn-taking and interactional structure as observed in real conversational corpora.

This modular framework is engineered to be language-agnostic, contingent only upon the availability of language-supporting LLMs, a TTS system, and a sufficiently annotated speaker reference bank.

Experimental Design and Comparative Evaluation

Experimental validation is anchored on the Hungarian BEA-Dialogue corpus, using FastConformer-Large as the ASR backbone. The study isolates the impact of synthetic data composition by systematically varying three variables:

• Generator Identity: Five LLMs (GPT, Claude Haiku, Gemini, Grok, Qwen) are assessed for their effectiveness in single-generator and mixture scenarios.
• Synthetic Data Scale: The number of generated conversations is scaled incrementally ($K \in \{100, 200, 300, 400, 500\}$), with metrics tracked at each level.
• Generator Mixtures: Mixtures (pairwise up to five-way) are evaluated to probe whether generator diversity yields complementary benefits.

Baselines include zero-shot Whisper, a Hungarian monolingual model (trained on 2700h), real-data-only training, and prior speaker-aware conversation simulation using only real utterances.

Numerical Results and Empirical Findings

Single-Generator Scaling: All LLMs demonstrate positive transfer when augmenting real data, with GPT-5.4 mini yielding the lowest cpWER (17.75%) and cpCER (8.20%) at $K=500$. Notably, performance improves monotonically with larger synthetic datasets, although performance differences persist between generators, likely attributable to generated data quality and diversity, not just volume.

Figure 1: Ablation (scale-down) comparison for the single-generator LLM setup.

Mixture Effects: The optimal fixed-budget mixture (GPT+Haiku) achieves a further reduction in cpWER to 17.56%, surpassing the best single-generator configuration. However, increasing mixture cardinality does not guarantee continued improvements; mixtures with weaker or less complementary generators result in diminished or even degraded performance due to sample dilution.

Figure 2: Subset-level cpWER (BEA-Dialogue eval) across generator mixtures.

Complementarity and Interactions: Pairwise mixture analysis reveals that performance gains are only realized when combining generative models with sufficiently diverse and complementary conversational content distributions. The complementarity does not correlate strictly with standalone generator efficacy, indicating non-linear interaction effects in data-driven augmentation.

Figure 3: Pairwise mixture performance on the BEA-Dialogue eval set. Diagonal entries correspond to single-generator configurations, while off-diagonal entries correspond to uniform two-generator mixtures.

Scale-Up and Baseline Comparison: The full-scale, four-generator mixture (GPT+Haiku+Qwen+Grok) outperforms all real-data and simulated-data baselines, achieving 16.65% cpWER and 7.97% cpCER using only 67 hours of real data plus 427 hours of synthetic conversations. The system not only surpasses a model trained zero-shot on 2700 hours of Hungarian broadcast/broadcast conversational speech, but does so with orders-of-magnitude less human-labeled data. Moreover, combining LLM-based synthetic data with additional simulated conversations derived from real utterances yields the overall best result: 15.40% cpWER and 7.57% cpCER.

Theoretical and Practical Implications

This work demonstrates that scenario-driven, TTS-synthesized conversations generated via LLMs can substantially offset deficits in real conversational training data. Generator choice and data composition, particularly the selective combination of complementary LLMs, are shown to be decisive in achieving maximum augmentation benefits. The strong numerical results, especially surpassing a conventional 2700-hour supervised model using a small real-plus-synthetic corpus, challenge the orthodoxy of simply scaling labeled data collection in low-resource or domain-specific settings.

The analysis of generator mixtures carries practical insight for augmentation protocol design: diversity alone does not guarantee gains—models with weak or non-complementary conversational structures may in fact be detrimental if not carefully filtered. The findings also indicate that the utility of synthetic data plateaus if additional samples do not introduce new, high-value lexical or acoustic variability.

On a theoretical level, these results underscore the critical role of interactional and discourse coherence in synthetic augmentation. Approaches that systematize scenario, metadata, and dialogue generation via LLMs—rather than treating utterance synthesis as an independent, context-free process—are consistently more effective in end-to-end ASR training.

Future Directions

The pipeline's language independence, coupled with the open-ended possibilities of LLM-driven dialogue generation, portends wide applicability to other low-resource languages, dialects, and specialized conversational domains. Future research directions include scaling beyond 500 conversations, integrating in-domain or domain-adaptive LLMs, and exploring corpora with even more nuanced conversational phenomena (e.g., code-switching, multi-lingual interaction). Enhanced generator selection criteria, either via automatic dialogue feature analysis or discriminative adversarial filtering, may optimize mixture performance further.

Direct adaptation of this pipeline to domain-specific or application-specific conversational corpora—customer support, telemedicine, low-resource dialect recognition—offers a path toward robust ASR in settings where real data acquisition and annotation are prohibitively expensive.

Conclusion

LLM-driven scenario-based synthetic conversation generation, when coupled with speaker-matched TTS and interactional timing simulation, enables efficient and potent augmentation of ASR training corpora. Synthetic conversations constructed by this pipeline demonstrably enhance recognition performance in data-poor domains, with effectiveness conditional on careful generator selection and mixture strategy. The empirical evidence suggests a recalibration of resource allocation in ASR system development for low-resource languages: targeted synthetic data, generated at scale using strong LLMs, can rival or surpass the value of much larger annotated corpora. This marks a substantive advance in data-centric ASR training methodology and sets a new benchmark for future research in speech recognition for under-resourced or domain-specialized scenarios.