How to Leverage Synthetic Speech for LLM-Based ASR Systems?

Abstract: In regulated domains such as banking and healthcare, where privacy constraints make real speech costly to collect and retain, synthetic speech from modern text-to-speech (TTS) is an appealing alternative for training automatic speech recognition (ASR) without exposing sensitive customer recordings. Yet a persistent distributional gap between synthetic and real data limits how far it can replace genuine recordings. Prior work largely treats this gap as a black box to be engineered around, but in our work, we instead examine its origin directly by probing a SLAM-ASR architecture. Then, we localise where its LLM backbone separates real from synthetic speech and find the discriminative signal concentrated in the early-to-middle layers, where temporal and prosodic perturbations disrupt it most. We further show that representation-level separability, help, but does not directly predict downstream ASR gains. On the other hand, convolving synthetic audio with room impulse responses (RIRs) narrows the gap not by making synthetic speech sound cleaner or more natural, but by reproducing the acoustic irregularities of real recordings. Translating these findings into the training procedure, by adding a layer-selection module combined with RIR augmentation matches a fully real-data baseline using only 25% of the real speech (13.6h) and surpasses it at all higher proportions.

• The paper's main contribution is revealing that using RIR augmentation and layer-wise pooling narrows the synthetic/real discrepancy in LLM-based ASR pipelines.
• It shows that RIR-augmented synthetic data can achieve comparable or better WER with as little as 25% real audio, optimizing privacy and data efficiency.
• The study finds that while early layers capture acoustic differences, later layers form domain-agnostic representations, highlighting the role of environmental simulation.

Leveraging Synthetic Speech for LLM-Based ASR: Probing Representational Gaps and Training Strategies

Introduction

The adoption of synthetic speech for Automatic Speech Recognition (ASR) training in regulated domains is motivated by escalating privacy constraints and data scarcity of real recordings. However, distributional discrepancies between real and synthetic speech challenge the direct substitution of real audio by synthetic counterparts, especially with state-of-the-art LLM-based ASR architectures. This work systematically characterizes where and how these discrepancies manifest in a modern LLM-backed ASR pipeline (SLAM-ASR), proposes architectural interventions and training augmentations, and validates their efficacy through detailed synthetic-real discrimination analyses and WER evaluations across varying real/synthetic ratios.

Architecture and Layer-wise Pooling Mechanism

The system is built on the SLAM-ASR paradigm, coupling a WavLM-Large encoder to a Llama-3.2-3B-Instruct LLM via a linear projector, with LoRA adapters for parameter-efficient domain adaptation. The canonical decoder feeds only the terminal LLM layer to the language modeling head. To probe and potentially mitigate synthetic/real separation, a Layer-wise Weighted Pooling (LWP) module is applied. This module learns softmax-normalized scalars, aggregating all transformer hidden states per token, thus optimizing which LLM layers to propagate to the output. The addition of an acoustic residual stream at speech-token positions is also explored.

Figure 1: Layer-wise Weighted Pooling mechanism for flexible aggregation of Llama hidden states, incorporating both learned weighting and optional acoustic residuals.

Synthetic Corpus Design and Speaker Diversity

Synthetic utterances are created using Qwen3-TTS VoiceDesign, driven by persona-level prompts matching target speaker demographics but not requiring reference speaker audio. The resulting synthetic corpus exhibits higher within-corpus speaker diversity than the real telephone domain data, as measured by cosine distance distributions over pyannote speaker embeddings. Notably, the synthetic set shows a more diffuse, bimodal speaker space—indicating preservation of real-like characteristics and introduction of enhanced vocal variety.

Figure 2: Pairwise cosine distance distributions showing greater speaker diversity among synthetic voices compared to the homogeneous real corpus.

Representational Analysis: Localization of the Synthetic/Real Gap

The work probes the LLM backbone at all transformer layers, before any domain-specific fine-tuning, quantifying separation via silhouette scores and distributional overlaps post-PCA. Results reveal that synthetic/real discrimination is maximized at lower and mid-level layers (layers 0–14), rapidly diminishing toward the final layers where task-relevant semantic representations are formed and the domain gap largely collapses. Several low-level perturbations (Time Stretch, Pitch Shift, Band/Low-pass filtering) are evaluated for their efficacy in masking the synthetic/real boundary, with prosodic modifications such as Time Stretch (rate 1.2) causing the most overlap in the early layers.

Figure 3: Layer-wise overlap trajectories across all ablation conditions; lower values denote higher synthetic/real representational mixing, predominantly in early-to-mid layers.

Data Augmentation and Acoustic Adaptation: RIRs Dominate

A core finding is that convolving synthetic speech with measured Room Impulse Responses (RIRs) narrows the distributional gap more effectively than perceptual enhancement alone. RIR augmentation introduces reverberation and channel effects mimicking the real domain, thus aligning domain statistics despite objectively reducing naturalness metrics (lower UTMOS), and only marginally affecting PESQ. This shift is pivotal: downstream ASR improvements stem from augmented acoustic diversity and not perceptual realism of synthetic audio.

Training Regimes and Downstream ASR Performance

Three main axes are empirically dissected:

• Synthetic-for-Real Substitution: Synthetic data with RIR augmentation matches, and often surpasses, the all-real baseline using as little as 25% real audio. The optimal mixture for substitution is 10–30% synthetic augmented with RIR, with diminishing returns and eventual degradation beyond 50% synthetic data.
• Synthetic Augmentation: Adding RIR-augmented synthetic data to the full real set yields consistent improvements, with the best results (minimal WER) at the highest amounts of synthetic augmentation. Raw, unaugmented synthetic audio delivers little to no benefit beyond 50%.
• Interpretability-guided Filtering: Among prosodic/temporal filters, Pitch Shift and High Pass + Pitch Shift combinations offer additional WER reduction (relative improvement up to 7.72%) when coupled with RIRs, confirming that early-layer overlap metrics (as revealed in the probe analysis) correlate with downstream ASR benefits only when the acoustic gap is also addressed.

Layer-wise Pooling Efficacy

Deployment of the LWP mechanism, especially in low-resource (substitution-heavy) scenarios, enables the ASR system to approach or exceed the fully real baseline at reduced annotation cost. Without RIRs, LWP shows limited or negative benefit except at very low real data fractions. With RIR augmentation, the pooling module consistently improves WER, with learned layer weights highly concentrated on the LLM's final transformer layer (layer 28) for both real and synthetic test sets. The acoustic residual stream is found redundant.

Implications and Future Directions

This analysis demonstrates that the synthetic/real gap is not primarily a function of waveform fidelity, but of unmodeled acoustic variability. Architectural changes (LWP), data augmentations replicating environmental conditions (RIRs), and targeted prosodic perturbations can together render synthetic speech a viable, privacy-compliant alternative in LLM-based ASR training. However, interpretability-based findings must be operationalized—early/mid-layer overlap does not guarantee downstream gains unless the acoustic distribution is holistically matched.

The results suggest future work in at least three directions:

• Extension to larger, multilingual synthetic sets and alternative encoder backbones for greater generalizability.
• Fine-grained, possibly contrastive or reinforcement-based, adaptation objectives directly optimizing domain invariance or WER.
• Systematic study of which perturbations or augmentations (including TTS model diversity) most efficiently bridge the synthetic/real chasm across both task and non-task metrics.

Conclusion

Layer-wise probing reveals that synthetic/real discrimination in LLM-based ASR models concentrates in early-to-middle layers, while optimal decoding relies on final-layer representations agnostic to input domain when sufficient acoustic augmentation (RIRs) is used. Incorporating a layer-wise pooling mechanism and interpretability-driven filtering enables significant reduction of real data requirements, supporting efficient, privacy-compatible deployment of large speech LLMs in regulated domains.