Mega-ASR: Towards In-the-wild^2 Speech Recognition via Scaling up Real-world Acoustic Simulation

Abstract: Despite rapid advances in automatic speech recognition (ASR) and large audio-LLMs, robust recognition in real-world environments remains limited by an "acoustic robustness bottleneck": models often lose acoustic grounding and produce omissions or hallucinations under severe, compositional distortions. We propose Mega-ASR, a unified ASR-in-the-wild framework that combines scalable compound-data construction with progressive acoustic-to-semantic optimization. We introduce Voices-in-the-Wild-2M, covering 7 classic acoustic phenomena and 54 physically plausible compound scenarios, and train Mega-ASR with Acoustic-to-Semantic Progressive Supervised Fine-Tuning and Dual-Granularity WER-Gated Policy Optimization. Extensive experiments demonstrate that Mega-ASR achieves significant advantages over prior state-of-the-art systems on adverse-condition ASR benchmarks (45.69% vs. 54.01% on VOiCES R4-B-F, and 21.49% vs. 29.34% on NOIZEUS Sta-0). On complex compositional acoustic scenarios, Mega-ASR further delivers over 30% relative WER reduction against strong open- and closed-source baselines, establishing a scalable paradigm for robust ASR in-the-wild.

• The paper introduces Mega-ASR, which uses a large-scale compound acoustic simulation dataset (VOICES-IN-THE-WILD-2M) to address real-world ASR degradation.
• It employs a multi-phase Acoustic-to-Semantic fine-tuning and dual-granularity WER-gated reinforcement learning to optimize both local error correction and global semantic recovery.
• Experimental results demonstrate significant WER reductions across clean, adverse, and compound conditions, setting a new state-of-the-art benchmark for robust ASR.

Mega-ASR: A Scalable Paradigm for In-the-Wild Speech Recognition via Realistic Acoustic Simulation

Introduction and Motivation

Automatic speech recognition (ASR) has achieved near-human performance on canonical clean and mildly degraded benchmarks with the advent of large audio-LLMs (LALMs) and billion-parameter architectures. However, when deployed in complex, real-world acoustic conditions characterized by severe, compositional degradations—such as simultaneous noise, reverberation, echo, occlusion, electronic distortion, and transmission dropout—state-of-the-art (SOTA) systems experience a marked erosion in robustness, with word error rates (WER) frequently rising to 30–70% and problematic phenomena including omissions and hallucinations. Existing robust ASR efforts primarily focus on isolated perturbations and fail to address the compositionality and scale of in-the-wild acoustic challenges, leading to both a lack of generalization and mismatch with real deployment settings.

VOICES-IN-THE-WILD-2M: Large-Scale Compound Acoustic Simulation

Mega-ASR introduces VOICES-IN-THE-WILD-2M, a large-scale dataset specifically constructed for evaluating and training ASR systems under realistic, multi-factor acoustic conditions. This dataset comprises 2.4M clips modeled across 7 atomic acoustic effects—noise, far-field, obstructed, echo&reverb, recording coloration, electronic distortion, and transmission dropout—and enumerates 54 physically plausible compound scenarios, capturing scenarios such as "far-field with echo and electronic distortion" or "obstructed with channel dropout and noise."

Key innovations in dataset construction include:

• Spectral-manipulation-based simulation: Each atomic effect is realized via parameterized signal-level transformations (e.g., convolution, filtering, resampling) tuned to replicate real world conditions.
• Agentically-validated composition: Only physically plausible combinations are allowed in compound scenarios, with an agent check to ensure realism (e.g., a church maps to far-field plus echo, but not to contradictory configurations).
• Difficulty calibration: Dataset samples are controlled via a global severity variable $m \in [0,1]$, mapped from uniformly sampled $x$ using several candidate distributions (linear is adopted for balanced coverage), with parameter sharing across effects for internal consistency.
• Learnability filtering: Utterances for which WER exceeds 70% are discarded to maintain training stability and avoid catastrophic failures that degrade optimization.

The result is a highly challenging training and evaluation regime: SOTA LALMs such as Qwen3-ASR achieve an average WER of 35% on VOICES-IN-THE-WILD-2M, in contrast with WERs below 10% on conventional benchmarks.

Mega-ASR Architecture: Progressive Acoustic-to-Semantic Robustification

Acoustic-to-Semantic Progressive Supervised Fine-Tuning (A2S-SFT)

Mega-ASR is initialized and developed atop Qwen3-ASR-1.7B using a novel multi-phase curriculum, A2S-SFT, reflecting the observation that models must be adaptively conditioned to extract and recover semantic content from progressively degraded acoustics.

• Stage I: Encoder and aligner modules are adapted using supervisory learning with a curriculum on utterances graded by WER, expanding from WER < 30% to WER < 70%, sharpening local acoustic evidence extraction.
• Stage II: LLM-side parameters are fine-tuned to activate robust semantic reconstruction capabilities in the face of unreliable or ambiguous acoustic evidence.
• Stage III: Joint acoustic-semantic adaptation is performed to harmonize the model’s representations and decoding between the acoustic encoder, aligner, and LLM, facilitating end-to-end robust transcription.

Dual-Granularity WER-Gated Policy Optimization (DG-WGPO)

Conventional reinforcement learning approaches, using WER as the reward, fail when WER>30%, as error modes shift from local substitutions to catastrophic, sentence-level failures (e.g., hallucinations, semantic drift, omissions). Mega-ASR employs DG-WGPO, a DAPO-based RL framework that partitions the training signal into:

• Static rewards: Standard WER and anti-repetition penalties for direct, sample-independent anchoring.
• Dynamic dual-granularity rewards:
  • Token-level refinement—edit-similarity-based soft/hard substitution classification guides correction of local recognition errors.
  • Sentence-level reconstruction—LCS/length-based rewards shape faithful semantic backbone preservation under heavy degradation.
• WER-Gated Fusion: For WER < T (default T=0.3), the reward prioritizes local token correction; for WER ≥ T, it heavily weights sentence-level structure to focus optimization on semantic recovery.

This design bridges the reward landscape discontinuity identified in high-noise or compound settings and empirically supports the transition from local error correction to holistic semantic preservation.

Environment-Aware Routing

An auxiliary lightweight classifier (single-layer Transformer on log-Mel features) routes incoming audio to either the robust Mega-ASR weights (via LoRA delta activation) or the original Qwen3-ASR backbone, based on predicted degradation, preserving clean-speech and streaming performance and incurring negligible inference overhead (<1%).

Experimental Results

Mega-ASR is evaluated against a comprehensive suite of competitive baselines, including both closed-source (Gemini-3-Flash, GPT-4o) and open-source (Qwen2.5-Omni, Whisper-v3, Parakeet, Step-Audio-2, Kimi-Audio) systems, across three experimental regimes:

• Standard ASR benchmarks: LibriSpeech (clean/other), Common Voice22, FLEURS, AISHELL-1, WenetSpeech, VoxPopuli.
• Adverse-condition ASR: CHIME-4, VOICES, NOIZEUS, capturing noise, reverberation, far-field, and device effects.
• Compositional real-world settings: Voices-in-the-Wild-Bench, derived from the simulated dataset with both synthetic and real recordings.

Strong Numerical Results

• Clean Speech: With appropriate routing, Mega-ASR matches or marginally outperforms SOTA models on LibriSpeech (1.63/3.37 WER with routing) and remains highly competitive on multi-domain multilingual benchmarks.
• Robustness in Adverse Conditions: On CHIME-4, VOICES, and NOIZEUS, Mega-ASR sets new SOTA, achieving an average WER of 6.70, outperforming Qwen3-ASR (7.93) and Whisper-Large-v3 (10.72). Under the hardest NOIZEUS setting, it reduces WER by 17.4% relative to the strongest open-source competitor and by 64.5% vs. Gemini-3-Flash.
• Compound Acoustic Scenarios: On Voices-in-the-Wild-Bench with mixed degradations, Mega-ASR achieves WER=2.73/4.57, yielding >65% relative reduction over Whisper-Large-v3 and >65% over Gemini-3-Flash in scenarios involving multiple, concurrent distortions.

Error-Mode and Semantic Recovery

Qualitative analysis verifies that Mega-ASR, unlike its predecessors, is capable of converting catastrophic failure patterns (cross-lingual hallucinations, empty outputs, entity loss, semantic drift) prevalent under severe degradation into nearly correct or entirely correct transcriptions, shifting the error mode from total collapse to partial/local substitution. This is quantitatively validated: missed content drops from 14.2 to 5.9 on adversarial subsets, and hallucination rates and empty outputs diminish substantially.

Theoretical and Practical Implications

Mega-ASR demonstrates that robust, in-the-wild ASR is most effectively addressed through large-scale, realistic acoustic simulation with compositional scenario enumeration, and requires training protocols that explicitly bridge the gap between local acoustic evidence recovery and global semantic reconstruction. Importantly:

• Compound scenario coverage is non-negotiable: Isolated-perturbation augmentation is insufficient; physically motivated, scalable compound data drives tangible robustness improvements.
• Rewards must account for error-regime transitions: RLHF for ASR should dynamically adapt reward granularity in line with shifting failure modes.
• Modularity with task routing preserves generality and efficiency in practical deployment: Robustification can be decoupled from clean-speech or downstream specializations via plug-and-play inference design.

This work establishes a compelling new baseline and methodology for evaluating, training, and deploying ASR systems robust to the full range of real-world acoustic complexity.

Future Directions

Potential avenues include: extension to multilingual and cross-lingual scenarios, tighter integration of downstream instruction-following or reasoning tasks into the ASR-in-the-wild setting, and more detailed analyses of generalization to unseen environments from the compound simulation benchmark. The released dataset, code, and routing infrastructure provide a strong foundation for such advances, and the scalable paradigm of Mega-ASR is well-poised for integration with next-generation audio-ML architectures.

Conclusion

Mega-ASR represents a formal, empirically grounded advance in robust ASR, systematically addressing acoustic robustness bottlenecks by leveraging a scalable, compositional simulation dataset and principled dual-granularity optimization framework. The solution not only consistently outperforms previous SOTA on clean, adverse, and compound conditions, but also bridges the gap between controlled-benchmark evaluation and robust in-the-wild deployment, offering crucial insights for both academic inquiry and real-world speech AI systems.