GigaSpeechBench: A Real-World Multilingual Speech-to-Text Benchmark

Abstract: While modern ASR systems achieve low error rates on high-resource benchmarks, such performance often overestimates real-world robustness. Existing evaluations address challenges in isolation, lacking a unified benchmark for domain terminology, age variation, dialects, accents, and low-resource languages, particularly across the Middle East and Southeast Asia, representing over one billion under-evaluated speakers. To address this gap, we introduce GigaSpeechBench, a comprehensive multilingual and multidimensional in-the-wild ASR & AST benchmark comprising 680 hours of human-annotated speech. It features five modules: (1) 12 low-resource Middle Eastern and Southeast Asian languages, plus challenging Japanese and Korean; (2) 6 Chinese dialects; (3) 6 English accents; (4) dense terminology across 12 vertical domains for Chinese and English; and (5) older adult and child speech. We further provide human-annotated Chinese and English translations for 11 languages to support AST evaluation. Extensive evaluations of leading foundation models and commercial APIs reveal significant performance degradation in these challenging settings, exposing critical evaluation blind spots.

• The paper presents a unified benchmark that evaluates ASR models on real-world, spontaneous, and diverse speech samples across language, accent, and domain variations.
• It demonstrates substantial performance drops with error rates up to 30–70%, revealing critical gaps in both commercial and open-source ASR systems.
• The benchmark highlights demographic and domain-specific challenges, emphasizing the need for more robust and adaptable speech recognition models.

GigaSpeechBench: A Unified Real-World Multilingual Speech-to-Text Benchmark

Motivation and Limitations of Existing ASR Benchmarks

Automatic Speech Recognition (ASR) models demonstrate near-saturating performance on high-resource, controlled benchmarks, but real-world deployment exposes persistent weaknesses in robustness, especially when confronted with diverse acoustic environments, linguistic variation, and underrepresented demographic or domain axes. Existing multilingual benchmarks disproportionately feature read or prompted speech, emphasizing language coverage at the expense of acoustic and lexical diversity. This design occludes acute challenges present in spontaneous, noisy, multi-speaker, or dialect-rich contexts as well as in speech by children, the elderly, or in terminology-dense vertical domains. Furthermore, high-population language regions in the Middle East and Southeast Asia remain under-evaluated, contributing to critical blind spots in model assessment.

The GigaSpeechBench project responds to the absence of a unified, challenging, multidimensional evaluation platform for real-world speech recognition, addressing gaps across language resource disparity, dialect and accent coverage, domain specificity, and demographic diversity.

Figure 1: Real-world speech presents diverse acoustic, linguistic, and lexical challenges, ranging from noise, speaker overlap, and far-field recordings to dialects, accents, and domain-specific terminology.

Benchmark Construction and Dataset Overview

GigaSpeechBench comprises 680 hours of manually-annotated, in-the-wild speech segments, curated along five complementary axes:

• Low-Resource and Regional Languages: 12 languages from the Middle East and Southeast Asia, plus Japanese and Korean, each with 20 hours, focusing on challenging, non-overlapping, spontaneous conversational audio.
• Chinese Dialects: Xiang, Jin, Gan, Min, Yue, and Wu dialects, each with 10 hours.
• English Accents: Chinese, Indian, Japanese, Filipino, Scottish, and Singaporean English accents, each with 10 hours.
• Vertical Domains: 12 specialized domains in Chinese and English (e.g., Medicine, Law, AI, Finance), each with 10 hours, annotated for technical entity recognition.
• Age Groups: 10 hours each for older adult and child speech in Chinese and English.

Robust selection protocols ensured retention of spontaneous, multi-speaker, and noisy environments while maintaining annotatability. Quality-assured segmentation and transcription produce diverse natural utterance distributions, avoiding the bias of short, script-driven audio common in prior corpora.

Figure 2: Distribution of audio segment duration and transcript length, capturing the spontaneous, conversational nature of collected speech.

In addition to native-language transcripts, 11 languages are supplemented with human-annotated English and Chinese translations, enabling direct ASR and speech-to-text translation (AST) evaluation.

Experimental Evaluation and Key Results

Comprehensive evaluation covers a spectrum of state-of-the-art commercial (Azure, Google Chirp, OpenAI GPT-4o Transcribe, Gemini), open-source (Qwen3-ASR, Whisper Large v3, NVIDIA NeMo, FunASR, Dolphin, SeedASR), and foundation models (Qwen3.5-Omni-Plus, Meta OmniASR).

Key empirical findings:

• Substantial Degradation in Real-World Conditions: All evaluated systems exhibit sharp performance drops on GigaSpeechBench compared to legacy benchmarks (Common Voice, FLEURS). For example, error rates exceeding 30–70% (WER/CER) are observed on low-resource and dialectal speech, even as the same models achieve <10% error on controlled corpora.
• Pronounced Difficulty with Accent, Dialect, and Demographic Variation: Even leading models display 2–5× higher word or character error rates on Chinese dialects and non-native English accents. Child and elderly speech recognition remains especially unreliable, highlighting persistent demographic bias in ASR robustness.
• Entity Recognition in Vertical Domains Remains Vulnerable: Biased word error rates (B-WER/CER) for key technical entities in professional domains are high, despite adequate general transcription performance—underscoring that progress on average metrics conceals deficiencies on critical long-tail vocabulary.
• Speech Translation: AST systems, even those optimized for noise, yield low sacreBLEU, chrF++, COMET, and BLEURT on in-the-wild, low-resource language test sets. This reflects ongoing challenges in multilingual, real-world transcription and translation, including for models such as Azure Translate, Gemini 3.0 Flash, and SeamlessM4T-v2-Large.

No system evaluated achieves robust performance across all axes—there exists a systematic mismatch between benchmark performance in sanitized conditions and in-the-wild generalization.

Benchmark Implications and Future Research Directions

GigaSpeechBench constitutes an authoritative diagnostic tool for measuring progress on the underrepresented axes that delimit real-world ASR utility. Its empirical results reveal:

• Existing foundation models are insufficiently robust for deployment in low-resource, dialect-rich, age-diverse, and entity-dense domains.
• Aggregate metrics, such as global WER/CER on high-resource languages, provide limited insight into true model reliability—application-sensitive axes require targeted, individually stratified evaluation.
• Theoretical advances are needed to address domain adaptation, robust entity modeling, and acoustic/linguistic variance, as incremental scaling of current architectures and datasets will not close the performance gap exposed by this benchmark.
• Benchmarking protocols must embrace hierarchical or entity-aware error metrics to better reflect operational success in downstream, high-stakes speech applications.

The released protocol, annotation pipeline, and evaluation suite facilitate reproducibility and drive the development of models explicitly optimized for long-tail, real-world axes.

Conclusion

GigaSpeechBench establishes a comprehensive, multidimensional, in-the-wild evaluation standard for multilingual ASR and AST. The benchmark demonstrates that leading systems remain brittle in the presence of acoustic, linguistic, demographic, and domain diversity characterizing global speech. Recognizing and overcoming these failures will be necessary for the next generation of robust, universal speech models—both theoretically and practically—in speech recognition and understanding (2606.28884).