Open ASR Leaderboard

• Open ASR Leaderboard is a benchmarking platform that standardizes evaluation protocols for automatic speech recognition across diverse datasets and languages.
• It employs rigorous text normalization and standardized metrics like WER and RTFx to ensure fair, reproducible comparisons of model performance and efficiency.
• The open-source infrastructure and detailed performance insights help researchers balance trade-offs between transcription accuracy and inference speed.

The Open ASR Leaderboard is a fully reproducible, transparent, and extensible benchmarking platform for automatic speech recognition (ASR) systems, designed to comprehensively compare state-of-the-art open-source and proprietary models across multiple datasets, domains, and languages. Central to its methodology are standardized protocols for data preparation, text normalization, and evaluation, enabling fair and reproducible assessments of both transcription accuracy and computational efficiency. The leaderboard distinguishes itself by offering dedicated tracks for multilingual and long-form speech recognition—areas typically neglected by conventional short-form, English-centric evaluations—and by open-sourcing all code, dataset loaders, and evaluation pipelines. Through rigorous comparison across more than sixty ASR systems, the Open ASR Leaderboard provides actionable insights into model performance, architectural trade-offs, robustness, and scalability.

1. Platform Architecture and Methodological Foundations

The leaderboard is structured around an open-source evaluation pipeline and interactive dashboard. Models from a diverse array of origins (more than 60 systems from 18 organizations) are submitted and processed under unified protocols. Evaluation is partitioned into three major tracks: an English leaderboard (short-form transcription), a Multilingual leaderboard (covering German, French, Italian, Spanish, Portuguese), and a Long-form leaderboard (assessing segments exceeding 30 seconds) (Srivastav et al., 8 Oct 2025). All code used for evaluation—including text normalization scripts and dataset loaders—is published for public inspection and reuse, ensuring reproducibility and transparency.

A standardized text normalization procedure, inspired by Whisper model protocols, is applied to both ASR outputs and ground truth references prior to metric computation. This includes removal of punctuation, conversion to lowercase, normalization of numerals to their textual forms, spelling standardization, and exclusion of filler words such as "uh." Such normalization minimizes confounding factors in error rate analyses and ensures results are robust across models that differ in their output formatting.

2. Evaluation Datasets and Metrics

The Open ASR Leaderboard utilizes 11 diverse datasets encompassing various speech phenotypes and languages. Short-form English tasks use datasets like LibriSpeech (read audiobooks), TED-LIUM v3 (conference talks), GigaSpeech, MLS, SPGISpeech, and VoxPopuli (Srivastav et al., 8 Oct 2025). Multilingual tracks leverage CoVoST-2 and FLEURS, which include utterances in German, French, Italian, Spanish, and Portuguese. Long-form tracks rely on datasets such as AMI Meeting Corpus (spontaneous meetings) and Earnings21/22 (earnings call transcriptions), with segment lengths exceeding 30 seconds and mixed speech domains.

The primary evaluation metric is Word Error Rate (WER), with the computation standardized as:

$\text{WER} = \frac{S + D + I}{N}$

where $S$ = substitutions, $D$ = deletions, $I$ = insertions, and $N$ = total words in the reference. All WER computations follow the pre-normalization pipeline described above.

To measure computational efficiency, the leaderboard introduces the inverse real-time factor (RTFx), computed as:

$$\text{RTFx} = \frac{\text{Total Duration of Audio}}{\text{Transcription Time}}$$

Here, larger RTFx values correspond to faster inference. Both WER and RTFx are reported for each system-dataset pair, enabling joint assessments of accuracy and efficiency.

3. System Architectures and Performance Insights

Submissions to the leaderboard reflect a broad spectrum of ASR architectures, including encoder-decoder systems (e.g., Conformer encoders with LLM decoders), CTC (Connectionist Temporal Classification) and TDT (Time-Delayed Transformer) frameworks, as well as modified Whisper-derived encoders.

Comparative results indicate that Conformer encoders paired with LLM decoders yield the lowest average WER, particularly on English short-form transcription tasks (Srivastav et al., 8 Oct 2025). However, these architectures exhibit relatively low RTFx, denoting slower inference speeds in practice. Systems employing CTC and TDT decoders, by contrast, converge to higher RTFx values—sometimes several times faster than LLM-based systems—making them preferable when throughput or latency is paramount (e.g., for long-form or offline applications). A plausible implication is that practitioners should tailor system selection to target deployment scenarios, balancing error rates against resource constraints.

In the multilingual track, Whisper-based architectures fine-tuned specifically for English demonstrate improved accuracy for that language but often exhibit degraded performance or reduced coverage in other languages. Models optimized for broad multilingual support (e.g., supporting 25 languages instead of 4) sometimes trade off English transcription accuracy for more balanced cross-lingual WER performance.

4. Technical Innovations: Text Normalization and Reproducibility

The leaderboard’s standardized text normalization protocol is a salient technical contribution. By systematically applying normalization steps (punctuation removal, spelling/case normalization, number expansion, filler word exclusion) before scoring, the pipeline addresses pervasive issues in ASR benchmark comparisons where superficial discrepancies in formatting might confound reported error rates (Srivastav et al., 8 Oct 2025).

Additionally, the open-source release of all scoring code, dataset loaders, and evaluation scripts under permissive licenses permits direct replication and extension of benchmarking results. This process enables continuous integration of new models, datasets, and tracks by both academic and industrial researchers, democratizing the evaluation process and reducing barriers to entry.

5. Analysis of Accuracy-Efficiency Trade-offs and Model Robustness

Leaderboard findings demonstrate a characteristic trade-off between transcription accuracy and computational efficiency. Conformer-LLM models achieve top-tier WER but often with lower throughput, whereas CTC- or TDT-based models process audio significantly faster at the expense of modestly higher error rates (Srivastav et al., 8 Oct 2025). This dichotomy is especially pronounced in long-form tracks, suggesting that offline or non-real-time contexts may favor highly accurate but slower models, while real-time scenarios (e.g., live transcription services) may require faster, albeit less precise, architectures.

In multilingual and multi-domain tracks, observations reveal that fine-tuning for a specific language (e.g., English) may inversely impact performance on other languages, and that the expansion of supported languages can decrease monolingual accuracy. Model robustness to phenomena such as spontaneous speech, domain shift, and audio segment length is an open area for future benchmarking.

6. Challenges and Future Directions

The leaderboard identifies persistent challenges related to domain coverage and efficiency reporting. Most submissions remain focused on short-form, English-centric transcription, with far-field, accented, or code-switched speech underrepresented (Srivastav et al., 8 Oct 2025). Efficiency comparisons—particularly for closed-source or commercial systems—can be confounded by hardware variability (GPU usage, upload latency), limiting cross-system interpretability.

Future work foresees the expansion of language and domain coverage (including additional robustness tracks), the introduction of complementary metrics such as token error rate, and deeper investigation of new encoder-decoder paradigms (including LLM-augmented ASR and multi-modal pipelines). There is ongoing interest in refining benchmarking for large-scale and live transcription use-cases, establishing unified protocols for long-form audio—which is increasingly relevant for meeting, broadcast, and open-domain conversational applications.

7. Impact and Community Relevance

By leveraging open-source resources, standardized pipelines, and transparent reporting, the Open ASR Leaderboard stands as a central benchmarking infrastructure for the ASR research community. Its extensible architecture enables ongoing integration of new models and tasks, supporting both state-of-the-art progress tracking and longitudinal analysis of accuracy and scalability trends. The inclusion of multilingual and long-form tracks helps redirect research attention to historically neglected areas, supporting more equitable and practically useful advances in speech recognition technology (Srivastav et al., 8 Oct 2025).

Open sourcing the evaluation framework and dataset loaders democratizes benchmarking and facilitates community-driven progress, inviting broad participation and enabling fair, reproducible, and actionable comparisons across heterogeneous ASR systems.