OLMoASR: Open Data ASR with Zero-Shot Robustness

• OLMoASR is an open speech recognition project that uses web-scale, English-only audio data and targeted heuristic filtering to enhance zero-shot performance.
• The project demonstrates that careful data curation can narrow performance gaps with systems like OpenAI’s Whisper, achieving competitive WER scores on diverse benchmarks.
• Its suite of encoder–decoder Transformer models, ranging from 39M to 1.5B parameters, validates the impact of large-scale, high-quality data on robust ASR performance.

OLMoASR is an open speech recognition project centered on the proposition that data scale and data quality are first-order determinants of robust zero-shot ASR. It comprises the large-scale dataset OLMoASR-Pool, the curated subset OLMoASR-Mix, and a suite of English-only encoder–decoder Transformer models trained on that corpus. In the reported study, OLMoASR-Pool contains 3M hours of English audio and 17M transcripts; the curation pipeline yields OLMoASR-Mix with approximately 1M hours of high-quality audio-transcript pairs; and the resulting models span 39M to 1.5B parameters while achieving comparable average performance to OpenAI’s Whisper on short and long-form speech recognition benchmarks (Ngo et al., 28 Aug 2025).

1. Project scope and naming

OLMoASR denotes more than a model family. In the underlying work, it names a data-and-model program composed of OLMoASR-Pool, OLMoASR-Mix, and the OLMoASR-Mix suite of models, together with filtering, training, and evaluation code (Ngo et al., 28 Aug 2025). A common simplification is to treat OLMoASR as a new neural architecture; the paper instead presents it as a controlled investigation of robust ASR under web-scale weak supervision, with the architectural baseline intentionally kept close to Whisper.

The project is explicitly English-only. OLMoASR-Pool is described as a collection of English audio with associated transcripts, and the released model suite is organized as tiny.en, base.en, small.en, medium.en, large.en, and large.en-v2. This English-only scope is consequential for interpreting the scaling results: the work reports strong zero-shot English robustness, but it does not claim multilingual coverage.

The project’s central thesis is that targeted text heuristic filters can convert heterogeneous web data into a training mixture that materially improves zero-shot ASR. The paper’s summary states that a controlled experiment, holding architecture and optimizer constant, confirms the central role of data curation. This suggests that OLMoASR should be understood as a data-centric research program rather than as an attempt to supersede Whisper through architectural novelty alone (Ngo et al., 28 Aug 2025).

2. Corpus construction and curation pipeline

OLMoASR-Pool is built from publicly accessible audio+text pairs scraped from web video and audio platforms (e.g., YouTube), audiobooks, podcasts, lectures, etc. The transcripts include both manually uploaded subtitles and (unlabeled) automatic captions, and the corpus is characterized by heterogenous audio quality, diverse speakers, accents, and domains. The final curated set, OLMoASR-Mix, is drawn from this pool after a four-stage filtering process intended to remove low-quality or mis-transcribed examples while preserving breadth of acoustic and topical coverage (Ngo et al., 28 Aug 2025).

The curation stages are reported with both retained hours and short-form WER on a held-out average over 14 benchmarks.

Stage	Remaining data	Held-out short-form WER
No quality filters	3M hr	37.2%
Audio/Text Language Alignment	2.4M hr	—
Casing Filter	1.37M hr	32.4%
Repeating-Lines Filter	1.21M hr	22.7%
Manual–Machine WER Filter	908,923 hr	20.7%

The first stage uses VoxLingua107 for spoken-language identification on audio and pycld2 for language detection on transcript text, removing any pair for which either tag is not English. The second stage removes transcripts tagged as “mostly uppercase” or “mostly lowercase.” The third stage detects exact line repetition within each transcript, a pattern noted as common in synthetic captions. The fourth stage compares a manual transcript with a transcript generated by a strong ASR model, then removes any audio-text pair with document-level WER $> 0.5$ or segment-level WER $> 0.7$ (Ngo et al., 28 Aug 2025).

Two additional controls address contamination and redundancy. Fuzzy transcript deduplication via 5-gram MinHash at a Jaccard $\sim 75\%$ threshold removes 505K duplicate transcripts (3% of 17M), and n-gram based decontamination against evaluation sets removes 286 transcripts. After these steps, OLMoASR-Mix contains approximately 1,000,000 hours of high-quality English audio/transcript pairs. The reported quality change is substantial: short-form held-out WER improves from 37.2% to 20.7%, and long-form benchmarks show similar relative gains ($\sim 16$ pp WER improvement) (Ngo et al., 28 Aug 2025).

3. Model family, architecture, and training protocol

All OLMoASR models adopt OpenAI Whisper’s encoder–decoder Transformer backbone, with two explicit modifications: FlashAttention replaces standard attention for speed and memory efficiency, and the implementation integrates combined causal and padding masks to enable efficient batched training (Ngo et al., 28 Aug 2025). The study states that no new attention or position-encoding scheme is introduced beyond FlashAttention, and that the tokenizer and vocabulary remain unchanged.

The suite spans the following English-only scales: tiny.en (39M), base.en (74M), small.en (244M), medium.en (769M), large.en (1,550M), and large.en-v2 (1,550M). The paper specifies that large.en-v2 is re-trained on larger data sample, 680 K hr/epoch. For the small.en configuration, the reported pseudo-architecture is identical to Whisper small.en: 12 encoder + 12 decoder layers, model dim = 512, and 8 heads (Ngo et al., 28 Aug 2025).

Training uses standard autoregressive token prediction with cross-entropy: $$\mathcal{L}_{CE} = -\sum_{t=1}^T \log\,p(y_t \mid x).$$ The optimization configuration is fixed across the suite: 524,288 total updates; AdamW with $\beta_1 = 0.9$, $\beta_2 = 0.98$, $\epsilon = 1\times 10^{-6}$, and weight decay = 0.1; linear warmup for 1,049 updates up to max $1.5\times 10^{-3}$ followed by linear decay to zero; and gradient clipping with max-norm = 1.0. Gaussian fan-in initialization is used. The reported precision and parallelism setup is FP16, DDP on 1× H100 node for tiny/base/small, and bfloat16 + activation checkpointing, FSDP on 2–4× H100 nodes for medium/large. The training pipeline applies no additional audio augmentations such as noise injection or speed perturbation; the models rely solely on the weakly labeled web-scale data (Ngo et al., 28 Aug 2025).

4. Benchmarking and recognition performance

Evaluation is split between short-form and long-form speech recognition. The short-form suite contains 14 sets: LibriSpeech[test-clean/other], TED-LIUM3, WSJ, CallHome, Switchboard, CommonVoice 5.1, Artie, CORAAL, CHiME-6, AMI-IHM, AMI-SDM, VoxPopuli.en, and Fleurs.en.us. The long-form suite contains 7 sets: TED-LIUM3, Meanwhile, Kincaid46, Rev16, Earnings-21, Earnings-22, and CORAAL (Ngo et al., 28 Aug 2025).

The paper reports greedy-decoding WER for short-form and beam-search WER for long-form, comparing OLMoASR with Whisper at matched scales.

Model	Short-form WER	Long-form WER
tiny.en (39M)	20.5 vs. 20.1	15.6 vs. 16.6
base.en (74M)	16.6 vs. 16.9	12.9 vs. 13.2
small.en (244M)	13.8 vs. 13.7	11.5 vs. 11.2
medium.en (769M)	12.8 vs. 12.4	11.0 vs. 10.5
large.en (1.55B)	13.0 vs. 12.2*	11.4 vs. 10.4*
large.en-v2 (1.55B)	12.6 vs. 12.2*	11.5 vs. 10.4*

A representative result highlighted in the abstract is that OLMoASR-medium.en achieves 12.8% short-form WER and 11.0% long-form WER, compared with Whisper-medium.en’s 12.4% and 10.5% at equivalent parameter count. The scale analysis reports that up to 244M parameters, OLMoASR matches or slightly beats Whisper. At 769M, the reported gap is 0.4 pp on short-form and 0.5 pp on long-form WER. The paper further states that scaling to 1.5B yields diminishing returns on zero-shot English-only WER, likely due to lack of multilingual data or hyperparameter tuning (Ngo et al., 28 Aug 2025).

5. Ablations, robustness, and recurring interpretive issues

The ablation studies are designed to isolate the effects of data scale, data composition, and filtering quality. In the data-scale ablation, a 74M-parameter OLMoASR model is trained on 50K, 220K, 440K, 680K, 880K, and 1M hours sampled from OLMoASR-Mix. The reported short-form WER drops from $\approx 23.7\%$ to 22.8% from 50K to 220K hours, then plateaus up to 880K hr, and improves by 1.5 pp at the full 1M hours. Long-form WER gains are reported as < 1 pp across scaling, which the paper interprets as moderate saturation for a 74M-param model (Ngo et al., 28 Aug 2025).

The data composition ablations compare OLMoASR-Mix with other available corpora under similar compute. Training on OWSM-Eng ($> 0.7$0 K hr) yields 1–2 pp worse WER and weaker out-of-distribution robustness than the same compute on OLMoASR-Mix. Against YODAS (190 K hr multilingual YouTube data), OLMoASR-Mix outperforms across all tiny→small scales by up to 2.7 pp WER. These comparisons are central to the paper’s argument that scale alone is insufficient; curation and composition matter materially (Ngo et al., 28 Aug 2025).

The robustness analysis distinguishes in-distribution and out-of-distribution behavior. Under Effective Robustness, OLMoASR, evaluated zero-shot, has higher in-distribution WER on LS-clean than supervised LibriSpeech models but a substantial positive gap on OOD sets including AMI, CHiME-6, CORAAL, and CommonVoice. Under Relative Robustness, OLMoASR-Mix shows consistently better robustness than a “no-quality-filtering” baseline, supporting the claim that the filtering pipeline improves generalization to unseen acoustic domains.

A recurring misunderstanding is that OLMoASR demonstrates an architectural breakthrough. The evidence presented does not support that reading. The models intentionally retain Whisper’s backbone and tokenizer. The reported novelty lies in the 3M-hour pool, the text heuristic filters, and the controlled demonstration that curation can materially narrow or match performance gaps with closed-source systems (Ngo et al., 28 Aug 2025).

6. Open release and position within open ASR research

The OLMoASR project is intended for public release under an academic-research license. The specified assets are the dataset IDs https://huggingface.co/datasets/allenai/OLMoASR-Pool and …/OLMoASR-Mix, model checkpoints at https://huggingface.co/allenai/OLMoASR, and the code repository https://github.com/allenai/OLMoASR. The release includes full filtering parameters, dedup/decontam code, training scripts, and evaluation pipelines (Ngo et al., 28 Aug 2025).

Within open ASR research, OLMoASR occupies a distinct position. SALM studies a frozen GPT-style LLM augmented with an audio encoder, modality adapter, and LoRA layers for ASR and AST, and emphasizes zero-shot in-context learning such as keyword boosting (Chen et al., 2023). HLoRA studies a Language-agnostic Hierarchical LoRA-MoE integrated into an mHuBERT-CTC model, using LID-posterior-driven LoRA routing for single-pass multilingual decoding (Zheng et al., 2 Jan 2026). OLMoASR, by contrast, preserves a Whisper-like encoder–decoder formulation and concentrates on the interplay between web-scale weak supervision, heuristic filtering, and robust zero-shot English recognition. This suggests that its main contribution is infrastructural and empirical: open data, open checkpoints, and a controlled account of how curation changes ASR behavior.

The acronym also benefits from disambiguation. In speech research, OLMoASR refers to the open-model, open-data ASR project described above (Ngo et al., 28 Aug 2025). An unrelated wireless-communications summary uses OLMoASR for an OAM-Mode-Assignment-and-Secrecy-Rate framework in RIS-assisted orbital-angular-momentum secure communications (Wang et al., 2024).