---
title: 'OLMoASR: Open Data ASR with Zero-Shot Robustness'
url: https://www.emergentmind.com/topics/olmoasr
type: topic
---

# OLMoASR: Open Data ASR with Zero-Shot Robustness

OLMoASR is an open speech recognition project centered on the proposition that data scale and data quality are first-order determinants of robust zero-shot ASR. It comprises the large-scale dataset **OLMoASR-Pool**, the curated subset **OLMoASR-Mix**, and a suite of English-only encoder–decoder Transformer models trained on that corpus. In the reported study, OLMoASR-Pool contains **3M hours of English audio** and **17M transcripts**; the curation pipeline yields **OLMoASR-Mix** with approximately **1M hours of high-quality audio-transcript pairs**; and the resulting models span **39M** to **1.5B** parameters while achieving **comparable average performance to OpenAI’s Whisper on short and long-form speech recognition benchmarks** [2508.20869].

## 1. Project scope and naming

OLMoASR denotes more than a model family. In the underlying work, it names a data-and-model program composed of **OLMoASR-Pool**, **OLMoASR-Mix**, and the **OLMoASR-Mix suite of models**, together with filtering, training, and evaluation code [2508.20869]. A common simplification is to treat OLMoASR as a new neural architecture; the paper instead presents it as a controlled investigation of robust ASR under web-scale weak supervision, with the architectural baseline intentionally kept close to Whisper.

The project is explicitly **English-only**. OLMoASR-Pool is described as a collection of English audio with associated transcripts, and the released model suite is organized as **tiny.en**, **base.en**, **small.en**, **medium.en**, **large.en**, and **large.en-v2**. This English-only scope is consequential for interpreting the scaling results: the work reports strong zero-shot English robustness, but it does not claim multilingual coverage.

The project’s central thesis is that **targeted text heuristic filters** can convert heterogeneous web data into a training mixture that materially improves zero-shot ASR. The paper’s summary states that a controlled experiment, **holding architecture and optimizer constant**, confirms the central role of data curation. This suggests that OLMoASR should be understood as a data-centric research program rather than as an attempt to supersede Whisper through architectural novelty alone [2508.20869].

## 2. Corpus construction and curation pipeline

OLMoASR-Pool is built from **publicly accessible audio+text pairs scraped from web video and audio platforms (e.g., YouTube), audiobooks, podcasts, lectures, etc.** The transcripts include both **manually uploaded subtitles** and **(unlabeled) automatic captions**, and the corpus is characterized by **heterogenous audio quality**, diverse speakers, accents, and domains. The final curated set, OLMoASR-Mix, is drawn from this pool after a four-stage filtering process intended to remove low-quality or mis-transcribed examples while preserving breadth of acoustic and topical coverage [2508.20869].

The curation stages are reported with both retained hours and short-form WER on a held-out average over 14 benchmarks.

| Stage | Remaining data | Held-out short-form WER |
|---|---:|---:|
| No quality filters | 3M hr | 37.2% |
| Audio/Text Language Alignment | 2.4M hr | — |
| Casing Filter | 1.37M hr | 32.4% |
| Repeating-Lines Filter | 1.21M hr | 22.7% |
| Manual–Machine WER Filter | 908,923 hr | 20.7% |

The first stage uses **VoxLingua107** for spoken-language identification on audio and **pycld2** for language detection on transcript text, removing any pair for which either tag is not English. The second stage removes transcripts tagged as **“mostly uppercase”** or **“mostly lowercase.”** The third stage detects **exact line repetition within each transcript**, a pattern noted as common in synthetic captions. The fourth stage compares a **manual** transcript with a transcript generated by a **strong ASR model**, then removes any audio-text pair with **document-level WER \(> 0.5\)** or **segment-level WER \(> 0.7\)** [2508.20869].

Two additional controls address contamination and redundancy. **Fuzzy transcript deduplication via 5-gram MinHash** at a **Jaccard \(\sim 75\%\)** threshold removes **505K duplicate transcripts (3% of 17M)**, and **n-gram based decontamination against evaluation sets** removes **286 transcripts**. After these steps, OLMoASR-Mix contains approximately **1,000,000 hours** of high-quality English audio/transcript pairs. The reported quality change is substantial: short-form held-out WER improves from **37.2%** to **20.7%**, and long-form benchmarks show **similar relative gains (\(\sim 16\) pp WER improvement)** [2508.20869].

## 3. Model family, architecture, and training protocol

All OLMoASR models adopt **OpenAI Whisper’s encoder–decoder Transformer backbone**, with two explicit modifications: **FlashAttention** replaces standard attention for speed and memory efficiency, and the implementation integrates **combined causal and padding masks** to enable efficient batched training [2508.20869]. The study states that **no new attention or position-encoding scheme is introduced beyond FlashAttention**, and that the **tokenizer and vocabulary remain unchanged**.

The suite spans the following English-only scales: **tiny.en (39M)**, **base.en (74M)**, **small.en (244M)**, **medium.en (769M)**, **large.en (1,550M)**, and **large.en-v2 (1,550M)**. The paper specifies that **large.en-v2** is **re-trained on larger data sample, 680 K hr/epoch**. For the **small.en** configuration, the reported pseudo-architecture is identical to Whisper small.en: **12 encoder + 12 decoder layers**, **model dim = 512**, and **8 heads** [2508.20869].

Training uses standard autoregressive token prediction with cross-entropy:
\[
\mathcal{L}_{CE} = -\sum_{t=1}^T \log\,p(y_t \mid x).
\]
The optimization configuration is fixed across the suite: **524,288 total updates**; **AdamW** with \(\beta_1 = 0.9\), \(\beta_2 = 0.98\), \(\epsilon = 1\times 10^{-6}\), and **weight decay = 0.1**; **linear warmup for 1,049 updates up to max \(1.5\times 10^{-3}\)** followed by linear decay to zero; and **gradient clipping with max-norm = 1.0**. **Gaussian fan-in** initialization is used. The reported precision and parallelism setup is **FP16, DDP on 1× H100 node** for **tiny/base/small**, and **bfloat16 + activation checkpointing, FSDP on 2–4× H100 nodes** for **medium/large**. The training pipeline applies **no additional audio augmentations** such as noise injection or speed perturbation; the models rely solely on the weakly labeled web-scale data [2508.20869].

## 4. Benchmarking and recognition performance

Evaluation is split between **short-form** and **long-form** speech recognition. The short-form suite contains **14 sets**: **LibriSpeech[test-clean/other]**, **TED-LIUM3**, **WSJ**, **CallHome**, **Switchboard**, **CommonVoice 5.1**, **Artie**, **CORAAL**, **CHiME-6**, **AMI-IHM**, **AMI-SDM**, **VoxPopuli.en**, and **Fleurs.en.us**. The long-form suite contains **7 sets**: **TED-LIUM3**, **Meanwhile**, **Kincaid46**, **Rev16**, **Earnings-21**, **Earnings-22**, and **CORAAL** [2508.20869].

The paper reports greedy-decoding WER for short-form and beam-search WER for long-form, comparing OLMoASR with Whisper at matched scales.

| Model | Short-form WER | Long-form WER |
|---|---:|---:|
| tiny.en (39M) | 20.5 vs. 20.1 | 15.6 vs. 16.6 |
| base.en (74M) | 16.6 vs. 16.9 | 12.9 vs. 13.2 |
| small.en (244M) | 13.8 vs. 13.7 | 11.5 vs. 11.2 |
| medium.en (769M) | 12.8 vs. 12.4 | 11.0 vs. 10.5 |
| large.en (1.55B) | 13.0 vs. 12.2* | 11.4 vs. 10.4* |
| large.en-v2 (1.55B) | 12.6 vs. 12.2* | 11.5 vs. 10.4* |

A representative result highlighted in the abstract is that **OLMoASR-medium.en** achieves **12.8%** short-form WER and **11.0%** long-form WER, compared with **Whisper-medium.en’s 12.4% and 10.5%** at equivalent parameter count. The scale analysis reports that **up to 244M parameters, OLMoASR matches or slightly beats Whisper**. At **769M**, the reported gap is **0.4 pp** on short-form and **0.5 pp** on long-form WER. The paper further states that scaling to **1.5B** yields **diminishing returns on zero-shot English-only WER**, **likely due to lack of multilingual data or hyperparameter tuning** [2508.20869].

## 5. Ablations, robustness, and recurring interpretive issues

The ablation studies are designed to isolate the effects of data scale, data composition, and filtering quality. In the **data-scale ablation**, a **74M-parameter** OLMoASR model is trained on **50K, 220K, 440K, 680K, 880K, and 1M hours** sampled from OLMoASR-Mix. The reported short-form WER drops from **\(\approx 23.7\%\)** to **22.8%** from **50K** to **220K** hours, then **plateaus up to 880K hr**, and improves by **1.5 pp** at the full **1M** hours. Long-form WER gains are reported as **< 1 pp across scaling**, which the paper interprets as **moderate saturation for a 74M-param model** [2508.20869].

The **data composition ablations** compare OLMoASR-Mix with other available corpora under similar compute. Training on **OWSM-Eng (\(\sim 113\) K hr)** yields **1–2 pp worse WER** and weaker out-of-distribution robustness than the same compute on OLMoASR-Mix. Against **YODAS (190 K hr multilingual YouTube data)**, OLMoASR-Mix **outperforms across all tiny→small scales by up to 2.7 pp WER**. These comparisons are central to the paper’s argument that scale alone is insufficient; curation and composition matter materially [2508.20869].

The robustness analysis distinguishes in-distribution and out-of-distribution behavior. Under **Effective Robustness**, OLMoASR, evaluated zero-shot, has **higher in-distribution WER on LS-clean** than supervised LibriSpeech models but a **substantial positive gap on OOD sets** including **AMI**, **CHiME-6**, **CORAAL**, and **CommonVoice**. Under **Relative Robustness**, OLMoASR-Mix shows **consistently better robustness than a “no-quality-filtering” baseline**, supporting the claim that the filtering pipeline improves generalization to unseen acoustic domains.

A recurring misunderstanding is that OLMoASR demonstrates an architectural breakthrough. The evidence presented does not support that reading. The models intentionally retain Whisper’s backbone and tokenizer. The reported novelty lies in the **3M-hour pool**, the **text heuristic filters**, and the controlled demonstration that curation can materially narrow or match performance gaps with closed-source systems [2508.20869].

## 6. Open release and position within open ASR research

The OLMoASR project is intended for public release under an **academic-research license**. The specified assets are the dataset IDs **https://huggingface.co/datasets/allenai/OLMoASR-Pool** and **…/OLMoASR-Mix**, model checkpoints at **https://huggingface.co/allenai/OLMoASR**, and the code repository **https://github.com/allenai/OLMoASR**. The release includes **full filtering parameters, dedup/decontam code, training scripts, and evaluation pipelines** [2508.20869].

Within open ASR research, OLMoASR occupies a distinct position. **SALM** studies a **frozen GPT-style LLM** augmented with an **audio encoder**, **modality adapter**, and **LoRA layers** for **ASR and AST**, and emphasizes **zero-shot in-context learning** such as keyword boosting [2310.09424]. **HLoRA** studies a **Language-agnostic Hierarchical LoRA-MoE** integrated into an **mHuBERT-CTC** model, using **LID-posterior-driven LoRA routing** for **single-pass multilingual decoding** [2601.00557]. OLMoASR, by contrast, preserves a Whisper-like encoder–decoder formulation and concentrates on the interplay between **web-scale weak supervision**, **heuristic filtering**, and **robust zero-shot English recognition**. This suggests that its main contribution is infrastructural and empirical: open data, open checkpoints, and a controlled account of how curation changes ASR behavior.

The acronym also benefits from disambiguation. In speech research, **OLMoASR** refers to the open-model, open-data ASR project described above [2508.20869]. An unrelated wireless-communications summary uses **OLMoASR** for an **OAM-Mode-Assignment-and-Secrecy-Rate** framework in **RIS-assisted orbital-angular-momentum secure communications** [2407.11322].

Source: https://www.emergentmind.com/topics/olmoasr