OLMoASR-Mix: Curated Web-Scale ASR Dataset

• The paper demonstrates that transcript quality heuristics significantly affect zero-shot ASR performance beyond mere data scale.
• The multi-stage curation pipeline—featuring segmentation, language alignment, heuristic filtering, WER comparison, and deduplication—ensures high data fidelity.
• OLMoASR-Mix serves as both a training resource and benchmark for evaluating cross-domain generalization in robust speech recognition.

Searching arXiv for the specified paper and closely related work to ground the article. OLMoASR-Mix is a curated corpus of roughly 1 million hours of English audio–transcript pairs derived from OLMoASR-Pool, a weakly supervised web harvest containing 3 million hours of English audio and 17 million transcripts. Within the OLMoASR program, OLMoASR-Mix is both a dataset and an experimental vehicle: it is produced by a multi-stage curation pipeline designed to remove low-quality or mistranscribed pairs, and it serves as the training substrate for a family of zero-shot speech recognition models ranging from 39M to 1.55B parameters (Ngo et al., 28 Aug 2025). The central finding associated with the dataset is that transcript quality heuristics are not secondary to scale; rather, they materially affect out-of-distribution generalization and competitive zero-shot word error rate (WER).

1. Position within the OLMoASR project

OLMoASR-Mix occupies the intermediate layer between a raw web-scale pool and trained automatic speech recognition (ASR) models. The source collection, OLMoASR-Pool, is described as a raw “weakly-supervised” web harvest of 3 million hours of English audio paired with 17 million transcripts. From that source, the curation pipeline distills a higher-fidelity subset intended for robust zero-shot ASR. The resulting dataset is then used to train the OLMoASR-Mix suite of models, denoted tiny.en, base.en, small.en, medium.en, and large.en (Ngo et al., 28 Aug 2025).

The paper’s framing is explicitly empirical rather than architectural. It studies how far a largely Whisper-like sequence-to-sequence Transformer can be pushed when trained on a carefully filtered public-web corpus. This places OLMoASR-Mix at the intersection of dataset engineering, weak supervision, and robustness evaluation. A plausible implication is that the dataset is meant not only as a resource for immediate model training, but also as a benchmarkable object for research on corpus curation strategies.

2. Curation pipeline and dataset formation

The construction of OLMoASR-Mix proceeds through segmentation, language alignment, text-heuristic filtering, manual-machine WER comparison, and finally deduplication and decontamination (Ngo et al., 28 Aug 2025).

Stage	Hours retained	Noted effect
Segmentation	2.4 M h	VAD, max $\approx 30$ s
Language alignment	2.01 M h	VoxLingua107 + PyCLD2, English only
Text heuristics	$\sim 1.14$ M h	best trade-off from repeats + upper-case filtering
Manual-machine WER comparison	0.91 M h	remove pairs above WER thresholds
Deduplication and decontamination	roughly 1 M h final	MinHash and TED-LIUM3 overlap removal

Segmentation reduces the source audio from 3 M to 2.4 M hours by cutting long files into shorter segments using voice-activity detection, with a maximum duration of approximately 30 seconds while preserving the original transcript alignments. Language alignment then reduces the corpus from 2.4 M to 2.01 M hours by applying VoxLingua107 to the audio and PyCLD2 to the transcript text, removing any pair for which either side is not tagged as English.

The transcript-quality stage applies three types of text heuristics, with impact measured on short-form WER, defined as the average WER across 14 held-out short-form benchmarks. Removing all-upper or all-lower case transcripts reduces the aligned pool from 2.01 M to 1.37 M hours, leaving 68% of the data and lowering short-form WER by 4.8 percentage points. Repeating-lines removal drops any segment whose transcript contains consecutive identical lines, reducing the corpus to 1.21 M hours, or 60.1% of the aligned pool, and lowering WER by 14.4 percentage points. Among tested combinations, filtering on repeats plus upper-case transcripts gives the best trade-off, leaving 56.7% of the aligned pool and yielding a short-form WER of 21.9%.

A further quality screen compares human transcripts with machine-generated transcripts from a strong ASR system. The document-level and segment-level WER are computed as

$\mathrm{WER}=\frac{s+i+d}{n},$

where $s$, $i$, and $d$ are substitutions, insertions, and deletions, and $n$ is the reference length. Any audio–text pair with document-level WER greater than 0.5 or segment-level WER greater than 0.7 is rejected. This reduces the corpus from 1.14 M to 0.91 M hours, or 45.2% of the aligned pool, and lowers WER by 16.5 percentage points.

Deduplication uses MinHash with 5-grams, 112 hashes, and Jaccard $\approx 75\%$, removing 3% of transcripts. Decontamination removes any training transcript sharing a 10-gram with TED-LIUM3 evaluation data; only 286 transcripts are removed. The final result is described as roughly 1 million hours of high-fidelity English audio–transcript pairs harvested from the public web.

3. Corpus characteristics and implied coverage

OLMoASR-Mix consists entirely of English after language alignment. The source summary describes a domain mixture including audiobooks, lectures, and vlog or podcast clips, with segment durations from 5 seconds to 30 seconds and a median around 12 seconds, reflecting the VAD-driven segmentation procedure (Ngo et al., 28 Aug 2025).

The paper also notes language variety spanning American, British, Australian, Indian, and other accents, but qualifies this as being implied by the original web sources rather than established by a separate annotation layer. This suggests that the corpus is intended to capture naturally occurring accent and domain variation through source diversity rather than through balanced sampling or controlled labeling. That distinction matters methodologically: OLMoASR-Mix is a web-derived robustness resource, not an explicitly stratified sociophonetic corpus.

A common simplification would be to treat OLMoASR-Mix as merely a larger English ASR dataset. The reported construction argues against that reading. Its defining property is not scale alone, but the combination of web-scale collection with selective removal of low-value transcript noise. In that sense, the dataset is closer to a filtered weak-supervision regime than to a conventional supervised corpus aggregation.

4. Model architectures and training regime

The OLMoASR-Mix dataset is used to train a family of sequence-to-sequence Transformer models based on the Whisper backbone, with two modifications: FlashAttention in every multi-head block for speed and memory efficiency, and unified causal plus padding masking for batch training (Ngo et al., 28 Aug 2025). The models share the same layer structure and differ only in depth and width.

The parameter scales are as follows. OLMoASR-tiny.en has 39 M parameters, with a 6-layer encoder and 6-layer decoder, $d_{\text{model}}=384$, $d_{ff}=1536$, and 6 heads. OLMoASR-base.en has 74 M parameters, also with 6 encoder and 6 decoder layers, $\sim 1.14$0, $\sim 1.14$1, and 8 heads. OLMoASR-small.en has 244 M parameters, with 12 encoder and 12 decoder layers, $\sim 1.14$2, $\sim 1.14$3, and 12 heads. OLMoASR-medium.en has 769 M parameters, with 24 encoder and 24 decoder layers, $\sim 1.14$4, $\sim 1.14$5, and 16 heads. OLMoASR-large.en has 1.55 B parameters, with 32 encoder and 32 decoder layers, $\sim 1.14$6, $\sim 1.14$7, and 16 heads.

Training uses a unified recipe across model sizes, varying only the parallel-training strategy. The setup specifies 524,288 total updates, batch size 512 tokens, 1,049 warmup steps, AdamW with $\sim 1.14$8, $\sim 1.14$9, $\mathrm{WER}=\frac{s+i+d}{n},$0, and weight decay 0.1, and a linear warmup followed by linear decay with peak learning rate $\mathrm{WER}=\frac{s+i+d}{n},$1. The loss is standard cross-entropy on decoder outputs,

$\mathrm{WER}=\frac{s+i+d}{n},$2

For precision and parallelism, tiny, base, and small use DDP with FP16 and dynamic loss scaling, whereas medium and large use FSDP with bfloat16 and activation checkpointing. No per-model hyperparameter tuning is performed.

This training design is significant because it narrows the interpretive space. The paper does not attribute performance gains to extensive architecture search or per-scale optimization. Instead, it isolates the interaction between a stable training recipe and a curated web-scale corpus.

5. Zero-shot evaluation and comparative performance

Evaluation is conducted in strict zero-shot mode on 14 short-form tasks and 7 long-form tasks (Ngo et al., 28 Aug 2025). The short-form suite includes examples such as LibriSpeech test-clean and test-other, TED-LIUM3, WSJ, CommonVoice, Switchboard, CallHome, AMI-IHM and SDM, CHiME-6, Artie, CORAAL, Fleurs, and VoxPopuli. The long-form suite includes TED-LIUM3 long, Meanwhile, Kincaid46, REV’16, Earnings21 and 22, and CORAAL. Decoding uses greedy search for short-form and beam search with temperature fallback for long-form.

The headline comparison is against OpenAI’s Whisper English-only models at matched scales. OLMoASR-tiny reports average WERs of 20.5 on short-form and 15.6 on long-form, compared with Whisper-tiny at 20.1 and 16.6. OLMoASR-base reports 16.6 and 12.9, compared with Whisper-base at 16.9 and 13.2. OLMoASR-small reports 13.8 and 11.5, compared with Whisper-small at 13.7 and 11.2. OLMoASR-medium reports 12.8 and 11.0, compared with Whisper-medium at 12.4 and 10.5. OLMoASR-large reports 13.0 and 11.4, compared with Whisper-large at 12.2 and 10.4.

The medium-scale comparison is especially emphasized: OLMoASR-medium.en attains 12.8% short-form WER and 11.0% long-form WER, against Whisper-medium.en at 12.4% and 10.5% at equivalent parameter count. The large model does not improve over medium in the reported averages, which is a useful reminder that scaling behavior is not strictly monotonic under a fixed recipe and fixed data mixture.

The paper also reports a direct curation ablation. Models trained on the fully curated OLMoASR-Mix outperform models trained on a version with no quality filtering beyond language alignment by 1 to 14 percentage points on both short- and long-form evaluation. This result grounds the broader claim that transcript quality heuristics are as critical as data scale for zero-shot ASR generalization.

6. Robustness claims, interpretation, and release context

Robustness analysis follows Taori et al. (2020) through two measures: effective robustness, defined as out-of-distribution WER gain relative to an in-distribution LibriSpeech baseline, and relative robustness, defined as the direct WER gap on out-of-distribution tasks versus supervised LibriSpeech models (Ngo et al., 28 Aug 2025). Under these measures, zero-shot OLMoASR models have higher LibriSpeech WER than fully supervised systems, yet outperform them by 10 to 20 percentage points on out-of-distribution sets including AMI, CHiME-6, CORAAL, and CommonVoice.

These results support a specific interpretation of OLMoASR-Mix. The dataset is not presented as a replacement for supervised in-domain optimization on benchmark speech. Rather, it is presented as a route to stronger cross-domain and cross-condition transfer through large, high-quality, paired web data. A plausible implication is that OLMoASR-Mix is especially relevant for robustness-oriented ASR research, where the central problem is not peak performance on a single standardized domain but degradation under distribution shift.

Several clarifications follow from the reported evidence. First, OLMoASR-Mix is English-only in the reported instantiation; multilinguality is mentioned only as a future direction. Second, the gains do not depend on a substantially novel model family, since the architecture remains largely Whisper-like. Third, the paper states that even at equal data volumes, web-scale OLMoASR-Mix outperforms academic corpus mixes such as OWSM-Eng by several WER points on held-out tasks, reinforcing the view that source diversity and transcript filtering are consequential variables.

The project is also framed as an open research resource. OLMoASR-Pool and OLMoASR-Mix ID lists are publicly released on HuggingFace; filtering, training, and evaluation code are released on GitHub; and model weights from tiny through large are released on HuggingFace under the allenai/OLMoASR namespace. This release strategy is consistent with the paper’s broader aim: to make data, models, and curation procedures available for further work on robust speech processing.