---
title: 'DCLM-Pool: Standardized Data Pool for LMs'
url: https://www.emergentmind.com/topics/dclm-pool
type: topic
---

# DCLM-Pool: Standardized Data Pool for LMs

DCLM-Pool denotes the principal data pool underlying the DataComp for Language Models (DCLM) benchmark, a controlled testbed for large-scale web-text dataset curation and downstream analysis in language model pretraining. DCLM-Pool provides an unprecedentedly large, transparent, and uniformly processed corpus serving as the canonical upstream source for all DCLM filtering and mixing track experiments. Its design establishes the empirical basis for direct comparison of curation strategies at multiple compute and scale regimes, thus enabling quantifiable progress in training-set engineering for generative language models [2406.11794].

## 1. Origin, Motivation, and Design Rationale

DCLM-Pool was created to provide a publicly available, unfiltered, and maximally comprehensive web-text corpus, addressing the need for a standardized, experiment-ready playground for data-centric research on large language models. The corpus is constructed from all Common Crawl WARC dumps through 2022, with text extraction performed by the “resiliparse” HTML parser. This approach yields improved text quality and throughput compared to the standard WET-derived pipelines, providing a uniform basis for downstream evaluation and data curation across the research community [2406.11794].

## 2. Corpus Scale and Partitioning

DCLM-Pool encompasses:
- 200 billion documents,
- 370 terabytes of gzip-compressed HTML,
- $T_{\rm pool} = 240 \times 10^{12}$ GPT-NeoX tokens.

To facilitate experiments at various compute budgets, the DCLM framework partitions DCLM-Pool into five sub-pools of increasing size, each aligned to a specific competition scale. The partitioning is as follows:

| Sub-pool Name    | Token Budget ($T_s$) [tokens]      | Model Scale    |
|------------------|------------------------------------|----------------|
| 400M–1×          | $0.469\times 10^{12}$              | 400M           |
| 1B–1×            | $1.64\times 10^{12}$               | 1B             |
| 1B–5×            | $8.20\times 10^{12}$               | 1B (5× tokens) |
| 7B–1×            | $7.85\times 10^{12}$               | 7B             |
| 7B–2×            | $15.7\times 10^{12}$               | 7B (2× tokens) |

This granularity enables rigorous, apples-to-apples evaluation of data pipelines and model recipes under precisely matched data constraints [2406.11794].

## 3. Data Provenance, Metadata, and Transparency

Each entry in DCLM-Pool maintains a one-to-one mapping to the original WARC file and snapshot, providing:
- All original WARC headers (such as WARC-Date, WARC-Target-URI, WARC-IP-Address, Content-Type, Content-Digest),
- The raw (resiliparse-extracted) text field,
- URL, and
- Complete document provenance.

Uniform tokenization is performed via the GPT-NeoX tokenizer (50,000 vocabulary), and all subsets are globally shuffled. This full metadata preservation and traceability guarantee that any document can be unambiguously audited, filtered, or re-extracted in future studies [2406.11794].

## 4. Preprocessing, Deduplication, and Quality Filtering

DCLM-Pool is processed via a modular, standardized pipeline comprising at minimum:
- Text extraction (resiliparse), yielding on average $\sim1{,}330$ tokens per document compared to $\sim2{,}800$ for standard WET pipelines,
- Heuristic filtering, reproducing RefinedWeb's RW filters to remove boilerplate, non-English and low-entropy documents,
- Deduplication using either MinHash + suffix-array as in RefinedWeb, or a Bloom Filter-based “Big Friendly Filter” (BFF), capable of eliminating exact and near-duplicates at both document and paragraph levels. With typical BFF parameters (false-positive rate $\epsilon=0.01$, $n$-gram length $n=13$, similarity threshold $0.8$, 100 shards), up to $40\%$ of tokens can be removed while preserving corpus diversity.

Model-based quality filtering is essential to state-of-the-art results. The principal filter is a fastText classifier trained to discriminate “high-quality” (OH-2.5 + ELI5) instruction-style text from random web pages. Documents scoring above the 90th percentile are retained. This learned filter is empirically shown to increase accuracy by $\sim4$ percentage points at the 7B–1× scale. Other filtering methods tested, such as perplexity scores or embedding-based classifiers, yielded lower improvements [2406.11794].

## 5. Curation Tracks and Competitive Use Cases

DCLM defines two principal tracks:
- **Filtering track:** Participants receive a scale-specific, pre-shuffled sub-pool and are tasked to select exactly the required token budget using custom curation approaches. This isolates the effect of data pipeline changes from model, compute, or hyperparameter variability, supporting systematic exploration of deduplication, filtering, and data mixing strategies.
- **Mixing track:** Participants may augment DCLM-Pool data with arbitrarily chosen corpora (e.g., GitHub, Wikipedia). Results indicate that mixing external data improves weaker Common Crawl subsets but degrades the already strong DCLM-baseline when polluted with “extras” (e.g., RedPajama-CC), demonstrating the nontrivial impact of source selection even for large LMs [2406.11794].

Across over 400 experiments, DCLM-Pool serves as the stable upstream baseline—directly enabling scaling studies, ablation experiments, and iterative small-to-large model recipe transferability assessments.

## 6. Downstream Impacts and Key Empirical Observations

The standardized, transparently processed DCLM-Pool has direct, empirically measured effects on model performance:
- FastText-based model filtering increases Core evaluation scores beyond perplexity- or LLM-based filtering at 7B scale.
- Deduplication (either BFF or MinHash) removes $30$–$40\%$ of tokens while maintaining or increasing Core scores.
- Quality of upstream data decisively influences downstream model accuracy, with DCLM-baseline 7B models achieving $64\%$ 5-shot MMLU at $2.6T$ tokens, a $6.6$ percentage point improvement over MAP-Neo and near parity with Mistral-7B and Llama 3 8B despite an order of magnitude lower compute requirement.
- Sankey diagrams illustrate substantial data attrition: from $\sim200$ billion initial documents, $\sim160$ billion survive extraction, $\sim90$ billion pass deduplication, and $\sim20$ billion remain after model-based filtering, yielding $\sim2$ trillion training tokens for the baseline [2406.11794].

## 7. Research Significance and Benchmarking Implications

DCLM-Pool, by fixing the upstream source and enforcing end-to-end pipeline transparency, enables for the first time controlled, interpretable, and reproducible evaluation of data-centric language model advances at massive scale. This infrastructure exposes the direct performance impact of each preprocessing or sampling decision and provides an extensible template for future community benchmarks. The structure and scale of DCLM-Pool support generalization to new domains, finer-grained ablation, and more robust language model research by separating data effects from architectural or optimization variables [2406.11794].

Source: https://www.emergentmind.com/topics/dclm-pool