DCLM-Pool: Standardized Data Pool for LMs

• DCLM-Pool is a standardized, large-scale web-text corpus culled from Common Crawl data and processed with advanced extraction methods.
• It partitions data into scale-specific sub-pools, facilitating controlled, apples-to-apples comparisons of filtering, deduplication, and data mixing techniques.
• Its full metadata preservation and rigorous quality filtering enhance reproducibility and quantitative evaluation in language model pretraining.

DCLM-Pool denotes the principal data pool underlying the DataComp for LLMs (DCLM) benchmark, a controlled testbed for large-scale web-text dataset curation and downstream analysis in LLM pretraining. DCLM-Pool provides an unprecedentedly large, transparent, and uniformly processed corpus serving as the canonical upstream source for all DCLM filtering and mixing track experiments. Its design establishes the empirical basis for direct comparison of curation strategies at multiple compute and scale regimes, thus enabling quantifiable progress in training-set engineering for generative LLMs (Li et al., 2024).

1. Origin, Motivation, and Design Rationale

DCLM-Pool was created to provide a publicly available, unfiltered, and maximally comprehensive web-text corpus, addressing the need for a standardized, experiment-ready playground for data-centric research on LLMs. The corpus is constructed from all Common Crawl WARC dumps through 2022, with text extraction performed by the “resiliparse” HTML parser. This approach yields improved text quality and throughput compared to the standard WET-derived pipelines, providing a uniform basis for downstream evaluation and data curation across the research community (Li et al., 2024).

2. Corpus Scale and Partitioning

DCLM-Pool encompasses:

• 200 billion documents,
• 370 terabytes of gzip-compressed HTML,
• $T_{\rm pool} = 240 \times 10^{12}$ GPT-NeoX tokens.

To facilitate experiments at various compute budgets, the DCLM framework partitions DCLM-Pool into five sub-pools of increasing size, each aligned to a specific competition scale. The partitioning is as follows:

Sub-pool Name	Token Budget ($T_s$) [tokens]	Model Scale
400M–1×	$0.469\times 10^{12}$	400M
1B–1×	$1.64\times 10^{12}$	1B
1B–5×	$8.20\times 10^{12}$	1B (5× tokens)
7B–1×	$7.85\times 10^{12}$	7B
7B–2×	$15.7\times 10^{12}$	7B (2× tokens)

This granularity enables rigorous, apples-to-apples evaluation of data pipelines and model recipes under precisely matched data constraints (Li et al., 2024).

3. Data Provenance, Metadata, and Transparency

Each entry in DCLM-Pool maintains a one-to-one mapping to the original WARC file and snapshot, providing:

• All original WARC headers (such as WARC-Date, WARC-Target-URI, WARC-IP-Address, Content-Type, Content-Digest),
• The raw (resiliparse-extracted) text field,
• URL, and
• Complete document provenance.

Uniform tokenization is performed via the GPT-NeoX tokenizer (50,000 vocabulary), and all subsets are globally shuffled. This full metadata preservation and traceability guarantee that any document can be unambiguously audited, filtered, or re-extracted in future studies (Li et al., 2024).

4. Preprocessing, Deduplication, and Quality Filtering

DCLM-Pool is processed via a modular, standardized pipeline comprising at minimum:

• Text extraction (resiliparse), yielding on average $\sim1{,}330$ tokens per document compared to $\sim2{,}800$ for standard WET pipelines,
• Heuristic filtering, reproducing RefinedWeb's RW filters to remove boilerplate, non-English and low-entropy documents,
• Deduplication using either MinHash + suffix-array as in RefinedWeb, or a Bloom Filter-based “Big Friendly Filter” (BFF), capable of eliminating exact and near-duplicates at both document and paragraph levels. With typical BFF parameters (false-positive rate $\epsilon=0.01$, $T_s$0-gram length $T_s$1, similarity threshold $T_s$2, 100 shards), up to $T_s$3 of tokens can be removed while preserving corpus diversity.

Model-based quality filtering is essential to state-of-the-art results. The principal filter is a fastText classifier trained to discriminate “high-quality” (OH-2.5 + ELI5) instruction-style text from random web pages. Documents scoring above the 90th percentile are retained. This learned filter is empirically shown to increase accuracy by $T_s$4 percentage points at the 7B–1× scale. Other filtering methods tested, such as perplexity scores or embedding-based classifiers, yielded lower improvements (Li et al., 2024).

5. Curation Tracks and Competitive Use Cases

DCLM defines two principal tracks:

• Filtering track: Participants receive a scale-specific, pre-shuffled sub-pool and are tasked to select exactly the required token budget using custom curation approaches. This isolates the effect of data pipeline changes from model, compute, or hyperparameter variability, supporting systematic exploration of deduplication, filtering, and data mixing strategies.
• Mixing track: Participants may augment DCLM-Pool data with arbitrarily chosen corpora (e.g., GitHub, Wikipedia). Results indicate that mixing external data improves weaker Common Crawl subsets but degrades the already strong DCLM-baseline when polluted with “extras” (e.g., RedPajama-CC), demonstrating the nontrivial impact of source selection even for large LMs (Li et al., 2024).

Across over 400 experiments, DCLM-Pool serves as the stable upstream baseline—directly enabling scaling studies, ablation experiments, and iterative small-to-large model recipe transferability assessments.

6. Downstream Impacts and Key Empirical Observations

The standardized, transparently processed DCLM-Pool has direct, empirically measured effects on model performance:

• FastText-based model filtering increases Core evaluation scores beyond perplexity- or LLM-based filtering at 7B scale.
• Deduplication (either BFF or MinHash) removes $T_s$5–$T_s$6 of tokens while maintaining or increasing Core scores.
• Quality of upstream data decisively influences downstream model accuracy, with DCLM-baseline 7B models achieving $T_s$7 5-shot MMLU at $T_s$8 tokens, a $T_s$9 percentage point improvement over MAP-Neo and near parity with Mistral-7B and Llama 3 8B despite an order of magnitude lower compute requirement.
• Sankey diagrams illustrate substantial data attrition: from $0.469\times 10^{12}$0 billion initial documents, $0.469\times 10^{12}$1 billion survive extraction, $0.469\times 10^{12}$2 billion pass deduplication, and $0.469\times 10^{12}$3 billion remain after model-based filtering, yielding $0.469\times 10^{12}$4 trillion training tokens for the baseline (Li et al., 2024).

7. Research Significance and Benchmarking Implications

DCLM-Pool, by fixing the upstream source and enforcing end-to-end pipeline transparency, enables for the first time controlled, interpretable, and reproducible evaluation of data-centric LLM advances at massive scale. This infrastructure exposes the direct performance impact of each preprocessing or sampling decision and provides an extensible template for future community benchmarks. The structure and scale of DCLM-Pool support generalization to new domains, finer-grained ablation, and more robust LLM research by separating data effects from architectural or optimization variables (Li et al., 2024).