A Bitter Lesson for Data Filtering

Abstract: We investigate data filtering for large model pretraining via new scaling studies that target the high compute, data-scarce regime. In spite of an apparently common belief that filtering data to include only high-quality information is essential, our experiments suggest that with enough compute, the best data filter is no data filter. We find that sufficiently trained large parameter models not only tolerate low-quality and distractor data, but in fact benefit from nominally ``poor'' data.

• The paper demonstrates that unfiltered data, when used with sufficient compute and model capacity, achieves lower validation loss compared to filtered datasets.
• It employs extensive experiments across various scales and noise injections to reveal that large models can effectively leverage even low-quality data.
• The study derives scaling laws indicating that as compute increases, traditional data filtering becomes less beneficial, challenging established data curation practices.

The Limits of Data Filtering in Large-Scale LLM Pretraining

Introduction

This paper, "A Bitter Lesson for Data Filtering" (2605.19407), presents an extensive empirical and theoretical investigation into the necessity and efficacy of data filtering in large-scale LLM pretraining. Contrary to prevailing expert intuition and standard practice which typically emphasize stringent data filtering to maximize "data quality," the authors rigorously demonstrate that, in the high-compute regime, aggressive data filtering not only becomes unnecessary, but can even be suboptimal. In sufficiently large models trained for sufficient length, even data typically considered "low-quality" or "junk" may be beneficial or, at worst, not harmful.

Experimental Methodology and Setup

The study utilizes various randomly sampled subsets of the full pre-2023 Common Crawl (CC) corpus, scaling from 670 million to 10 billion tokens, and applies a variety of standard filtering pipelines (e.g., DCLM-Baseline, RefinedWeb). Model architectures range from 15M to 7B parameter dense transformers in the Llama-style, trained with standard hyperparameter tuning and model scaling protocols. The loss metric is primarily validation NLL on multiple benchmark datasets (C4 English portion, Fineweb-Edu, and Cosmopedia), with secondary consideration to downstream tasks (ARC-Easy, PIQA, SocialIQA).

The experiments sweep compute budgets both by scaling model size and by varying training steps (and thus, implicitly, number of epochs), while meticulously maintaining the relative size and filter ratios across datasets for fair comparisons. Supplementary experiments include deliberate injection of purely stochastic or heavily permuted data ("junk data") to probe model robustness.

Key Findings and Numerical Results

1. Unfiltered Data Outperforms Filtered Data at Sufficient Scale

The main empirical finding is robust: in the high-compute limit, models trained directly on unfiltered Common Crawl universally outperform their filtered counterparts across multiple scaling axes. For example, on a 1B parameter model, the unfiltered CC subset achieves an average validation loss of 3.37, improving upon all filtered pools at the same scale. This effect intensifies as model scale increases and with more training steps—a consistent manifestation across two orders of magnitude in data and model size.

2. Robustness to Low-Quality and Junk Data

Injecting low-quality or random data, including randomly sampled word strings and documents with shuffled word order, reveals an unexpected level of resilience. While smaller models show clear performance degradation proportional to the volume of noise, sufficiently large models, given enough optimization steps, close the gap entirely with clean data performance. In several configurations, models trained on mixtures with substantial proportions (even up to eight times) of "junk" data ultimately match or exceed the baseline on clean data, provided model capacity is high and training is sufficiently extensive.

3. Scaling Laws and Compute Projections

The authors derive empirical scaling laws that relate data pool size, model size, and number of training steps to the crossing point where unfiltered data overtakes filtered data in performance. Notably, these scaling laws are approximately linear in compute, predicting that the entire 240T-token CC pool becomes optimal at pretraining compute levels around $10^{30}$ FLOPs. Although such compute remains aspirational, forecasts suggest feasibility within this decade.

4. Distribution Shift and Degradation Edge Cases

Experiments investigating synthetically induced distribution shifts (e.g., shuffling word order) show that the observed robustness mostly extends to such cases, except on specific metrics that heavily depend on word order or initial token prediction. However, these effects are minor in typical LM applications.

5. Theoretical Justification

A formal analysis in the simplified setting of rank-constrained matrix factorization supports the empirical findings: models of sufficient capacity can allocate enough representational degrees of freedom to partition "good" from "bad" data, learning to ignore noise and extract signal even when signal is sparse. Theoretical models further illustrate when filtering could be beneficial—specifically, when the prevalence of incorrect or adversarially mislabeled data is high and indistinguishable from signal with current inductive biases.

Implications and Contradictory Claims

The study makes the strong and explicit claim: with sufficient compute and model size, the optimal data filter for LLM pretraining is, in practice, "no filter." This assertion directly challenges standard paradigms in data curation, which typically invest heavily in large-scale filtering operations premised on maximizing so-called data quality. The findings empirically refute the hypothesis that such filtering universally improves downstream metrics, showing that larger model capacity and compute alleviate or completely nullify the previously observed benefits of filtering.

Additional findings contradict prior work suggesting a persistent need for filtering strategies even at large scale, instead situating filtered datasets as only locally optimal in low-compute contexts.

Practical and Theoretical Implications

Practical implications are significant for both budget allocation in large-scale pretraining and dataset construction practices. Over-filtering, as commonly practiced, may waste potentially useful data and delay the attainment of optimal model performance, especially as scaling trends push toward ever larger models and compute budgets. Compute resources might be better spent training larger models for longer on maximal data pools rather than investing in additional data engineering efforts. The work highlights, however, that filtering remains necessary when harmful data (e.g., systematically incorrect or adversarial material) is prevalent—although, the authors find no evidence of this being common in current CC extractions.

On the theoretical side, the work reinforces Sutton’s "bitter lesson": in the long run, scalable, optimization-driven methodologies that eschew human-coded heuristics in favor of ingesting more raw data and compute outperform finely engineered, heavily supervised alternatives. The findings urge a reconsideration of inductive biases about what constitutes "useful" data and about model robustness to noise in the high-capacity regime.

Limitations and Future Directions

The study acknowledges several limitations. All experiments use dense transformer architectures; more complex, potentially less stable models (e.g., MoEs) may respond differently to low-quality data. Additionally, the compute requirements for the crossing point favoring unfiltered data are substantial, so in practical, compute-constrained settings, filtering still yields benefits. The projected increase of AI-generated content in web data is another open variable.

Potential future work includes:

• Extending experiments to newer architectures and hybrid finetuning regimes.
• Evaluating the impact of increasingly sophisticated or adversarial noise distributions.
• Exploring the effects of synthetic and heavily engineered data in contrast to organic web-scale corpora.
• Revising theoretical frameworks to accommodate more complex real-world signal/noise structures.

Conclusion

This paper provides compelling evidence that, for large-scale LLM pretraining, filtering for data “quality” is ultimately rendered obsolete by scaling model size and compute. Sufficiently large models not only tolerate noise, repetition, and even synthetic "junk," but also extract additional signal from data pools previously considered not usable. These results deliver a strong, quantitatively validated challenge to established best practices in data-centric ML, demonstrating that the returns of unfiltered data—when paired with adequate computational resources—ultimately surpass those obtainable from filtered corpora in the high-compute regime. Future model scaling endeavors and dataset construction should internalize these findings, with “less filtering, more data” as the guiding principle if compute permits.