FinerWeb-10BT: Refining Web Data with LLM-Based Line-Level Filtering (2501.07314v1)

Abstract: Data quality is crucial for training LLMs. Traditional heuristic filters often miss low-quality text or mistakenly remove valuable content. In this paper, we introduce an LLM-based line-level filtering method to enhance training data quality. We use GPT-4o mini to label a 20,000-document sample from FineWeb at the line level, allowing the model to create descriptive labels for low-quality lines. These labels are grouped into nine main categories, and we train a DeBERTa-v3 classifier to scale the filtering to a 10B-token subset of FineWeb. To test the impact of our filtering, we train GPT-2 models on both the original and the filtered datasets. The results show that models trained on the filtered data achieve higher accuracy on the HellaSwag benchmark and reach their performance targets faster, even with up to 25\% less data. This demonstrates that LLM-based line-level filtering can significantly improve data quality and training efficiency for LLMs. We release our quality-annotated dataset, FinerWeb-10BT, and the codebase to support further work in this area.

• The paper introduces an LLM-based line-level filtering method that precisely distinguishes high-quality from low-quality web data.
• It employs GPT-4o mini for detailed labeling and a DeBERTa-v3 classifier to scale filtering over 10 billion tokens, enhancing training efficiency.
• This approach achieves comparable model performance with up to 25% less data, reducing training time and computational costs while promoting sustainable AI.

FinerWeb-10BT: Refining Web Data with LLM-Based Line-Level Filtering

The paper under consideration introduces a methodological advancement aimed at refining the quality of datasets used for training LLMs, specifically employing LLM-based line-level filtering techniques. The authors propose a mechanism to enhance the quality of web-sourced text data, arguing that traditional heuristic filtering methods often miss the nuanced differentiation between low-quality and high-quality data, potentially curtailing the efficacy of model training. The paper utilizes a subset of the FineWeb dataset, labeled as FinerWeb-10BT, offering insights into an innovative approach that could redefine data preprocessing paradigms in the context of LLM training.

Methodological Overview

The proposed methodology is centered on the application of an LLM-based line-level filtering, where GPT-4o mini is utilized to label a sample of 20,000 documents from the FineWeb dataset. Each line within the documents is classified as either 'Clean' or identified under one of several low-quality categories. The LLM autonomously generates descriptive labels, subsequently grouped into nine overarching categories utilizing OpenAI's o1-preview model. This categorization facilitates the training of a DeBERTa-v3 classifier, extending the line-level filtering process to a much larger subset of the FineWeb dataset, encapsulating 10 billion tokens.

Evaluation and Results

The paper's experimental framework involves training GPT-2 models on both filtered and unfiltered versions of the dataset, measuring performance using the HellaSwag benchmark. Empirical findings indicate that the models benefiting from LLM-based filtering surpass their counterparts trained on unfiltered data. Remarkably, the filtered models achieve performance benchmarks with up to 25% less data. This underscores the premise that the removal of low-quality content can significantly uplift model accuracy and expedite the attainment of performance targets, consequently optimizing training time and resources.

The application of line-level filtering stands as a substantive shift from traditional document-level approaches, emphasizing precision in data cleansing processes. The potential reduction in dataset size without compromising model performance holds particular appeal given the escalating computational and environmental costs associated with training state-of-the-art LLMs.

Implications and Future Directions

The implications of this research are both practical and theoretical. Practically, the introduction of LLM-based filtering techniques poses a compelling argument for reevaluating data preprocessing standards, particularly in terms of efficiency and sustainability in dataset management. The research propounds a model that potentially reduces computational overhead, aligning with broader objectives of sustainable AI development by addressing the carbon output associated with exhaustive LLM training.

From a theoretical standpoint, the research elucidates how LLMs can be leveraged beyond their current capabilities, integrating them into the data curation process, and harnessing their understanding to refine training datasets. It demonstrates a scalable path for improving training data quality across diverse linguistic contexts and potentially, less-resourced languages.

Future work could extend these methodologies to other datasets and investigate the scalability of LLM-based filtering across various model architectures. Additionally, further research could focus on fine-tuning this approach to ensure robustness across multilingual data, given the observed variance in LLM performance in different language environments. Thorough evaluations, possibly incorporating diverse baselines, would further validate the utility and generalizability of these methods.

Conclusion

This paper represents a significant contribution to the ongoing discourse on optimizing LLM training efficacy through advanced data preprocessing techniques. By utilizing LLMs for line-level filtering, it provides a nuanced approach to data management that marries quality with efficiency, heralding a potential paradigm shift in machine learning model training protocols. As the field advances, the methodologies outlined could become instrumental in shaping more efficient, sustainable, and high-performance LLMs, aligning with both the cognitive capabilities of LLMs and the evolving needs of researchers and practitioners in natural language processing.