RedStone: Curating General, Code, Math, and QA Data for Large Language Models (2412.03398v1)

Abstract: Pre-training LLMs on high-quality, meticulously curated datasets is widely recognized as critical for enhancing their performance and generalization capabilities. This study explores the untapped potential of Common Crawl as a comprehensive and flexible resource for pre-training LLMs, addressing both general-purpose language understanding and specialized domain knowledge. We introduce RedStone, an innovative and scalable pipeline engineered to extract and process data from Common Crawl, facilitating the creation of extensive and varied pre-training datasets. Unlike traditional datasets, which often require expensive curation and domain-specific expertise, RedStone leverages the breadth of Common Crawl to deliver datasets tailored to a wide array of domains. In this work, we exemplify its capability by constructing pre-training datasets across multiple fields, including general language understanding, code, mathematics, and question-answering tasks. The flexibility of RedStone allows for easy adaptation to other specialized domains, significantly lowering the barrier to creating valuable domain-specific datasets. Our findings demonstrate that Common Crawl, when harnessed through effective pipelines like RedStone, can serve as a rich, renewable source of pre-training data, unlocking new avenues for domain adaptation and knowledge discovery in LLMs. This work also underscores the importance of innovative data acquisition strategies and highlights the role of web-scale data as a powerful resource in the continued evolution of LLMs. RedStone code and data samples will be publicly available at \url{https://aka.ms/redstone}.

• The paper introduces RedStone, a scalable pipeline that leverages extraction and filtering modules to convert raw Common Crawl data into high-quality, structured training datasets.
• The pipeline creates specialized datasets—RedStone-Web, RedStone-Code, RedStone-Math, and RedStone-QA—that significantly improve benchmarks in common sense reasoning, code generation, mathematics, and question answering.
• The open-source release of RedStone’s code and datasets offers valuable resources for researchers and developers to further enhance the capabilities and accuracy of large language models.

RedStone: Curating Domain-Specific Data for Enhanced LLMs

The paper under discussion introduces "RedStone," an innovative data pipeline designed to systematically harness and curate datasets from Common Crawl for the pre-training of LLMs. RedStone aims to expand the capabilities of these models by leveraging the vast, diverse range of knowledge available on the web. The methodology adopted incorporates both general and domain-specific data, enhancing model performance across various tasks.

Overview of RedStone

RedStone is structured around two core modules: extraction and filtering. These tools facilitate the transformation of raw data from Common Crawl into structured datasets usable for training LLMs. The extraction module is tasked with processing raw web data through pattern recognition, natural language processing, and various computational methods to obtain training-ready formats. Concurrently, the filtering module utilizes advanced criteria—such as keyword searches, regular expressions, and machine learning models—to ensure the selection of only the most relevant data, discarding noise and redundancy.

Dataset Construction

The authors present a comprehensive approach to constructing datasets, categorizing them into general domain data and domain-specific data. General domain data aid in enhancing the model's language comprehension capabilities across diverse topics, while domain-specific datasets focus on areas such as code, mathematics, and question answering. The RedStone-Web dataset, derived from general domain data, comprises approximately 3.17 trillion tokens, crafted to deliver a rich representation of general knowledge.

For domain-specific endeavors, the RedStone pipeline creates several large-scale datasets: RedStone-Code, with 250.2 billion tokens, is tailored for programming knowledge; RedStone-Math, with 15.9 billion tokens, addresses mathematical reasoning; and RedStone-QA, encompassing 51.4 billion tokens, enhances question-answering capabilities. Each dataset is meticulously curated to bolster the model's specialized skills in pertinent fields.

Evaluation and Implications

The evaluation of RedStone datasets is conducted through various benchmark tasks, demonstrating improved performance over existing open-source datasets. Notably, RedStone-Web significantly enhances model outcomes in common sense reasoning challenges, outperforming comparable datasets in various tasks such as ARC-E and HellaSwag. The integration of RedStone-Code remarkably boosts performance in code generation benchmarks like HumanEval and MBPP. RedStone-Math also shows promise by surpassing current datasets on mathematics benchmarks, while RedStone-QA achieves high accuracy in question-answering tests.

With these findings, the paper concludes that the systematic utilization of Common Crawl datasets, guided by effective and scalable pipelines like RedStone, can unlock new domains for LLMs, enhancing both their versatility and accuracy. The open access to RedStone's code and data further provides an invaluable resource for researchers and developers, encouraging the continued evolution of LLMs.

Future Directions

The paper suggests several paths for future exploration. These include extending the pipeline to encompass multilingual datasets and incorporating multimedia content (videos, images, audio) to build multimodal LLMs. Additionally, the introduction of more sophisticated filtering mechanisms can further refine data quality and domain-specificity, ensuring datasets remain robust and applicable to emerging use cases. Collectively, these advancements hold the promise of driving further innovations in artificial intelligence, setting the stage for more adaptive and context-aware LLMs.