Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
80 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

MathPile: A Billion-Token-Scale Pretraining Corpus for Math (2312.17120v2)

Published 28 Dec 2023 in cs.CL, cs.AI, and cs.LG

Abstract: High-quality, large-scale corpora are the cornerstone of building foundation models. In this work, we introduce MathPile, a diverse and high-quality math-centric corpus comprising about 9.5 billion tokens. Throughout its creation, we adhered to the principle of "less is more", firmly believing in the supremacy of data quality over quantity, even in the pre-training phase. Our meticulous data collection and processing efforts included a complex suite of preprocessing, prefiltering, language identification, cleaning, filtering, and deduplication, ensuring the high quality of our corpus. Furthermore, we performed data contamination detection on downstream benchmark test sets to eliminate duplicates and conducted continual pre-training experiments, booting the performance on common mathematical reasoning benchmarks. We aim for our MathPile to boost LLMs' mathematical reasoning abilities and open-source its different versions and processing scripts to advance the field.

PDF HTML Abstract

Introduction to the Corpus

This paper introduces MATH PILE, a comprehensive and multifaceted corpus designed specifically for the mathematical domain. The corpus contains approximately 9.5 billion tokens collected from a variety of sources, each with a strong mathematical focus. These sources include academic papers from arXiv, textbooks, ProofWiki entries, discussions from StackExchange, and web pages from Common Crawl.

Corpus Characteristics and Quality

To ensure the high quality of MATH PILE, a multi-step data processing regimen was executed. This included language identification to filter out non-English documents and a set of heuristic rules for cleaning and filtering, tailored to the unique aspects of mathematical text. Deduplication processes further refined the corpus by removing duplicative content from within and across sources. Additionally, data contamination detection was performed to eliminate instances found in downstream benchmark test sets, an often overlooked but critical step in corpus construction.

Structural Diversity and Annotations

The documents within MATH PILE are diverse not only in content but also in structure. Content ranges from the definitions and theorem proofs to more narrative styles found in textbook excerpts and website articles. They are enriched with annotations that enhance their utility for AI tasks, providing metadata such as language identification scores and the ratio of symbols to words, which future users can leverage for tailored filtering.

Utility and Future Directions

The creators aspire for MATH PILE to be a valuable asset for advancing AI's capabilities in mathematical reasoning. Envisaged as either a standalone tool or in conjunction with other general domain corpora, MATH PILE is well-positioned to improve AI systems' performance on various mathematically-oriented tasks. Moreover, ongoing updates and expansions to the corpus are planned, fostering continuous improvement and widening applicability.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (64)
  1. Miltiadis Allamanis. 2019. The adverse effects of code duplication in machine learning models of code. In Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software, Onward! 2019, page 143–153, New York, NY, USA. Association for Computing Machinery.
  2. AllenAI. 2023. allenai/dolma · datasets at hugging face. https://huggingface.co/datasets/allenai/dolma.
  3. Anthropic. 2023. Anthropic \ introducing claude. https://www.anthropic.com/index/introducing-claude.
  4. Proofnet: Autoformalizing and formally proving undergraduate-level mathematics. CoRR, abs/2302.12433.
  5. Llemma: An open language model for mathematics. CoRR, abs/2310.10631.
  6. Jack Bandy and Nicholas Vincent. 2021. Addressing "documentation debt" in machine learning research: A retrospective datasheet for bookcorpus. CoRR, abs/2105.05241.
  7. Emily M. Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587–604.
  8. GPT-NeoX-20B: An open-source autoregressive language model. In Proceedings of BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large Language Models, pages 95–136, virtual+Dublin. Association for Computational Linguistics.
  9. Andrei Z. Broder. 1997. On the resemblance and containment of documents. In Compression and Complexity of SEQUENCES 1997, Positano, Amalfitan Coast, Salerno, Italy, June 11-13, 1997, Proceedings, pages 21–29. IEEE.
  10. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
  11. Sparks of artificial general intelligence: Early experiments with GPT-4. CoRR, abs/2303.12712.
  12. Cerebras. 2023. Slimpajama: A 627b token, cleaned and deduplicated version of redpajama - cerebras. http://tinyurl.com/slimpajama.
  13. Generative ai for math: Abel. https://github.com/GAIR-NLP/abel.
  14. Palm: Scaling language modeling with pathways. J. Mach. Learn. Res., 24:240:1–240:113.
  15. Training verifiers to solve math word problems. CoRR, abs/2110.14168.
  16. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  17. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1286–1305, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  18. What’s in my big data? CoRR, abs/2310.20707.
  19. The pile: An 800gb dataset of diverse text for language modeling. CoRR, abs/2101.00027.
  20. Datasheets for datasets. Commun. ACM, 64(12):86–92.
  21. Similarity search in high dimensions via hashing. In VLDB’99, Proceedings of 25th International Conference on Very Large Data Bases, September 7-10, 1999, Edinburgh, Scotland, UK, pages 518–529. Morgan Kaufmann.
  22. Learning word vectors for 157 languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
  23. Textbooks are all you need. CoRR, abs/2306.11644.
  24. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
  25. Measuring mathematical problem solving with the MATH dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual.
  26. Scaling laws and interpretability of learning from repeated data. CoRR, abs/2205.10487.
  27. Mistral 7b. CoRR, abs/2310.06825.
  28. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427–431, Valencia, Spain. Association for Computational Linguistics.
  29. The stack: 3 TB of permissively licensed source code. CoRR, abs/2211.15533.
  30. Quality at a glance: An audit of web-crawled multilingual datasets. Transactions of the Association for Computational Linguistics, 10:50–72.
  31. The bigscience ROOTS corpus: A 1.6tb composite multilingual dataset. In NeurIPS.
  32. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8424–8445, Dublin, Ireland. Association for Computational Linguistics.
  33. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
  34. Solving quantitative reasoning problems with language models. In NeurIPS.
  35. Textbooks are all you need II: phi-1.5 technical report. CoRR, abs/2309.05463.
  36. Let’s verify step by step. CoRR, abs/2305.20050.
  37. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 158–167, Vancouver, Canada. Association for Computational Linguistics.
  38. A pretrainer’s guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity. CoRR, abs/2305.13169.
  39. Alexandra Luccioni and Joseph Viviano. 2021. What’s in the box? an analysis of undesirable content in the Common Crawl corpus. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 182–189, Online. Association for Computational Linguistics.
  40. Udi Manber and Eugene W. Myers. 1993. Suffix arrays: A new method for on-line string searches. SIAM J. Comput., 22(5):935–948.
  41. Data statements: From technical concept to community practice. ACM J. Responsib. Comput. Just Accepted.
  42. Measuring data. CoRR, abs/2212.05129.
  43. Chenghaomou/text-dedup: Reference snapshot.
  44. OpenAI. 2022. Introducing chatgpt. https://openai.com/blog/chatgpt.
  45. OpenAI. 2023. GPT-4 technical report. CoRR, abs/2303.08774.
  46. Openwebmath: An open dataset of high-quality mathematical web text. CoRR, abs/2310.06786.
  47. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080–2094, Online. Association for Computational Linguistics.
  48. The refinedweb dataset for falcon LLM: outperforming curated corpora with web data, and web data only. CoRR, abs/2306.01116.
  49. Improving language understanding by generative pre-training.
  50. Language models are unsupervised multitask learners.
  51. Scaling language models: Methods, analysis & insights from training gopher. CoRR, abs/2112.11446.
  52. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67.
  53. Analysing mathematical reasoning abilities of neural models. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
  54. BLOOM: A 176b-parameter open-access multilingual language model. CoRR, abs/2211.05100.
  55. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
  56. Together. 2023a. Redpajama, a project to create leading open-source models, starts by reproducing llama training dataset of over 1.2 trillion tokens. https://www.together.ai/blog/redpajama.
  57. Together. 2023b. Redpajama-data-v2: An open dataset with 30 trillion tokens for training large language models. https://www.together.ai/blog/redpajama-data-v2.
  58. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971.
  59. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288.
  60. Finetuned language models are zero-shot learners. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
  61. Naturalproofs: Mathematical theorem proving in natural language. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual.
  62. CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4003–4012, Marseille, France. European Language Resources Association.
  63. LIMA: less is more for alignment. CoRR, abs/2305.11206.
  64. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 19–27. IEEE Computer Society.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Zengzhi Wang (13 papers)
  2. Rui Xia (53 papers)
  3. Pengfei Liu (191 papers)
  4. Xuefeng Li (36 papers)
Citations (27)
View on Semantic Scholar
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - GAIR-NLP/MathPile: Generative AI for Math: MathPile (414 stars)
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube