FineMath: Corpus & Benchmark Overview
- FineMath is a dual-purpose resource that includes a curated mathematical pretraining corpus (e.g., FineMath4+ with up to 10B tokens) and a benchmark of 1,584 Chinese elementary math problems.
- The pretraining corpus is generated from Common Crawl using multi-stage LLM-based filtering to preserve LaTeX equations and step-by-step reasoning, leading to improved performance in models like SmolLM2.
- The evaluation benchmark rigorously assesses concept coverage and reasoning depth across 17 categories, exposing LLM sensitivity to prompt modifications and varying difficulty levels.
FineMath denotes two distinct resources in contemporary language-model research. In one usage, it is an open mathematical pretraining corpus derived from Common Crawl and curated for reasoning-focused, step-by-step mathematics in the SmolLM2 project. In another, it is a fine-grained evaluation benchmark of Chinese elementary-school math word problems designed to assess concept coverage, reasoning depth, and evaluation sensitivity in Chinese LLMs (Allal et al., 4 Feb 2025, Liu et al., 2024).
1. Name, scope, and disambiguation
The term FineMath is therefore polysemous. In data-centric pretraining work, it refers to a filtered mathematical web corpus with variants such as FineMath4+, FineMath-4plus, FineMath3+, and FineMath-3+. In evaluation work, it refers to a benchmark dataset of manually curated Chinese math word problems. These are separate artifacts with different goals, data-generation procedures, and evaluation roles.
| FineMath usage | Primary role | Reported scale |
|---|---|---|
| FineMath pretraining corpus | Mathematical pretraining data for LLMs | Up to 54B tokens; FineMath4+ at 10B tokens and 6.7M docs |
| FineMath benchmark | Fine-grained evaluation of Chinese LLM math reasoning | 1,584 elementary school math word problems in 17 categories |
Subsequent papers preserve this distinction while reusing the same name. Later comparisons describe FineMath-4plus as 9.57B tokens or 9.6B tokens, whereas the SmolLM2 paper rounds the same high-grade subset to 10B tokens. This suggests that the name is stable, but token accounting and variant nomenclature depend on the reporting context (Allal et al., 4 Feb 2025, Wang et al., 25 Jun 2025, Mahabadi et al., 20 Aug 2025).
2. FineMath as a mathematical pretraining corpus
The FineMath corpus in SmolLM2 was motivated by two stated deficiencies in existing open mathematical pretraining data: insufficient size and content misalignment. OpenWebMath was described as roughly 12B tokens and InfiMM-WebMath as roughly 40B tokens, both relatively small for robust math pretraining; both were also described as overrepresenting advanced academic material and underrepresenting step-by-step, reasoning-focused content relevant to high school and early undergraduate mathematics. Ablations on these datasets were reported as disappointing, including after multiple epochs, which motivated construction of a larger and better-targeted corpus (Allal et al., 4 Feb 2025).
The corpus was built from Common Crawl, beginning with the roughly 5.8B unique URLs in FineWeb and then expanding domains by adding all domains with at least 10 high-quality mathematical pages, together with all domains appearing in OpenWebMath and InfiMM-WebMath. Text extraction used Resiliparse and the OpenWebMath extraction pipeline so that mathematical structures such as LaTeX equations were preserved while boilerplate-only pages were filtered out. Curation then proceeded in two LLM-assisted stages. In the “silver” phase, Llama-3.1-70B-Instruct labeled documents on a 3-point scale ranging from “some mathematical content” to “good step-by-step problem solutions at an appropriate level”; a classifier trained on those labels selected domains with at least 10 URLs scoring at least 2. In the “gold” phase, the same model produced a 5-point scoring scale whose highest grades were reserved for content “outstanding in educational value for teaching and studying mathematics in middle school and high school” with detailed, easy-to-follow explanations, and a second classifier graded the full candidate set. Deduplication used MinHash LSH with 10 hashes, English filtering used a fastText-based classifier, and decontamination against GSM8K, MATH, and MMLU used 13-gram matching plus a required LCS overlap of at least 0.6 (Allal et al., 4 Feb 2025).
Several variants were then defined. FineMath4+ retained only grades 4–5 and was reported at 10B tokens and 6.7M documents. FineMath3+ retained grades 3–5 and was reported at 34B tokens and 21.4M documents. The full FineMath dataset was reported as up to 54B tokens. For comparison, the authors also regraded InfiMM-WebMath into Infi-WebMath4+ at 8.5B tokens and Infi-WebMath3+ at 20.5B tokens (Allal et al., 4 Feb 2025).
3. Role in SmolLM2 training and measured effects
Within SmolLM2, FineMath was not used as a uniformly upsampled corpus throughout training. The reported strategy was explicitly multi-stage: early and mid training did not heavily upsample math because of math’s small relative size, whereas the final annealing phase—the last 10% of training tokens—heavily upsampled FineMath4+ together with other high-quality math data such as Infi-WebMath3+, OpenWebMath, and AugGSM8K. In that late phase, math constituted 14% of the mixture (Allal et al., 4 Feb 2025).
The paper’s ablation protocol introduced each dataset or mixture through annealing at mid-training after 3T tokens of mostly web data, and evaluation used lighteval on GSM8K, MATH, and MMLU-STEM. Under this protocol, FineMath outperformed both OpenWebMath and InfiMM-WebMath. The paper states that FineMath4+ achieved a 2x improvement on GSM8K and a 6x improvement on MATH compared to InfiMM-WebMath, with example accuracies of roughly 28% versus roughly 14% on GSM8K and roughly 12% versus roughly 2% on MATH. The same study notes that performance increases substantially only when high-quality, reasoning-focused math data such as FineMath4+ is included (Allal et al., 4 Feb 2025).
After the full multi-stage pretraining recipe, SmolLM2 reported 31.1% on GSM8K (5-shot) and 11.6% on MATH (4-shot), and the paper states that the resulting 1.7B-parameter model outperformed other recent small LLMs including Qwen2.5-1.5B and Llama3.2-1B. In this setting, FineMath functioned less as a broad math crawl than as a late-stage capability injector targeted at reasoning-focused mathematical behavior (Allal et al., 4 Feb 2025).
4. Reassessments, rewritings, and successor corpora
Later work did not treat FineMath-4plus as an uncontested optimum. In OctoThinker, controlled mid-training experiments on Llama-3.2-3B-Base for 20B tokens compared FineMath-4plus, MegaMath-Web-Pro, and MegaMath-Web-Pro-Max. That study reported that FineMath-4plus produced only marginal gains and abnormal behavior under subsequent reinforcement learning: outputs beginning with \boxed{} followed by repetitive “Solution” statements up to the 4096-token maximum response length, together with underperformance on hard reasoning tasks relative to MegaMath-Web-Pro. The authors attributed this to curation flaws, including classifiers overly sensitive to extraction pipelines and acceptance of noisy or formula-heavy but context-poor snippets (Wang et al., 25 Jun 2025).
Another line of work treated FineMath-4+ as a strong base resource that could nonetheless be materially improved by rewriting. SwallowMath, approximately 2.3B tokens, was constructed from Finemath-4+ by removing boilerplate, pruning metadata, restoring missing context, rewriting verbose explanations to be concise, and reformatting solutions into clear step-by-step form. Under a fixed 50B-token continual pretraining budget for Llama-3.1-8B, substituting SwallowMath for vanilla Finemath-4+ increased GSM8K accuracy from 52.9% to 65.3% and MATH accuracy from 24.0% to 31.6%, corresponding to gains of +12.4 and +7.6 respectively. Ablations in that work identify rewriting as the dominant source of improvement (Fujii et al., 5 May 2025).
Nemotron-CC-Math advanced a different critique: that FineMath and related corpora were limited by brittle HTML-to-text extraction. That paper states that heuristic extractors such as Resiliparse, jusText, and Trafilatura strip or corrupt equations, code syntax, and inline LaTeX, whereas its own pipeline uses the Lynx text browser plus an LLM-based cleaning stage to preserve math across MathJax, KaTeX, and MathML, standardize notation into LaTeX, and maintain code block structure. Nemotron-CC-Math-4+ is reported at 52.3B tokens and 45.1M documents, and Nemotron-CC-Math-3+ at 133.3B tokens and 101.2M documents; the 4+ subset is described as 5.5 times larger than FineMath-4+. In controlled pretraining on Nemotron-T 8B, Nemotron-CC-Math-4+ exceeded FineMath-4+ on MATH and MBPP+, including 40.6 versus 35.8 on MATH and 45.1 versus 28.9 on MBPP+ (Mahabadi et al., 20 Aug 2025).
Taken together, these results do not erase FineMath’s importance, but they do narrow its interpretation. They suggest that corpus utility depends not only on mathematical topic selection, but also on extraction fidelity, document normalization, formatting regularity, and the compatibility of those properties with downstream mid-training or RL objectives.
5. FineMath as a Chinese mathematical evaluation benchmark
The FineMath benchmark introduced in 2024 is a separate resource: a fine-grained mathematical evaluation benchmark for Chinese LLMs. It contains 1,584 elementary school-level math word problems and is designed to cover the major key mathematical concepts taught in elementary school math. The benchmark is organized into 17 categories spanning Number & Operations, Measurement, Data Analysis & Probability, Algebra, Geometry, and two “Others” categories; each category has at least 60 problems, and no difficulty level within a category has fewer than 20 problems (Liu et al., 2024).
Its central design choice is fine-grained annotation by both concept and reasoning depth. The benchmark evaluates models along three dimensions: accuracy on abstract mathematical concepts, accuracy on reasoning steps, and overall accuracy. Difficulty is manually annotated by the number of required reasoning steps: Level-1 for one atomic reasoning step, Level-2 for two steps, and Level-3 for three or more steps. Construction used textbooks, workbooks, and reputable internet sources, followed by preprocessing to remove non-math, overly short, visually dependent, or ambiguous questions. Each item was categorized, rewritten to contain a single clear query, annotated with atomic reasoning steps and final answer, and double-checked for answer quality and step correctness. In addition to open-ended form, every problem was transformed into a multiple-choice question by adding distractors. Contamination with public Chinese MWP datasets such as Ape210K was measured using 13-gram overlap (Liu et al., 2024).
Evaluation on this benchmark highlighted both capability gaps and evaluation fragility. GPT-4 achieved roughly 73% accuracy and GPT-3.5-Turbo roughly 62%; the best Chinese LLM reported, MathGLM-10B, achieved above 40%, while some Chinese LLMs scored below 10%. Performance decreased as reasoning depth increased: for Qwen-7B-Chat, accuracy dropped from 62% on single-step problems to 21% on multi-step problems, whereas GPT-4 maintained above 60% even on Level-3 problems. The study also found strong prompt sensitivity: changing even a single word such as “Answer:” could alter GPT-4 accuracy by up to 15%. Multiple-choice formatting sometimes inflated low-performing models’ results and could also reduce the scores of stronger models, leading the authors to recommend generation-based evaluation as the more faithful assessment mode (Liu et al., 2024).
6. Methodological position and continuing relevance
Across both usages, FineMath occupies an important methodological position in data-centric math LLM research. The pretraining corpus made LLM-assisted filtering, educational-value grading, LaTeX-preserving extraction, benchmark decontamination, and late-phase upsampling into explicit design variables. The benchmark made concept granularity, reasoning-step annotation, contamination auditing, prompt sensitivity, and task-format sensitivity into first-class concerns rather than afterthoughts.
FineMath has also served as a target-domain substrate for training-schedule research. In controlled two-stage experiments on low-resource target domains, FineMath was used as the mathematical target dataset while C4 served as generic replay data. That study reports that replaying generic data during fine-tuning improved FineMath data efficiency by up to 1.49x over standard fine-tuning; a mid-training baseline using a joint Warmup-Stable-Decay schedule reached 6.37x over standard fine-tuning; and combining replay with some target-data exposure in pre-training reached 15.9x over the standard fine-tuning baseline (Kotha et al., 5 Mar 2026).
A persistent misconception is therefore that FineMath names a single canonical mathematical dataset with uniform implications. The literature supports the opposite conclusion. FineMath can denote either a pretraining corpus or an evaluation benchmark, and even within the pretraining line, later work reached sharply different conclusions depending on extraction pipeline, rewriting strategy, model family, and post-training objective. What remains stable across these settings is the broader lesson: mathematical performance in LLMs is highly sensitive to data curation granularity, structural preservation of notation, and the exact interface between training data and evaluation protocol.