ScaleDiff-Math: Scalable Math SFT Corpus

• ScaleDiff-Math is a difficulty-aware SFT dataset that merges 558K curated and 1.15M generated math problem–solution pairs.
• It employs a one-token adaptive difficulty classifier and tailored generator to efficiently identify and synthesize challenging math problems.
• Fine-tuning on ScaleDiff-Math yields notable improvements in contest-style benchmarks, with gains up to 65.9% Pass@1 accuracy over baseline models.

Searching arXiv for the primary paper and closely related context papers. ScaleDiff-Math is the final mathematical supervised fine-tuning corpus produced by the ScaleDiff pipeline for advanced mathematical reasoning. It is neither the generator model nor the pipeline itself, but the combined dataset obtained by augmenting an original high-quality math SFT corpus with a large generated set of difficult problem–solution pairs. In the formulation given by the authors, if $\mathcal{D}$ denotes the original math SFT dataset and $\mathcal{D}_{\text{DiffGen}}$ denotes the generated difficult set, then $$\mathcal{D}_{\text{Final}} \equiv \text{ScaleDiff-Math} = \mathcal{D} \cup \mathcal{D}_{\text{DiffGen}}$$; concretely, $\mathcal{D}$ contains 558K pairs, $\mathcal{D}_{\text{DiffGen}}$ contains 1.15M pairs, and ScaleDiff-Math contains 1.7M total problem–solution pairs. This dataset is used to fine-tune Qwen2.5-Math-7B-Instruct into the ScaleDiff-7B reasoning model (Pei et al., 25 Sep 2025).

1. Definition and relation to the ScaleDiff pipeline

ScaleDiff is a three-stage pipeline designed to scale the creation of genuinely difficult mathematical problems in a cost-efficient manner. Its stages are: identifying difficult problems in an existing high-quality math SFT corpus using an adaptive thinking model, training a dedicated difficult-problem generator on those difficult problems, and using a cost-efficient teacher to solve generated problems, filter the results, and merge them back with the original corpus (Pei et al., 25 Sep 2025). Within this architecture, ScaleDiff-Math is the terminal artifact: the enlarged, difficulty-hiked corpus on which the final reasoning model is trained.

The distinction between pipeline, generator, and dataset is explicit. ScaleDiff denotes the method; DiffGen-8B denotes the 8B generator model trained only to produce problem statements; ScaleDiff-Math denotes the final combined original-plus-generated math SFT dataset (Pei et al., 25 Sep 2025). DiffGen-8B is derived from Qwen3-8B-Base and is trained only on difficult problem statements, not on solutions. The generator produces new difficult problems, which are then solved by Qwen3-8B in “Thinking” mode and filtered to form $\mathcal{D}_{\text{DiffGen}}$.

The starting corpus $\mathcal{D}$ is the math-domain subset of AM-Qwen3-Distilled. Its problem pool is a curated mixture of DeepMath-103K, OpenR1-Math-220K, OpenMathReasoning, and NuminaMath, among others. AM-Qwen3-Distilled performs deduplication, rule filtering, decontamination against downstream evaluation benchmarks, solution distillation from Qwen3-235B-A22B with iterative sampling until a correct solution is obtained, and additional filtering using perplexity and n-gram metrics, yielding 558K instances (Pei et al., 25 Sep 2025).

This positioning matters because ScaleDiff-Math is a dataset-level contribution rather than a new base architecture. A plausible implication is that its primary research value lies in data construction and curriculum design for difficult reasoning, rather than in architectural novelty.

2. Difficulty identification and synthetic data construction

The central mechanism for constructing ScaleDiff-Math is adaptive difficulty identification followed by large-scale difficult-problem synthesis. Difficulty identification is performed by AdaptThink-7B, an adaptive thinking model trained via RL to decide whether a problem should receive a long Chain-of-Thought response (“Thinking”) or a direct answer (“NoThinking”). In ScaleDiff, this behavior is reinterpreted as a binary difficulty classifier using only the first token of the response:

$$\text{Difficulty}(x) = \begin{cases} \text{Simple} & \text{if } y_1 = \verb|</think>| \ \text{Difficult} & \text{if } y_1 \neq \verb|</think>| \end{cases}$$

Determining difficulty therefore requires only a single forward pass until token 1, rather than multiple solution attempts or judge-style prompting (Pei et al., 25 Sep 2025).

Applying AdaptThink to the initial problem pool yields a difficult subset $\mathcal{P}_{\text{Diff}}$ of size 192K, with a corresponding 192K problem–solution subset $\mathcal{D}_{\text{Diff}}$, and a simple subset $\mathcal{D}_{\text{DiffGen}}$0 of 366K problems. The paper reports that fine-tuning Qwen2.5-Math-7B-Instruct on different 192K-sized subsets produces markedly different downstream results: $\mathcal{D}_{\text{DiffGen}}$1 achieves an average of 56.6 across AIME’24, AIME’25, HMMT-Feb’25, BRUMO’25, and MATH500; a random 192K subset reaches 53.3; a simple 192K subset reaches 45.1; and the full 558K original dataset reaches 59.2 (Pei et al., 25 Sep 2025). This shows that the difficulty labels are operationally meaningful and that difficult problems contribute disproportionately to reasoning performance at fixed sample count.

DiffGen-8B is then trained on the 192K difficult problem statements. The training objective is standard language modeling over problem tokens alone, with no solutions or Chain-of-Thought in the loss:

$\mathcal{D}_{\text{DiffGen}}$2

where the instruction prefix is $\mathcal{D}_{\text{DiffGen}}$3 (Pei et al., 25 Sep 2025). Training uses Qwen3-8B-Base, batch size 128, max sequence length 1024, learning rate $\mathcal{D}_{\text{DiffGen}}$4, one epoch, and 10% warmup with linear decay, implemented with LLaMA-Factory. At generation time, the paper reports decoding parameters such as temperature $\mathcal{D}_{\text{DiffGen}}$5, top-$\mathcal{D}_{\text{DiffGen}}$6, and top-$\mathcal{D}_{\text{DiffGen}}$7.

The generated problem set $\mathcal{D}_{\text{DiffGen}}$8 is then solved by Qwen3-8B in “Thinking” mode, producing a provisional solution corpus that is subjected to two-stage filtration. Rule-based filtering removes excessive repetition, overly verbose or degenerate reasoning, and responses lacking a clearly delimited final answer in \boxed{...}. Model-based filtering then removes generated problems that Qwen2.5-Math-7B-Instruct, the base student model, already solves correctly; such instances are treated as redundant (Pei et al., 25 Sep 2025). About 43% of the initial generated samples are filtered out, leaving $\mathcal{D}_{\text{DiffGen}}$9 with 1.15M problem–solution pairs.

3. Dataset composition, format, and difficulty profile

ScaleDiff-Math contains 1.7M math problem–solution pairs: 558K original instances and 1.15M generated difficult instances (Pei et al., 25 Sep 2025). The paper characterizes this scale as comparable to large open math instruction datasets such as OpenMathInstruct-1/2 and larger than many widely used curricula such as DeepSeekMath, InternLM-Math, or the original MATH training data. The distinctive property, however, is not only scale but explicit difficulty hiking through AdaptThink-based selection and difficult-problem generation.

The generated portion is difficulty-skewed by construction. DiffGen-8B is trained on difficult problems only, and when its outputs are re-evaluated by AdaptThink, about 88% of generated problems are classified as difficult (Pei et al., 25 Sep 2025). Model-based filtration further biases the retained data toward problems that the base Qwen2.5-Math-7B-Instruct model cannot solve, making hardness relative to the student part of the selection criterion.

The paper also reports response-length analysis showing a gradient of reasoning complexity: simple subsets produce very short solutions, difficult original subsets produce longer solutions, and generated plus filtered subsets produce the longest solutions on average (Pei et al., 25 Sep 2025). This suggests that ScaleDiff-Math is not merely larger than the source corpus but is structurally shifted toward problems eliciting long Chain-of-Thought trajectories. A plausible implication is that the dataset emphasizes reasoning depth as well as answer correctness.

Although the paper does not provide explicit per-domain counts, the provenance of the source corpora suggests broad coverage of competition-style algebra, number theory, combinatorics, geometry, word problems, functional equations, inequalities, and some calculus and advanced topics consistent with Olympiad- and contest-level benchmarks (Pei et al., 25 Sep 2025). The authors do not claim domain-wise rebalancing; the main controlled variable is difficulty rather than topic. This contrasts with efforts centered on web-scale math pretraining corpora, where structural fidelity and domain breadth are primary concerns rather than difficulty steering (Mahabadi et al., 20 Aug 2025).

The dataset follows the Qwen series default chat or instruction template. Problems appear as user messages, often containing LaTeX; solutions appear as assistant messages containing long reasoning traces and a final answer explicitly boxed as \boxed{...}. For generated problems, the solution is distilled from Qwen3-8B; for original tasks, solutions come from Qwen3-235B-A22B (Pei et al., 25 Sep 2025). There is no explicit stored difficulty-label field in ScaleDiff-Math itself; difficulty functions internally as a selection and filtering variable.

4. Fine-tuning on ScaleDiff-Math and empirical effects

ScaleDiff-Math is used to fine-tune Qwen2.5-Math-7B-Instruct into the ScaleDiff-7B model. The SFT setup uses the full 1.7M-example dataset, batch size 32, max sequence length 32,768 tokens, and 3 epochs, with other hyperparameters matching the DiffGen-8B configuration. Because the base model’s native context length is 4096, the authors follow OpenR1 and modify the RoPE parameter rope_theta from 10K to 300K to support 32k context (Pei et al., 25 Sep 2025).

On AIME’24, AIME’25, HMMT-Feb’25, BRUMO’25, and MATH500, ScaleDiff-7B achieves 73.0%, 58.7%, 43.3%, 66.7%, and 95.2%, respectively, for a 65.9% average (Pei et al., 25 Sep 2025). These results are reported as Pass@1 accuracy averaged over multiple seeds, using LightEval and vLLM. The baseline AM-Qwen3-Distilled-7B, trained only on the original 558K math data, achieves 63.0%, 51.7%, 33.3%, 60.7%, and 94.6%, for a 59.2% average. The move from the original corpus to ScaleDiff-Math therefore yields an absolute gain of 6.7 points and a relative improvement of 11.3% (Pei et al., 25 Sep 2025).

The paper compares ScaleDiff-7B with other strong 7B-scale LRMs. OpenThinker3-7B attains 63.4% average, AceReason-Nemotron-7B 59.2%, Light-R1-7B-DS 52.4%, MiMo-7B-SFT 53.2%, MiMo-7B-RL 64.1%, and OpenMath-Nemotron-7B 66.9%, while ScaleDiff-7B reaches 65.9% (Pei et al., 25 Sep 2025). The result places ScaleDiff-7B above several strong SFT and RL baselines and close to OpenMath-Nemotron-7B. The teacher Qwen3-8B itself reaches 68.9% average, so the student closes much of the gap to its 8B teacher despite having 7B parameters.

A further empirical result is a scaling phenomenon with respect to the amount of difficult data. The paper evaluates augmentation sizes of $$\mathcal{D}_{\text{Final}} \equiv \text{ScaleDiff-Math} = \mathcal{D} \cup \mathcal{D}_{\text{DiffGen}}$$0, $$\mathcal{D}_{\text{Final}} \equiv \text{ScaleDiff-Math} = \mathcal{D} \cup \mathcal{D}_{\text{DiffGen}}$$1, and $$\mathcal{D}_{\text{Final}} \equiv \text{ScaleDiff-Math} = \mathcal{D} \cup \mathcal{D}_{\text{DiffGen}}$$2 the size of the original dataset and finds that AIME’24 and AIME’25 accuracy increases monotonically with more difficult data and does not saturate at $$\mathcal{D}_{\text{Final}} \equiv \text{ScaleDiff-Math} = \mathcal{D} \cup \mathcal{D}_{\text{DiffGen}}$$3 augmentation, whereas MATH500 shows little gain because the baseline is already near saturation (Pei et al., 25 Sep 2025). The authors interpret this as direct evidence of a difficulty-aware scaling law: adding more difficult problems, rather than merely more problems, continues to improve performance on hard benchmarks.

Ablations reinforce that the gain is not reducible to raw volume alone. Under a fixed 192K fine-tuning budget, full ScaleDiff-generated difficult data with rule and model filtering reaches 58.4 average, removing filtration drops performance to 55.3, and removing both filtration and difficult-problem selection drops it further to 50.4 (Pei et al., 25 Sep 2025). The paper therefore attributes gains specifically to difficult-problem selection and filtration quality, not simply to synthetic augmentation.

5. Efficiency, methodological positioning, and related lines of work

A major claim of the ScaleDiff framework is cost efficiency. Prior work on question synthesis and difficulty control is described as frequently relying on large proprietary models, complex per-instance prompting, and rejection sampling over multiple attempts. ScaleDiff replaces these with a one-token difficulty classifier, a specialized 8B generator trained once on difficult data, and an 8B teacher used for single-sample distillation (Pei et al., 25 Sep 2025). The paper reports that using Qwen3-235B-A22B rather than Qwen3-8B as the distillation teacher for the 192K difficult subset changes average performance only from 55.6 to 56.6, a marginal 1-point difference (Pei et al., 25 Sep 2025). This supports the use of Qwen3-8B as a cost-effective teacher.

This cost-efficient positioning differs from other recent strategies for increasing mathematical difficulty. The MATH$$\mathcal{D}_{\text{Final}} \equiv \text{ScaleDiff-Math} = \mathcal{D} \cup \mathcal{D}_{\text{DiffGen}}$$4 pipeline, for example, creates difficult problems by composing two skills extracted from MATH, using a multi-step LLM-plus-human workflow with skill-pair validation, question generation, defeatist self-solution, rubric-based validation, and human verification (Shah et al., 2024). That method emphasizes compositional difficulty and benchmark novelty; ScaleDiff-Math instead emphasizes scalable corpus construction for SFT. The two are methodologically adjacent but operationally distinct.

ScaleDiff-Math is also distinct from large-scale math pretraining corpora such as Nemotron-CC-Math, which focus on recovering structurally faithful mathematical and code content from Common Crawl via layout-aware rendering and LLM-based cleanup (Mahabadi et al., 20 Aug 2025). Nemotron-CC-Math addresses the quality of raw math pretraining text; ScaleDiff-Math addresses the creation of difficult supervised reasoning examples. A plausible implication is that the two approaches occupy different stages of the training stack: one for broad math pretraining, the other for reasoning-specific post-training.

Another relevant comparison concerns forward-pass-based math specialization. Math Neurosurgery identifies math-specific parameters using only forward passes and then prunes or scales them to suppress or enhance GSM8K performance with minimal effect on non-math behavior (Christ et al., 2024). ScaleDiff-Math also exploits forward-pass efficiency, but at the dataset-construction level rather than the parameter-editing level: AdaptThink uses only the first generated token as a difficulty signal (Pei et al., 25 Sep 2025). The shared theme is that inexpensive inference-time signals can serve as surprisingly informative selectors for mathematical capability.

Finally, ScaleDiff-Math differs from RL-oriented curation efforts such as Big-Math, which prioritize uniquely verifiable, open-ended, closed-form math problems suitable for automatic reward computation (Albalak et al., 24 Feb 2025). Big-Math is a large RL-ready question set; ScaleDiff-Math is a teacher-distilled SFT corpus constructed to amplify difficult reasoning. The overlap lies in their shared concern with answer verifiability, filtration, and difficulty, but the intended downstream use differs.

6. Limitations, robustness, and reproducibility

The paper states several limitations. First, the mathematical correctness and solvability of generated problems is not fully verified; the authors explicitly describe this verification problem as “highly non-trivial” and leave it for future work (Pei et al., 25 Sep 2025). Filtration targets solution quality, answer format, repetition, and student-model redundancy, rather than complete proof of problem validity.

Second, generated solutions may contain errors because only a single teacher sample per problem is used and no rejection sampling or post-hoc verification is applied (Pei et al., 25 Sep 2025). The authors argue, citing prior work, that partially noisy Chain-of-Thought traces can still improve reasoning during SFT. This suggests a training philosophy in which diversity and difficulty can outweigh perfect label cleanliness, provided the noise level is not overwhelming.

Third, there is potential distribution shift. Generated problems are shaped by the difficult subset $$\mathcal{D}_{\text{Final}} \equiv \text{ScaleDiff-Math} = \mathcal{D} \cup \mathcal{D}_{\text{DiffGen}}$$5 and by the generative behavior of DiffGen-8B, so they need not match benchmark distributions such as AIME or HMMT exactly (Pei et al., 25 Sep 2025). The observed gains on those benchmarks indicate that the shift is not harmful in aggregate, but the paper does not claim formal alignment.

Fourth, the original source corpus is decontaminated against downstream tasks, but the paper does not report explicit cross-checking of generated problems for near-duplication against modern contest problems (Pei et al., 25 Sep 2025). Fifth, while evaluation follows multiple-seed “sober” practices, the paper does not present dedicated overfitting analyses.

Despite these limitations, the release is designed to be reproducible. The authors publicly release the ScaleDiff-Math dataset, DiffGen-8B, ScaleDiff-7B, and code for running AdaptThink, training DiffGen-8B, generating problems, distilling solutions from Qwen3-8B in “Thinking” mode, applying rule and model filtering, and packing data in Qwen chat format (Pei et al., 25 Sep 2025). The code repository is hosted at https://github.com/QizhiPei/ScaleDiff, and the associated Hugging Face collection includes the main artifacts (Pei et al., 25 Sep 2025).

Within the broader evolution of mathematical data resources, ScaleDiff-Math belongs to a shift from generic web-scale math pretraining and conventional math instruction corpora toward explicitly difficulty-aware data generation. Related work has explored skill-compositional question generation (Shah et al., 2024), RL-suitable open-ended problem curation (Albalak et al., 24 Feb 2025), structurally faithful math web extraction (Mahabadi et al., 20 Aug 2025), and forward-pass identification of math-specific model components (Christ et al., 2024). ScaleDiff-Math’s specific contribution is to operationalize “scaling difficult problems” as a concrete 1.7M-example training corpus for supervised reasoning, showing that increasing the quantity of filtered difficult problems can materially improve advanced contest-style mathematical performance in an open 7B model (Pei et al., 25 Sep 2025).