Harder Is Better: Boosting Mathematical Reasoning via Difficulty-Aware GRPO and Multi-Aspect Question Reformulation

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) offers a robust mechanism for enhancing mathematical reasoning in large models. However, we identify a systematic lack of emphasis on more challenging questions in existing methods from both algorithmic and data perspectives, despite their importance for refining underdeveloped capabilities. Algorithmically, widely used Group Relative Policy Optimization (GRPO) suffers from an implicit imbalance where the magnitude of policy updates is lower for harder questions. Data-wise, augmentation approaches primarily rephrase questions to enhance diversity without systematically increasing intrinsic difficulty. To address these issues, we propose a two-dual MathForge framework to improve mathematical reasoning by targeting harder questions from both perspectives, which comprises a Difficulty-Aware Group Policy Optimization (DGPO) algorithm and a Multi-Aspect Question Reformulation (MQR) strategy. Specifically, DGPO first rectifies the implicit imbalance in GRPO via difficulty-balanced group advantage estimation, and further prioritizes harder questions by difficulty-aware question-level weighting. Meanwhile, MQR reformulates questions across multiple aspects to increase difficulty while maintaining the original gold answer. Overall, MathForge forms a synergistic loop: MQR expands the data frontier, and DGPO effectively learns from the augmented data. Extensive experiments show that MathForge significantly outperforms existing methods on various mathematical reasoning tasks. The code and augmented data are all available at https://github.com/AMAP-ML/MathForge.

• The paper introduces MathForge, a framework that enhances LLM mathematical reasoning by rebalancing training on hard but solvable questions.
• MathForge employs DGPO with MAD-based normalization and difficulty-aware weighting to achieve up to a 4.56% performance boost across multiple benchmarks.
• It also leverages multi-aspect question reformulation to generate logically-equivalent, challenging training data, thereby improving model generalization.

MathForge: Advancing Mathematical Reasoning via Difficulty-Aware Optimization and Multi-Aspect Question Reformulation

Overview and Motivation

"Harder Is Better: Boosting Mathematical Reasoning via Difficulty-Aware GRPO and Multi-Aspect Question Reformulation" (2601.20614) introduces MathForge, a synergistic framework for elevating mathematical reasoning in LLMs through an algorithmic and data-centric approach. The efficacy of reinforcement learning with verifiable rewards (RLVR) in reasoning tasks is well established, but current techniques disproportionately favor questions of moderate difficulty, leading to under-trained capabilities precisely where model generalization lags—on harder, yet tractable, questions.

MathForge directly addresses two systemic limitations:

• Algorithmic Limitation: Standard Group Relative Policy Optimization (GRPO) updates model policies less on difficult questions due to its advantage normalization, providing maximal update magnitude for questions of intermediate difficulty.
• Data Limitation: Existing question augmentation techniques focus on rephrasing for diversity but ignore systematic increases in intrinsic problem difficulty.

To resolve these, MathForge leverages Difficulty-Aware Group Policy Optimization (DGPO) to rebalance and refocus model updates and Multi-Aspect Question Reformulation (MQR) to generate challenging, logically-equivalent synthetic data. This two-pronged strategy aims to enhance both training dynamics and data distribution.

Methodology

Difficulty-Aware Group Policy Optimization (DGPO)

DGPO extends GRPO with two critical enhancements:

1. Difficulty-Balanced Group Advantage Estimation (DGAE): DGPO normalizes group advantage by mean absolute deviation (MAD) of rewards, rather than the standard deviation used in GRPO. Formally, for $G$ responses to a question:

$$\hat{A}_{\text{DG},i} = \frac{r_{i} - \operatorname{mean}( \{ r_{i} \} ) }{ \operatorname{MAD}( \{ r_{i} \} ) },$$

where $r_{i}$ is the reward for response $i$, and $MAD$ ensures that the aggregate advantage magnitude is constant ($G$), irrespective of the underlying difficulty.

2. Difficulty-Aware Question-Level Weighting (DQW): DGPO assigns higher weights to questions with lower model accuracy ($D_s = -\operatorname{mean}( \{ r_{si} \} )$ across responses), prioritizing policy updates on harder instances. The weighting $\lambda_s$ follows a softmax distribution over current batch difficulties, modulated by a temperature $T$.

These adjustments enable DGPO to systematically upweight challenging, solvable questions during RLVR, eliminating the default bias toward questions of moderate difficulty imposed by standard GRPO.

Multi-Aspect Question Reformulation (MQR)

MQR introduces a set of targeted, answer-preserving augmentations to expand the distribution and complexity of training questions. Reformulation proceeds along three axes:

• Background Addition: Embeds distracting or contextually nuanced story elements to require more rigorous information filtering.
• Term Introduction: Defines new abstract terminology for core concepts, enforcing abstraction and mathematical composition.
• Sub-Problem Nesting: Converts fixed numerical values into sub-problems across different mathematical domains, challenging multi-step and cross-domain reasoning.

All reformulations strictly preserve the solution, ensuring the mathematical logic remains intact and no answer regeneration is required. The MQR process can be automated by powerful LLMs or capable open-source models.

Experimental Results

Extensive experiments using Qwen2.5-Math-7B and other models across six major benchmarks (AIME24, AIME25, AMC23, MATH500, Minerva, Olympiad) validate MathForge’s effectiveness. Notable findings:

• DGPO alone yields a 2.18% average improvement over GRPO; it robustly outperforms other advanced policy optimization methods across all tested models.
• MQR alone provides an even larger 3.43% average gain by systematically increasing training question difficulty.
• Combined MathForge achieves a 4.56% average boost, showcasing strong synergy between data and algorithmic reinforcement.

Further, ablation studies demonstrate that both DGAE and DQW are essential—removing either erodes gains, and the chosen temperature $T = 2.0$ balances sharpness and diversity in difficulty weighting. DGPO is compatible as an enhancement layer atop other RL approaches (e.g., GPG, DAPO, GSPO), and is effective in multimodal reasoning settings.

In the MQR analysis, the reformulated data consistently proves itself harder for the base model, leading to superior generalization and zero overfitting. Equivalence checks confirm a >97% fidelity in generated questions to the original solutions.

Implications and Future Directions

MathForge’s results reinforce the principle that upweighting and targeting training on hard-but-solvable tasks leads to measurable improvements in LLM mathematical reasoning without requiring larger models or prohibitive compute costs. The use of MAD normalization in group advantage estimation corrects a fundamental imbalance in RLVR, and multi-aspect data augmentation trains models to extract mathematical structure despite noise, abstraction, and compositional complexity.

Practical Implications:

• MathForge can be adopted in open-source RLVR pipelines and integrated with any RL-based reasoning framework.
• MQR reformulation workflows are lightweight and generalizable, requiring no solution generation and thus enabling scalable augmentation for broader mathematical domains.
• Difficulty-aware reweighting mechanisms should become standard for RL on synthetic or curated reasoning datasets.

Theoretical Implications:

• The demonstration of superior learning by focusing on challenging, low-accuracy queries suggests new directions for curriculum learning and adaptive sampling strategies in RL for LLMs.
• MAD-based normalization may outperform variance-based alternatives in domains with non-Gaussian reward structures and binary or sparse feedback.

Speculative Developments:

• Extensions to automatically estimate and control difficulty in multi-hop, cross-domain agentic environments.
• Generalizing MQR and DGPO principles for scientific, logical, or symbolic reasoning domains outside mathematics.
• Combining MathForge with methods that dynamically generate new hard problems based on model epistemic uncertainty, further bootstrapping capability.

Conclusion

MathForge systematically enhances mathematical reasoning in LLMs by optimally targeting hard questions through difficulty-aware policy optimization and multi-aspect reformulation. Extensive, model-agnostic benchmarks and analyses substantiate substantial improvements over existing approaches. These findings establish that prioritizing harder instances during training is both a robust and effective paradigm for advancing AI reasoning capabilities.