Two-Stage GRPO: Difficulty-Partitioned Approach

• The paper’s main contribution is introducing a two-stage GRPO framework that partitions samples by difficulty to overcome vanishing gradients and enhance data utilization.
• It employs dynamic sample weighting with adaptive curriculum strategies, resulting in measurable improvements in tasks like mathematical reasoning and vision-language processing.
• Empirical validations demonstrate significant advances, including boosted pass@k metrics and reduced variance, as evidenced by variants such as Hint-GRPO and DGPO.

A Difficulty-Partitioned Two-Stage GRPO Strategy refers to a reinforcement learning framework built upon Group Relative Policy Optimization (GRPO), in which reasoning and generative model optimization are adaptively structured by difficulty metrics at several algorithmic levels. This family of approaches specifically addresses inefficiencies of standard GRPO, such as vanishing gradient problems in hard or trivial samples, low data utilization, and a misalignment between gradient signal and problem difficulty. The two-stage mechanism typically couples dynamic partitioning or weighting of samples by difficulty with stage-wise adaptations in both sampling and optimization objectives, and is broadly validated across mathematical reasoning, multimodal LLMs/MLLMs, vision-language, and video reasoning domains (Huang et al., 31 Mar 2025, Panaganti et al., 27 Jan 2026, Qi et al., 10 Nov 2025, Dai et al., 28 Jan 2026, Guan et al., 29 Jul 2025, Park et al., 9 Jun 2025, Chen et al., 19 May 2025, Dipta et al., 11 Jan 2026, He et al., 6 Aug 2025).

1. Principles of Difficulty Partitioning in GRPO

Difficulty partitioning in the GRPO framework splits the data or optimization pipeline into buckets (or groups) based on quantitative metrics reflecting the “hardness” of samples. These metrics may be derived from empirical pass@k rates, model-intrinsic reward statistics, synthetic masking (for multimodal data), or predefined difficulty scores.

For example, pass@k-based bucketing partitions training prompts into B bins by maintaining a sliding-window estimate:

$$\widehat{\mathrm{pass@}k}_t(x)=\frac1H \sum_{s=t-H+1}^t \mathbb{I}\left\{\text{any correct rollout}\right\}$$

These bins are defined by edges $a_0<a_1<\dots<a_B=1$, with $g_t(x)=b$ if $\widehat{\mathrm{pass@}k}_t(x)\in[a_{b-1},a_b)$ (Panaganti et al., 27 Jan 2026). Empirical alternatives include U-shaped accuracy distributions by multi-round sampling (Chen et al., 19 May 2025), stepwise chain-of-thought coverage (Huang et al., 31 Mar 2025), masking robustness of image inputs (Qi et al., 10 Nov 2025), or reward-based proxies such as mean advantage (Guan et al., 29 Jul 2025).

Difficulty partitioning serves multiple roles: it ensures every batch contains samples with informative reward variance (avoiding σ=0 degenerate cases), focuses optimization on currently underperforming regions in the task frontier, and enables adaptive curriculum learning.

2. Two-Stage GRPO Mechanisms

The two-stage paradigm in GRPO-based difficulty-aware RL typically entails:

Stage 1: Identification or construction of informative/solvable subgroups, possibly with augmentation or hint-injection, and targeted application of the vanilla or modified GRPO objective on these samples.

Stage 2: Enhanced learning on remaining hard cases through specialized mechanisms, which can include:

• Response resampling to enforce at least one correct answer per group (Guan et al., 29 Jul 2025, Huang et al., 31 Mar 2025).
• Advantage reweighting or difficulty-weighted policy gradients (Dai et al., 28 Jan 2026, Chen et al., 19 May 2025, Guan et al., 29 Jul 2025).
• Curriculum progression—beginning with “moderate” samples and transitioning to the hardest or with synthetic curriculum schedules (Dipta et al., 11 Jan 2026, Qi et al., 10 Nov 2025).
• Difficulty-guided data augmentation, such as injection of hints for hard samples, or perturbation of easy samples (Huang et al., 31 Mar 2025, Park et al., 9 Jun 2025, Chen et al., 19 May 2025).

This structure ensures almost all training samples yield nonzero policy gradients, overcomes RL stagnation on unsolved data, and allows for adaptation across domains and modalities.

3. Algorithmic Formalizations and Variants

The formal two-stage GRPO framework is instantiated in several ways, summarized in the table below:

Variant / Paper	Difficulty Signal	Partition/Weighting	Stage 1	Stage 2
Hint-GRPO (Huang et al., 31 Mar 2025)	Hint coverage in CoT	Minimal hint ratio	Find minimal hint	GRPO on hint-augmented group
DGPO (Dai et al., 28 Jan 2026)	Response accuracy	Weight by exp(difficulty)	MAD group adv. norm.	Softmax-weighted question loss
EMIT (Guan et al., 29 Jul 2025)	≥1 correct in G rollouts	“Easy”/“Hard”	Vanilla GRPO	Resampling + difficulty advantage
Curriculum-GRPO (Dipta et al., 11 Jan 2026)	Evaluator pass@k	Easy/Med/Hard buckets	SFT + weighted GRPO	Sampling curriculum
Reg-GRPO (Park et al., 9 Jun 2025)	Running mean group reward	Above/below reference	Augmentation (hint/noise)	Advantage regression
GDRO (Panaganti et al., 27 Jan 2026)	Online pass@k	Dynamic bins, bandit	Adversarial sampling	Rollout budget reallocation
TempFlow-GRPO (He et al., 6 Aug 2025)	SDE step noise	Early/Late step	High-impact exploration	Weight decay for refinement

Key design choices are the type and granularity of partitioning (discrete group vs. continuous weighting), handling of degenerate gradients, and the way in which rollout or data augmentation is coupled to the observed difficulty.

4. Training Objectives and Practical Implementation

The core training objective in difficulty-partitioned GRPO variants extends the standard clipped surrogate loss of PPO:

$$L_{\mathrm{GRPO}} = \mathbb{E}\Bigg[ -\frac{1}{\sum_{i,t}|o_i|} \sum_{i=1}^G\sum_{t=1}^{|o_i|} \min\left(\rho_{i,t} \hat A_{i,t}^{(w)}, \mathrm{clip}\!\left(\rho_{i,t},1-\epsilon,1+\epsilon\right)\hat A_{i,t}^{(w)}\right) + \beta\, D_{\mathrm{KL}}(\pi_\theta \Vert \pi_{\rm ref}) \Bigg]$$

where $\hat{A}_{i,t}^{(w)}$ may be advantage estimates normalized per group, further reweighted as a function of empirical sample or group difficulty (Dai et al., 28 Jan 2026, Guan et al., 29 Jul 2025).

In the prompt/rollout-adaptive GDRO formulation (Panaganti et al., 27 Jan 2026), additional online bandit controllers manage (a) sampling weights over difficulty bins, and (b) allocation of rollout counts to maximize variance reduction, with pseudo-code and theoretical guarantees for both.

For multimodal models, the training process often includes modality-specific bias correction (e.g., test-time logit recalibration in Hint-GRPO (Huang et al., 31 Mar 2025)) or data-split pipelines that leverage SFT in early stages before RL is focused on informative difficulty partitions (Qi et al., 10 Nov 2025, Dipta et al., 11 Jan 2026). Hyperparameters (batch sizes, temperature, group size G, binning granularity) are consistently tuned to maximize stable exploration and reward variance.

5. Empirical Validation and Performance Characteristics

Difficulty-partitioned two-stage GRPO strategies yield substantial empirical gains across model types and tasks. Representative improvements include:

• Data utilization in Hint-GRPO rises from ∼45% to ∼85%, with +4–6 accuracy points on geometry reasoning and an additional +1–2 points on vision-sensitive benchmarks after text-bias calibration (Huang et al., 31 Mar 2025).
• DGPO-based MathForge boosts pass@8 by +2.18 points over GRPO, with further +2.38 from data augmentation (Dai et al., 28 Jan 2026).
• GDRO adversaries produce average pass@8 gains of +10.1%–+13.1% over base GRPO, with a 33–37% reduction in weighted standard-error proxies (Panaganti et al., 27 Jan 2026).
• Difficulty-driven curriculum training yields ~3.8–5.6× faster convergence without accuracy loss, and preserves native-language chain-of-thought (e.g., >88% Bengali token fraction) (Dipta et al., 11 Jan 2026).
• Two-stage reweighting and augmentation in video reasoning (Reg-GRPO) produces up to +16.7 points over base video LLMs, and ∼20% reduction in vanishing-advantage rate (Park et al., 9 Jun 2025).
• In multimodal pipelines, difficulty-stratified GRPO-only outperforms SFT+GRPO hybrids by 1–2.5 points across several vision/reasoning benchmarks (Qi et al., 10 Nov 2025).

These benefits arise both from improved coverage of “hard” and “moderate” problems and from the reduction in uninformative updates and sample wastage. Ablations universally demonstrate that naive uniform sampling or unpartitioned RL is consistently inferior.

6. Theoretical Guarantees, Limitations, and Research Directions

Multi-adversary GDRO variants of the two-stage strategy admit provable no-regret guarantees for both sample and compute allocation stages, with explicit analysis showing square-root rollout allocation is variance-minimizing under compute-neutral constraints (Panaganti et al., 27 Jan 2026). The combination of curriculum, targeted exploration, and dynamic sampling gives rise to emergent “curriculum waves” that track the evolving problem frontier.

However, difficulty partitioning hinges on fidelity and granularity of the adopted difficulty metric. Overly aggressive or narrow filtering may reduce exploratory diversity, while improper weighting can destabilize training dynamics or induce adversarial sample selection.

On domains with evolving or highly multi-modal difficulty structure, online pass@k binning, attention-based metrics, or deployed bandit controllers provide the necessary adaptability but increase system complexity.

Active open questions include automated curriculum design in the presence of label/reward noise, partitioning for non-verifiable reward settings, and the extension to tightly coupled multimodal, sequential, or continual learning tasks. Consistent ablation results and widespread empirical validation underscore the broad applicability of the strategy across reasoning-centric generative models.

---

Key References:

• "Boosting MLLM Reasoning with Text-Debiased Hint-GRPO" (Huang et al., 31 Mar 2025)
• "Group Distributionally Robust Optimization-Driven Reinforcement Learning for LLM Reasoning" (Panaganti et al., 27 Jan 2026)
• "Harder Is Better: Boosting Mathematical Reasoning via Difficulty-Aware GRPO and Multi-Aspect Question Reformulation" (Dai et al., 28 Jan 2026)
• "EMIT: Enhancing MLLMs for Industrial Anomaly Detection via Difficulty-Aware GRPO" (Guan et al., 29 Jul 2025)
• "DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPO" (Park et al., 9 Jun 2025)
• "Revisiting the Data Sampling in Multimodal Post-training from a Difficulty-Distinguish View" (Qi et al., 10 Nov 2025)
• "GanitLLM: Difficulty-Aware Bengali Mathematical Reasoning through Curriculum-GRPO" (Dipta et al., 11 Jan 2026)
• "Unlocking the Potential of Difficulty Prior in RL-based Multimodal Reasoning" (Chen et al., 19 May 2025)
• "TempFlow-GRPO: When Timing Matters for GRPO in Flow Models" (He et al., 6 Aug 2025)