GRPO-Based Post-Training

• The paper introduces the GRPO framework that uses group-level reward normalization and PPO-style clipping to fine-tune models without requiring learned critics.
• It demonstrates that focusing on hard examples yields up to 47% performance gains, efficiently enhancing reasoning under constrained annotation budgets.
• Practical variants like Off-Policy GRPO, GRPO-λ, and RiskPO extend the approach across modalities, stabilizing training and improving alignment for LLMs and VLMs.

Group Relative Policy Optimization (GRPO)-Based Post-Training

Group Relative Policy Optimization (GRPO)-based post-training is a dominant framework for fine-tuning LLMs, vision-LLMs (VLMs), and related architectures under verifiable reward supervision. GRPO post-training has catalyzed improvements in complex reasoning, generalization, and alignment across language, vision, and generative domains. The core principle is to sidestep the need for value functions or learned critics by leveraging group-level, within-sample reward normalization and PPO-style update stability, thus facilitating both efficient exploitation and controlled exploration.

1. Core GRPO Objective and Mathematical Formalism

GRPO formalizes policy optimization under verifiable reward using a groupwise sample normalization mechanism. For a prompt $q$, sample $G$ responses $o_1, \dots, o_G$ from policy $\pi_{\theta_{\text{old}}}$, assign each a reward $r_i$, and compute group mean $\bar r = \frac{1}{G}\sum_i r_i$. The group-relative advantage for rollout $i$ is $A_i = r_i - \bar r$. The canonical loss is

$$\mathcal{L}_{\rm GRPO} = \mathbb{E}_{q}\mathbb{E}_{o\sim\pi_\theta} \left[ A(o,q)\log\pi_\theta(o|q) - \beta\,\mathrm{KL}(\pi_\theta(\cdot|q)\|\pi_{\rm ref}(\cdot|q)) \right]$$

with $\beta$ controlling KL-regularization strength. Implementation typically follows policy-gradient or PPO-style algorithms, with clipped likelihood ratios to ensure update stability (Pikus et al., 15 Aug 2025, Dai et al., 12 May 2025, Ding et al., 9 Dec 2025, Wei et al., 28 May 2025).

This group-based normalization ensures that only samples where there is reward variance within the group provide learning signal, with the advantage nullified if all group members have identical reward. In sequence generation contexts, this results in identical advantage for all positions within a trajectory, unless more refined credit assignment (see §4) is used.

2. Data Selection and Difficulty-Aware Sampling

Data sampling strategy is critical in GRPO-based post-training, especially under annotation or compute budget constraints. Empirical work demonstrates that focusing RL updates on the hardest examples—those with the lowest base-model success rates—yields markedly larger reasoning gains than sampling on easy, middle, or random subsets. This is formally operationalized by probing base model performance via $K$ independent completions, and defining subset selection policies as:

• Hardest subset: $$S_{\rm hard} = \arg\min_{S:|S|=p|\mathcal{X}|} \sum_{x\in S}\hat p(x)$$ where $\hat p(x)$ is empirical success rate.
• Easier/middle/random subsets are constructed analogously.

Experimental results across GSM8K and BIG-Bench Hard demonstrate that hard-example GRPO post-training yields up to 47% larger gains (e.g., +39.4% on GSM8K for Qwen3-14B) and uniquely improves out-of-distribution performance (Table 1, 2). Analysis shows that this approach maximizes the fraction of "learnable" steps—i.e., groups with nonzero within-group reward variance—enabling sustained policy learning (Pikus et al., 15 Aug 2025, Fatemi, 6 Jan 2026, Qi et al., 10 Nov 2025).

Practical guidance: Always prioritize difficult prompts under annotation budgets and robustly estimate difficulty with multiple base model samples before annotation.

3. Practical Variants, Extensions, and Algorithmic Advances

Numerous extensions of GRPO adapt the framework to specialized or enhanced experimental goals:

• Off-Policy GRPO: Incorporates importance sampling and decouples sampling policy from the current policy, with careful clipping and KL penalty to ensure stable updates and permit batched, efficient data server implementations. Off-policy GRPO is empirically as strong as on-policy and offers significant infrastructure savings for large-scale LLM training (Mroueh et al., 28 May 2025).
• GRPO-λ: Introduces token-level eligibility traces and $\lambda$-returns for better credit assignment in non-Markovian reasoning models, leading to faster convergence (30–40% over vanilla GRPO) and improved accuracy (3–4.5 percentage points) (Parthasarathi et al., 30 Sep 2025).
• S-GRPO: Employs a serial-group, decaying-reward mechanism for chain-of-thought (CoT) models, assigning largest reward to earliest correct early exits and thus reducing overthinking and redundant reasoning, with substantial reductions in generation length while maintaining or improving accuracy +0.72%–6.08%.
• TreeGRPO: For diffusion and flow-based generative models, recasts multi-candidate sampling as a tree, with fine-grained per-edge advantage estimation and amortized policy-gradient updates, yielding 2.4× training speedup and improved reward efficiency (Ding et al., 9 Dec 2025).
• GRPO-RM: Adapts GRPO to closed-set representation models (e.g., vision classification/segmentation), with deterministic group construction and task-specific reward definitions, yielding strong out-of-distribution gains and rapid convergence (Xu et al., 19 Nov 2025).
• DRA-GRPO: Applies diversity-aware reward adjustment, penalizing redundant completions via submodular mutual information, encouraging semantic diversity, and improving overall accuracy (Chen et al., 14 May 2025).
• PM4GRPO: Augments GRPO with process mining–based sequence conformance rewards to ensure not just correct final answers, but reasoning chains that follow high-quality teacher traces, with state-of-the-art results on challenging math reasoning benchmarks (Park et al., 29 Oct 2025).

A summary table of representative GRPO variants and their core technical modifications:

Variant	Domain	Key Technical Change
Off-Policy	LLMs	Importance-sampled, clipped updates
GRPO-λ	LLMs	Token-level eligibility traces
S-GRPO	LLMs, CoT	Serial early-exit, decaying rewards
TreeGRPO	Visual Gen. (Diff.)	Tree-structured, per-edge advantages
GRPO-RM	Vision Rep.	Fixed group, alignment+uniformity rwd
DRA-GRPO	LLMs	Diversity-aware reward penalty
PM4GRPO	LLMs (Math)	Process conformance reward

4. Structural Analysis, Equivalence to Supervised Fine-Tuning, and Limitations

Formal analysis reveals that under common LLM-MDP formulations—where states are simply token concatenations and the terminal reward is divided equally across tokens—the GRPO objective collapses into a degenerate weighted cross-entropy equivalent to outcome-driven supervised fine-tuning (SFT) with positive and negative samples. Token-wise advantage splitting and outcome-centric rewards induce no genuine temporal credit assignment, explaining why filtered ISFT can match or outperform GRPO under these assumptions (Samineni et al., 19 May 2025).

This equivalence holds unless richer intermediate rewards, non-trivial state representations, or value critics for token-level return-to-go are introduced. Artifactually longer response lengths sometimes observed in GRPO-trained models are a consequence of length scaling in the vanishing advantage schedule rather than emergent deeper reasoning.

5. Stability, Risk Sensitivity, and Convergence Guarantees

Mean-based GRPO objectives concentrate probability mass on already common, high-reward trajectories, risking entropy collapse and limited exploration of rare but informative reasoning paths. RiskPO generalizes GRPO by optimizing quantile-based (Mixed Value-at-Risk) objectives—directing gradient signal toward low-probability, challenging samples and empirically outperforming GRPO (+6.2% Pass@1 on DAPOMATH, among others) in mathematical and code reasoning while maintaining exploration (Ren et al., 1 Oct 2025).

GVPO replaces the heuristic group normalization of GRPO with a mean-squared error between model-implied and empirical group-centered rewards. This analytic surrogate aligns exactly with the true KL-constrained maxima, avoids instability from negative weights, and admits on-policy or off-policy sampling without importance weights or clipping (Zhang et al., 28 Apr 2025).

Best practices for stability: use moderate group size ($G\sim4$–16), carefully tune KL penalty $\beta$ and clipping $\epsilon$, adopt group-level variance diagnostics, and prefer GVPO or RiskPO in domains with rare informative reward tails.

6. Applications Across Modalities and Data Regimes

GRPO-based post-training is applied broadly:

• Language Modeling: Mathematical reasoning, chain-of-thought, code generation, closed-set and open-ended response alignment.
• Vision and Representation Models: Label-free post-training for VLMs and self-supervised curriculum RL using synthetic puzzles with difficulty weighting and consistency metrics (Jeddi et al., 16 Dec 2025).
• Low-Level Vision: Image restoration (IRPO) with targeted hard-sample mining and composite reward modeling combining perceptual and structural fidelity (Liu et al., 30 Nov 2025).
• Diffusion Models: Efficient alignment of image and video generators via tree-structured branching and progress-aware reward mixing (Ding et al., 9 Dec 2025, Li et al., 24 Nov 2025).
• Multimodal Reasoning: Data stratification by progressive image semantic masking and attention analysis, with pure GRPO-only regimes matching or exceeding hybrid SFT+GRPO, provided that medium and hard examples dominate batch construction (Qi et al., 10 Nov 2025).

Difficulty-based curriculum and prioritized replay mechanisms adaptively allocate optimization effort toward problems with maximal learning signal (where group variance is high). Heap-based, timestamped retesting ensures solved or forgotten problems are periodically revisited to avoid catastrophic forgetting (Fatemi, 6 Jan 2026).

7. Best Practices and Limitations

Implementing GRPO-based post-training in modern research practice entails:

• For LLMs: prioritize hardest examples under fixed budget, perform group sampling ($G=8$ optimal in most studies), apply PPO-style clipping and KL-regularization, and mask zero-variance batches for stability (Pikus et al., 15 Aug 2025, Mroueh et al., 28 May 2025).
• For VLMs and non-language domains: employ annotation-free self-supervised tasks, difficulty-aware weighting ($w(d)=4\sigma d(1-d)$), and direct reasoning–answer consistency regularizers (Jeddi et al., 16 Dec 2025).
• Computational efficiency: 2-GRPO (group size 2) with increased batch sizes replicates the performance of standard 16-GRPO at an eighth of the rollout cost (Wu et al., 1 Oct 2025).
• Recognize that naive GRPO under superficial MDP and outcome reward assumptions will not outperform, nor behave differently from, iterative SFT (except for possible differences in parameter drift or the preservation of existing capabilities) (Samineni et al., 19 May 2025, Rajani et al., 13 Jul 2025).

Convergent results across foundation model families and benchmarks demonstrate the generality and limitations of GRPO-style post-training: its strengths lie in scalable policy improvement, annotation budget efficiency, and flexibility for model-driven data curation; its weaknesses are exposed when reward structure and credit assignment remain outcome-centric or highly degenerate.

---

References:

• "Hard Examples Are All You Need: Maximizing GRPO Post-Training Under Annotation Budgets" (Pikus et al., 15 Aug 2025)
• "RL in Name Only? Analyzing the Structural Assumptions in RL post-training for LLMs" (Samineni et al., 19 May 2025)
• "S-GRPO: Early Exit via Reinforcement Learning in Reasoning Models" (Dai et al., 12 May 2025)
• "RiskPO: Risk-based Policy Optimization via Verifiable Reward for LLM Post-Training" (Ren et al., 1 Oct 2025)
• "Group Variance Policy Optimization for LLM Post-Training" (Zhang et al., 28 Apr 2025)
• "Puzzle Curriculum GRPO for Vision-Centric Reasoning" (Jeddi et al., 16 Dec 2025)
• "TreeGRPO: Tree-Advantage GRPO for Online RL Post-Training of Diffusion Models" (Ding et al., 9 Dec 2025)
• "Prioritized Replay for RL Post-training" (Fatemi, 6 Jan 2026)
• "It Takes Two: Your GRPO Is Secretly DPO" (Wu et al., 1 Oct 2025)
• "Growing with the Generator: Self-paced GRPO for Video Generation" (Li et al., 24 Nov 2025)
• "GRPO-$\lambda$: Credit Assignment improves LLM Reasoning" (Parthasarathi et al., 30 Sep 2025)
• "GRPO-RM: Fine-Tuning Representation Models via GRPO-Driven Reinforcement Learning" (Xu et al., 19 Nov 2025)
• "DRA-GRPO: Exploring Diversity-Aware Reward Adjustment for R1-Zero-Like Training of LLMs" (Chen et al., 14 May 2025)
• "Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Training" (Mroueh et al., 28 May 2025)
• "Revisiting the Data Sampling in Multimodal Post-training from a Difficulty-Distinguish View" (Qi et al., 10 Nov 2025)
• "IRPO: Boosting Image Restoration via Post-training GRPO" (Liu et al., 30 Nov 2025)
• "Scalpel vs. Hammer: GRPO Amplifies Existing Capabilities, SFT Replaces Them" (Rajani et al., 13 Jul 2025)
• "Reasoning-Aware GRPO using Process Mining" (Park et al., 29 Oct 2025)
• "Towards a Unified View of LLM Post-Training" (Lv et al., 4 Sep 2025)