GRPO-based Reinforcement Learning

• GRPO-based Reinforcement Learning is a framework that uses group-normalized advantage estimation to eliminate critics and reduce variance in policy updates.
• It generalizes PPO by leveraging group statistics from rollouts, leading to robust and efficient performance across language, vision, and control tasks.
• Variants like KRPO, Rank-GRPO, and TIC-GRPO adapt the core idea for nonstationary rewards and structured outputs, cutting training time and improving empirical results.

Group Relative Policy Optimization (GRPO) is a reinforcement learning (RL) framework that generalizes Proximal Policy Optimization (PPO) by utilizing group-based, relative advantage estimation to enable critic-free optimization, robust policy improvement, and effective fine-tuning of generative models such as LLMs, autoregressive and diffusion-based image/video generators, and even closed-set representation learners. The GRPO paradigm has catalyzed a wave of recent advances in RL with verifiable and preference-based rewards, both in classical and language/vision domains, by streamlining the policy optimization process to eliminate value-function baselines and instead leveraging group statistics of sampled rollouts for variance reduction and stable training.

1. Core Algorithmic Concepts and Theoretical Foundations

At the foundation of GRPO is the replacement of single-sample or state-dependent advantage estimation with group-wise, relative normalization of returns. Given a policy $\pi_\theta$, an old policy $\pi_{\theta_{\mathrm{old}}}$, and a dataset of contexts (e.g., prompts for LLMs or images for vision models), $G$ rollouts (responses, trajectories, completions, etc.) are sampled per context, and each is assigned a scalar reward $r_i$. The group-normalized advantage for each sample is

$$\hat{A}_i = \frac{r_i - \bar{r}}{\sigma_r + \varepsilon}$$

where $\bar{r}$ is the group mean and $\sigma_r$ is the group standard deviation, with $\varepsilon$ a small positive constant for numerical stability. This advantage is then broadcast to all positions (e.g., tokens in a sequence), so every token or atomic decision within a completion receives the same advantage.

The GRPO objective is a clipped surrogate, akin to PPO, but without learned critics: $$\mathcal{L}_{\mathrm{GRPO}}(\theta) = -\frac{1}{G} \sum_{i=1}^G \sum_{t=1}^{|o_i|} \min\Big( r_{i,t}(\theta)\hat{A}_i,\, \mathrm{clip}(r_{i,t}(\theta),1-\epsilon,1+\epsilon)\hat{A}_i \Big) + \beta\, D_{\mathrm{KL}}[\pi_\theta\|\pi_{\mathrm{ref}}]$$ where $$r_{i,t}(\theta) = \frac{\pi_\theta(o_{i,t}|o_{i,<t})}{\pi_{\theta_{\mathrm{old}}}(o_{i,t}|o_{i,<t})}$$ is the token-level importance ratio (or an appropriate analogue in non-sequential models), $\epsilon$ is the PPO clip range, and $\beta$ controls a trust-region KL regularization to a reference policy $\pi_{\mathrm{ref}}$ (Mroueh, 9 Mar 2025, Pang et al., 4 Aug 2025, Wu et al., 1 Oct 2025).

Critically, the group-relative advantage—whitening returns within a mini-batch or per-context group—yields lower variance and effective learning signals, especially under binary or sparse reward settings typical in RL with verifiable rewards (RLVR), RLHF, and other programmatic feedback regimes.

2. Variants, Extensions, and Domain-Specific Adaptations

2.1 LLMs and RLVR

GRPO is foundational to SOTA RLVR pipelines, such as DeepSeek-R1, where binary correctness rewards are available for mathematical or programmatic reasoning. Theoretical analysis reveals that, under verifiable (binary) rewards, the GRPO update reduces to a KL-regularized contrastive loss that amplifies the policy's probability of successful completions over a reference, with provable upward success dynamics (Mroueh, 9 Mar 2025). It admits a closed-form for the optimal policy update, depending explicitly on the reward statistics and KL weight, and can be analyzed through a fixed-point recurrence for the improvement in task success rate.

2.2 Sample Efficiency, DPO Connection, and Group Size

Canonical implementations often use large group sizes ($m=8$ or $16$), but recent work shows that $m=2$ (2-GRPO) suffices and is theoretically equivalent to Direct Preference Optimization (DPO) in the pairwise case, both in algebraic objective and gradient structure (Wu et al., 1 Oct 2025). The empirical finding is that with matched rollout budgets, 2-GRPO performs comparably to 16-GRPO, but at 1/8 the rollout cost and over 70% reduction in training time. The equivalence extends to the unbiasedness of the estimated policy gradient up to a uniform scale factor.

2.3 Kalman-filtered and Adaptive Advantages

A recognized limitation of fixed group mean/variance normalization is susceptibility to high-variance, nonstationary rewards. Kalman Filter Enhanced GRPO (KRPO) replaces the batch mean and variance with dynamically adapted estimates using a 1D Kalman filter, centering returns at a running latent mean and normalizing by the filter's posterior uncertainty. This mechanism yields improved convergence rate and accuracy, especially for high-variance, nonstationary reward settings (e.g., harder math or reasoning tasks) (Wang et al., 12 May 2025).

2.4 Length, Rank, and Structured Preferences

Vanilla GRPO introduces length and granularity biases: longer completions receive the same per-token advantage, leading to verbosity and misaligned credit assignment in list or ranking tasks. $\lambda$-GRPO parametrizes a learnable token length preference, reweighting each rollout by a length-dependent function with gradients propagated to $\lambda$; this eliminates heuristic length bias and allows adaptation to dataset/task preference (Wang et al., 8 Oct 2025). For ranking/recommendation, Rank-GRPO moves the credit assignment from sequence/global to per-rank, constructing rank-wise returns, group advantages, and importance ratios, yielding improvements in coverage and convergence speed (Zhu et al., 23 Oct 2025).

2.5 Diffusion, Autoregressive, and Multimodal Models

GRPO has been extended to discrete diffusion models (MaskGRPO), autoregressive image generators (AR-GRPO), and video generation pipelines. These setups adapt the rollout, reward, and likelihood estimation to the sampling and optimization specificities of non-autoregressive or parallel generative architectures, leveraging importance reweighting of token-unmasking or chunked rollouts and custom reward schemes targeting perceptual, semantic, or structural alignment (Yuan et al., 9 Aug 2025, Ma et al., 3 Oct 2025, Meng et al., 16 Oct 2025).

2.6 Explicit Regret Regression

Addressing the issue of vanishing advantages, especially when group rewards are degenerate, Reg-GRPO reframes the GRPO loss as direct regression of policy log-likelihood ratios to group-normalized advantages, removing the need for heuristic clipping and preserving dense gradient flow even when standard GRPO would yield no update (Park et al., 9 Jun 2025).

3. Theoretical Guarantees, Bias, and Convergence

Recent work provides the first rigorous analysis of both classical and modified GRPO algorithms, showing:

• For group sizes and update schemes with stale (old) policy rollouts, GRPO approximates the true policy gradient at the old (rather than current) iterate, with bias controlled by learning rate and update lag. This bias is shown to be negligible in typical RLHF/RLVR inner-loop settings (Pang et al., 4 Aug 2025).
• By replacing token-wise importance ratios with trajectory-level ratios, Trajectory-corrected GRPO (TIC-GRPO) yields an unbiased estimator of the true on-policy gradient while retaining all practical advantages of the group surrogate method (Pang et al., 4 Aug 2025).
• Convergence rates match those of PPO/TRPO up to $O(\eta K)$ and $O(G^{-1})$ correction terms, with asymptotic convergence to stationarity under mild smoothness assumptions.
• In continuous control, the extension of GRPO achieves sample complexity and gradient variance reduction competitive with PPO by clustering trajectories into feature-based groups, normalizing within clusters, and regularizing policy updates with KL/Fisher penalties; convergence again follows Robbins-Monro arguments (Khanda et al., 25 Jul 2025).

4. Empirical Applications and Comparative Performance

GRPO serves as the backbone for RL fine-tuning in high-profile LLMs and multimodal models, and has been systematically benchmarked across domains:

• Mathematical reasoning, code generation, and general step-by-step reasoning: SOTA or SOTA-comparable performance, especially in RLVR settings (Zhang et al., 13 Apr 2025, Chen et al., 16 May 2025).
• Representation learning: GRPO-RM enables group-based advantage optimization for fixed-output-set models, yielding $+3–4\%$ accuracy improvements and faster convergence over baseline fine-tuning in both classification and dense prediction (Xu et al., 19 Nov 2025).
• Video and image generation: Identity-GRPO delivers $+18.9\%$ improvement on human identity consistency over existing video generators; AR-GRPO and MaskGRPO yield consistent improvements in image/sample quality for both class- and text-conditioned autoregressive and diffusion models (Yuan et al., 9 Aug 2025, Ma et al., 3 Oct 2025, Meng et al., 16 Oct 2025).
• Multimodal RL and perception: Syn-GRPO demonstrates scalable self-evolving RL, with online data synthesis pipelines improving diversity and task accuracy in vision-language tasks (Huang et al., 24 Nov 2025).
• Autonomous control and robotics: Flow-matching policies combined with GRPO-based RL outperform imitation and reward-weighted baselines in minimum-time and variable-horizon settings (Pfrommer et al., 20 Jul 2025).

A table summarizing select empirical improvements:

Domain	GRPO Baseline vs. Variant	Benchmark	Improvement	Source
Math Reasoning	GRPO vs. KRPO	OpenMath	+17.88% (hard)	(Wang et al., 12 May 2025)
Video Gen.	VACE vs. Identity-GRPO	ID Consist.	+18.9%	(Meng et al., 16 Oct 2025)
AR Image Gen.	AR baseline vs. AR-GRPO	CLIP/Recall	+0.03/+2 pts	(Yuan et al., 9 Aug 2025)
Recommenders	GRPO vs. Rank-GRPO	NDCG@20	+0.008–0.011	(Zhu et al., 23 Oct 2025)
Rep. Learn.	FT vs. GRPO-RM (Tiny-INet)	Softmax-Reg	+7.3%	(Xu et al., 19 Nov 2025)
MLLM Percep.	GRPO vs. Syn-GRPO	LISA	+6.04%	(Huang et al., 24 Nov 2025)

5. Limitations, Open Problems, and Future Directions

Despite its demonstrated flexibility and success, GRPO presents several limitations:

• In classical RL control, critic-free GRPO is competitive with PPO only in short-horizon or highly episodic problems; value-function baselines remain essential for long-horizon, continuous-action, or dense-reward settings (Oliveira et al., 5 Nov 2025, Khanda et al., 25 Jul 2025).
• Vanilla GRPO fails to provide gradient signal when entire groups are all negative; spectral policy optimization (SPO) and reward diversification via AI feedback address this by decomposing and "coloring" failures (Chen et al., 16 May 2025).
• Adaptive or group-specific normalization strategies, such as those employed in KRPO or difficulty-aware methods, significantly improve stability, but they demand careful tuning of noise/process parameters and reward shaping.
• Sample efficiency, compute overhead for group sampling, and the need for accurate reward models or verifiers can limit GRPO's usability in environments lacking access to compact verifiable reward signals.
• Ongoing work includes hybrid protocols combining empirical (group-based) returns with bootstrapped value baselines (Hybrid GRPO), adaptive sampling/entropy regularization, and modular reward shaping for open-ended, safety-critical, or compositional tasks (Sane, 30 Jan 2025, Khanda et al., 25 Jul 2025).

6. Implementation and Best Practices

• For RLHF/RLVR tasks with verifiable or rule-based rewards, GRPO (with group sizes as small as $m=2$) offers unbiased gradients and stable convergence, with rollout cost controllable via batch count (Wu et al., 1 Oct 2025, Mroueh, 9 Mar 2025).
• For LLMs, setting group size $G\sim8$ is typically sufficient. KL penalty/trust-region parameters must be tuned to match the capacity and exploration needs of the model (Mroueh, 9 Mar 2025, Wu et al., 1 Oct 2025).
• Length/rank/structure-aware extensions (λ-GRPO, Rank-GRPO, etc.) should be preferred wherever output granularity or credit assignment is misaligned with flat sequence-level reward (Wang et al., 8 Oct 2025, Zhu et al., 23 Oct 2025).
• For real-world or continuous control settings, regularize GRPO updates via KL/Fisher penalties, adapt group-based normalization to grouped trajectories, and ensure sufficient batch size per group (Khanda et al., 25 Jul 2025, Oliveira et al., 5 Nov 2025).
• Reward diversification (SPO, Syn-GRPO, etc.) is essential for RL on low-diversity or hard negative datasets (Chen et al., 16 May 2025, Huang et al., 24 Nov 2025).
• In settings with severe reward noise or nonstationarity, adaptive (e.g., Kalman-filtered) baselines significantly improve variance and stability (Wang et al., 12 May 2025).

---

References:

• (Mroueh, 9 Mar 2025) Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification
• (Wang et al., 12 May 2025) Kalman Filter Enhanced GRPO for Reinforcement Learning-Based LLM Reasoning
• (Chen et al., 16 May 2025) Spectral Policy Optimization: Coloring your Incorrect Reasoning in GRPO
• (Park et al., 9 Jun 2025) DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPO
• (Pfrommer et al., 20 Jul 2025) Reinforcement Learning for Flow-Matching Policies
• (Khanda et al., 25 Jul 2025) Extending Group Relative Policy Optimization to Continuous Control: A Theoretical Framework for Robotic RL
• (Pang et al., 4 Aug 2025) On the Theory and Practice of GRPO: A Trajectory-Corrected Approach with Fast Convergence
• (Yuan et al., 9 Aug 2025) AR-GRPO: Training Autoregressive Image Generation Models via RL
• (Wu et al., 1 Oct 2025) It Takes Two: Your GRPO Is Secretly DPO
• (Ma et al., 3 Oct 2025) Consolidating Reinforcement Learning for Multimodal Discrete Diffusion Models
• (Wang et al., 8 Oct 2025) $\lambda$-GRPO: Unifying the GRPO Frameworks with Learnable Token Preferences
• (Meng et al., 16 Oct 2025) Identity-GRPO: Optimizing Multi-Human Identity-preserving Video Generation via RL
• (Zhu et al., 23 Oct 2025) Rank-GRPO: Training LLM-based Conversational Recommender Systems with RL
• (Oliveira et al., 5 Nov 2025) Learning Without Critics? Revisiting GRPO in Classical RL Environments
• (Dechtiar et al., 10 Nov 2025) GRAPH-GRPO-LEX: Contract Graph Modeling and RL with GRPO
• (Xu et al., 19 Nov 2025) GRPO-RM: Fine-Tuning Representation Models via GRPO-Driven RL
• (Huang et al., 24 Nov 2025) Syn-GRPO: Self-Evolving Data Synthesis for MLLM Perception Reasoning

GRPO and its variants comprise a rapidly evolving toolkit for critic-free, sample-efficient, and scalable policy optimization in both classical and modern RL domains, supporting robust learning from verifiable supervision, structured or continuous outputs, and hybrid preference or diversity-based objectives.