Group Relative Policy Optimization (GRPO)

Group Relative Policy Optimization (GRPO) is a reinforcement learning algorithm designed to efficiently optimize large policy models—such as LLMs—via sample-efficient, variance-reducing groupwise updates. GRPO has seen increasing adoption and methodological extension across diverse research domains, including LLM reasoning, code and image generation, safe alignment, vision-LLMing, and low-resource code synthesis. This entry reviews the core principles, algorithmic structures, preference aggregation theory, major practical enhancements, benchmark outcomes, and ongoing research directions defining GRPO and its variants as of mid-2025.

1. Core Algorithmic Principles

GRPO performs policy optimization by comparing and normalizing the rewards of sampled outputs generated for the same input prompt, implementing a groupwise relative reinforcement signal. Given a prompt (context) $q$, a group $G$ of candidate outputs $\{o_1, \ldots, o_G\}$ is sampled from the current policy. Each output is assigned a reward based on task-specific criteria (e.g., correctness, code quality, aesthetic quality). The group of rewards $\{r_1, \dots, r_G\}$ is normalized by subtracting the group mean and dividing by the group standard deviation, yielding a relative (or whitened) advantage for each output: $$A_i = \frac{r_i - \text{mean}(\{r_j\}_{j=1}^G)}{\text{std}(\{r_j\}_{j=1}^G)}$$ These group-relative advantages serve as the policy gradient estimator, with updates regularized by a divergence to a reference policy, typically using a Kullback–Leibler (KL) penalty. This approach removes the need for a learned value function—a key source of variance and instability in conventional RL—and leverages group-level feedback for more robust updates (Dao et al., 20 Feb 2025 , Vojnovic et al., 25 Feb 2025 , Mroueh, 9 Mar 2025 ).

2. Preference Aggregation and Alignment Objective

Distinguishing itself from standard RLHF approaches, GRPO aggregates preferences using a mechanism fundamentally different from logarithmic (softmax) pooling. The optimal stationary policy under GRPO satisfies a fixed-point equation: $$\left(1 - \frac{\mathcal{P}_G(o \mid \pi_\theta, q) - \mathbb{E}_{o'\sim \pi_\theta}[\mathcal{P}_G(o' \mid \pi_\theta, q)]}{\beta}\right) \pi_\theta(o|q) = \pi_\mathrm{ref}(o|q)$$ where $\mathcal{P}_G(\cdot)$ is the expected groupwise preference and $\beta$ the regularization coefficient. In the limit, GRPO's aggregation aligns with pairwise comparison methods for group size two and with normalized reward maximization for large group size, but the nonlinearity and use of a reverse-KL penalty differ from typical RLHF logarithmic pooling schemes (Vojnovic et al., 25 Feb 2025 ).

3. Extensions, Modifications, and Practical Enhancements

Several major extensions of GRPO have been developed:

• Kalman Filter Enhanced GRPO (KRPO): The static group mean is replaced by a recursive Kalman filter estimate for the baseline, improving bias-variance tradeoff when group rewards are noisy or nonstationary. The updated advantage is:

$$A_i = \frac{r_i - \hat{x}_{i|i}}{\sqrt{P_{i|i} + \varepsilon}}$$

where $\hat{x}_{i|i}$ and $P_{i|i}$ are the filtered mean and its variance (Wang et al., 12 May 2025 ).

• Spectral Policy Optimization: In groupings where all outputs are incorrect (yielding zero group variance and hence no learning signal), LLM-based judges are used to evaluate partial correctness at the step/reasoning level. Fractional rewards are assigned proportional to the number of correct steps before the first error, enabling gradient propagation even for all-negative batches (Chen et al., 16 May 2025 ).
• Multi-layer GRPO (MGRPO): A two-stage hierarchy is constructed: standard GRPO generates initial outputs, then a second GRPO stage explicitly prompts the model to reflect, spot, and correct errors in its own reasoning chains. Rewards are assigned for successful self-correction, providing implicit process-level supervision (Ding et al., 5 Jun 2025 ).
• Prefix Grouper for Computational Efficiency: This method restructures attention computation to encode long shared prefixes only once, reducing redundancy and significantly improving compute/memory efficiency in group-based training for long-context scenarios. The policy output and gradient are guaranteed to be identical to standard GRPO (Liu et al., 5 Jun 2025 ).
• Off-Policy GRPO: Rewards and advantages are computed using batches sampled from an older policy, allowing batches to be reused for multiple policy updates. This reduces the serving/communication overhead in large-scale distributed training without loss in empirical or theoretical performance, provided policy drift is controlled by regularization (Mroueh et al., 28 May 2025 ).

4. Empirical Results and Domain-Specific Performance

GRPO and its extensions have demonstrated strong results across a diversified set of domains:

• LLM Reasoning: For mathematical, logical, and code generation reasoning tasks, GRPO-trained models consistently increase binary/verifiable success rates; e.g., DeepSeek-R1 achieves near-perfect pass rates post-RL (Mroueh, 9 Mar 2025 , Dao et al., 20 Feb 2025 ).
• Safety and Alignment: Employing a multi-label reward model, GRPO-tuned LLMs achieve robust and balanced improvements in safety, politeness, actionability, and meaningfulness, outperforming PPO and preference-only fine-tuning methods in terms of both quantitative metrics and human alignment (Li et al., 26 Mar 2025 ).
• Vision-Language and GUI Agents: In GUI agent tasks, GRPO enables generalist LVLMs to surpass SFT-based baselines using only 0.02% as much data, largely due to efficient rule-based rewards and superior generalization (Xia et al., 14 Apr 2025 ). In industrial anomaly detection, GRPO combined with process-aware rewards (ROAM) enables end-to-end multimodal LLMs to beat larger closed-source models (Chao et al., 16 Apr 2025 ).
• Image and Video Generation: In autoregressive image generation, GRPO offers superior out-of-domain generalization and more robust adaptation to new prompts compared to DPO, although DPO remains superior in in-domain convergence speed (Tong et al., 22 May 2025 ). In long video generation, GRPO-optimized inference-time context selection enables up to 9x length extension without loss of consistency or prompt relevance (Fang et al., 23 May 2025 ).
• Code Generation: By incorporating detailed, severity-weighted static and dynamic code quality metrics into the reward, code LLMs fine-tuned with GRPO produce outputs consistently preferred by human experts, with significant observed improvement in maintainability, reliability, and security without sacrificing test correctness (Robeyns et al., 2 Jun 2025 ).

5. Theoretical Guarantees and Policy Improvement

Theoretical analyses have established that GRPO, when combined with appropriate regularization and clipping, guarantees reward improvement under both on-policy and off-policy regimes. Policy improvement recurrences converge to a fixed point that strictly amplifies the reward (success probability) of the reference policy (Mroueh, 9 Mar 2025 , Mroueh et al., 28 May 2025 ). Reverse-KL regularization constrains deviation from the reference, supporting robust reward amplification while limiting overfitting. In memory-limited or communication-constrained settings, off-policy GRPO variants enable efficient large-scale deployment without instability (Mroueh et al., 28 May 2025 ).

6. Applications, Generalization, and Open Resources

GRPO’s flexibility and variant adaptability (KRPO, MGRPO, Prefix Grouper, off-policy, unsupervised MM-UPT) enable a wide scope of real-world applications:

• Training and aligning state-of-the-art LLMs, MLLMs, and visual autoregressive models for open-ended reasoning, code generation in underrepresented languages, autonomous agent control, and multi-modal industrial decision making (Pennino et al., 20 May 2025 , Wei et al., 28 May 2025 , Li et al., 26 Mar 2025 , Xia et al., 14 Apr 2025 ).
• Unsupervised self-improvement pipelines (e.g., MM-UPT) where group-majority voting provides self-rewarding signals, obviating the need for external data or human feedback. These approaches approach the performance of fully supervised RL while being more scalable (Wei et al., 28 May 2025 ).

Many open-source implementations and checkpoints are provided, including DeepSeek-R1, Open-R1, MM-UPT, Prefix Grouper, Spectral Policy Optimization, and Kalman filter-enhanced GRPO, facilitating rapid adoption and experimentation across academic and industrial AI research.

7. Future Directions and Research Opportunities

Several research trajectories are actively being explored:

• Reward Model Generalization: Improved reward modeling (e.g., via CLIP in image generation, ROAM in anomaly detection, code quality aggregation) is central to further advances in GRPO-aligned strategies.
• Hybrid and Multi-layer Techniques: The development of multi-layer, process-dense, or hybrid GRPO frameworks (combining on-policy and off-policy, or integrating with preference optimization) for denser supervision and broader applicability.
• Scalability and Efficiency: Advances in batching and prefix-sharing techniques—in particular, Prefix Grouper—are lowering large-scale RL training costs and enabling longer context/sequence modeling.
• Diverse Domains: Expansion of applications to low-resource domains, highly multi-modal settings, non-English and underrepresented code languages, and dynamic online adaptation.

A plausible implication is that as group-based, relative optimization is empirically and theoretically established as robust and easy to scale, GRPO and its variants may become foundational mechanisms for large-scale, real-world reinforcement learning involving complex, verifiable, and multi-objective rewards.