Group Reward Policy Optimization (GRPO)

• GRPO is a reinforcement learning paradigm that normalizes rewards across candidate groups to compute relative advantages and stabilize policy updates.
• It extends PPO by incorporating group-wise clipping, trust regions, and optional KL regularization, enabling robust fine-tuning in LLMs and deterministic models.
• Empirical results demonstrate that GRPO improves convergence speed, accuracy, and reward balance across domains like vision, language, and multi-objective control.

Group Reward Policy Optimization (GRPO) is a reinforcement learning paradigm that generalizes Proximal Policy Optimization (PPO)-style clipped policy gradients to the setting where agent performance is measured not just by individual reward signals but by an advantage computed relative to a group of candidate behaviors. It is widely used for fine-tuning LLMs and, more recently, for post-training representation learning and multi-objective RL. The method's defining characteristics are its group-wise normalization of rewards to compute relative advantages, a critic-free architecture, a PPO-like trust region, and, optionally, KL regularization for stability.

1. Mathematical Foundations of GRPO

The canonical GRPO objective is formulated as follows. Let $q$ denote the "input" (e.g., a prompt or a data instance), and $\{o_1, \dots, o_G\}$ be a group of $G$ candidate outputs sampled from a reference or old policy $\pi_{\theta_\mathrm{old}}$. For each $o_i$, compute a reward $r_i$, then normalize within-group via

$$A_i = \frac{r_i - \operatorname{mean}(\{r_j\})}{\operatorname{std}(\{r_j\})}$$

to obtain a group-relative advantage. The policy is updated by maximizing the surrogate loss

$$J_{\mathrm{GRPO}}(\theta) = \mathbb{E}_{q, \{o_i\} \sim \pi_{\theta_\mathrm{old}}} \left[ \sum_{i=1}^G \pi_\theta(o_i|q) ~A_i - \beta D_{\mathrm{KL}}(\pi_\theta(\cdot|q) \| \pi_{\mathrm{ref}}(\cdot|q)) \right]$$

with $\beta \geq 0$ controlling the strength of KL regularization to a reference policy $\pi_\mathrm{ref}$ (Xu et al., 19 Nov 2025). The group-normalized advantage ensures that the credit assignment is always relative to the group's empirical performance—an affine-invariant transformation that mitigates global reward scaling issues.

In practice, GRPO applies a PPO-like clipped surrogate per token or per output: $$L = \frac{1}{G} \sum_{i=1}^G \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \min\left( \rho_{i, t}(\theta) A_i, \operatorname{clip}(\rho_{i, t}(\theta), 1-\epsilon, 1+\epsilon) A_i \right)$$ where $\rho_{i,t}$ is the importance ratio between current and old policies (Fontana et al., 8 Jan 2026, Xu et al., 19 Nov 2025).

2. Methodological Adaptations and Extensions

2.1 GRPO for Representation Models (GRPO-RM)

GRPO-RM extends the method from LLMs to deterministic representation models such as pretrained Vision Transformers (ViTs), where stochastic sampling over output embeddings is not available. To operationalize GRPO in this regime:

• The candidate group $\{o_1,\dots,o_c\}$ is defined as the finite set of all possible output classes.
• A softmax head produces class probabilities, and each class index is treated as a candidate output, establishing the group structure required for GRPO.
• Customized reward components are constructed: (a) an accuracy-based reward assigning a value $c$ for the correct class and 0 otherwise (ensuring $\mathbb{E}[\text{reward}] = 1$), and (b) a uniformity penalty proportional to $-p_i$ for each class to penalize overconfident wrong predictions. For segmentation with dominant background, further adjustment is applied to mitigate class imbalance.
• The GRPO loss is built from these rewards, with per-group normalization and a straightforward policy-gradient update (no KL, $\beta=0$).

This approach yields consistent improvements on standard classification and segmentation benchmarks, including a +3.75% gain in softmax regression accuracy and notable gains in out-of-distribution settings. Convergence to loss plateau is typically achieved in $\sim$20 epochs, much faster than standard fine-tuning (Xu et al., 19 Nov 2025).

2.2 GRPO in Multi-Objective RL

Directly summing multiple heterogeneous rewards before group normalization in GRPO leads to reward domination by the highest-variance objective, causing "reward hacking." MO-GRPO resolves this via variance-scaled normalization: each reward is separately normalized to zero mean and unit variance across the group, then summed, so that all objectives contribute equally. This prevents the update from being dominated by any single objective and mitigates empirically-observed pathological behaviors. MO-GRPO consistently outperforms vanilla GRPO across bandit, control, and sequence generation tasks with multiple objectives (Ichihara et al., 26 Sep 2025).

GDPO (Group reward-Decoupled Normalization Policy Optimization) further improves multi-objective optimization by normalizing each reward across the group before aggregation and rebasing the final sum at the batch level, leading to more fine-grained optimization signal and significantly improved convergence, particularly in tool-use and math/coding settings (Liu et al., 8 Jan 2026).

2.3 Relative Rewards and Ranking Models

Traditional GRPO’s reliance on absolute reward signals can produce signal sparsity or instability, especially when the group has identical reward outcomes or when the reward model's range drifts. RLRR (Reinforcement Learning with Relative Rewards) builds on GRPO by replacing absolute reward differences with intra-group relative rankings, leveraging either hybrid (HRR) or pure (PRR) ranking-based assignments. This approach ensures every group yields a non-trivial optimization signal, leads to bounded variance per Popoviciu’s inequality, and improves data efficiency and accuracy on both verifiable (math) and open-ended (writing) benchmarks (Niu et al., 30 Jan 2026).

Listwise ranking reward models (Ranking RMs) further extend this by producing direct group-level orderings (permutations) as reward signals, optimizing a cross-entropy or Plackett–Luce surrogate over entire groups. This model improves the robustness of policy updates in group-based RL (Niu et al., 30 Jan 2026).

3. Empirical Performance, Applications, and Training Dynamics

GRPO and its extensions have demonstrated broad empirical success in various domains:

Domain	Architecture	Key Dataset(s)	Key Metrics	GRPO Gains Over Baseline
LLM Posttraining	Llama/DeepSeek	Reasoning/math (GSM8K/AIME)	Pass@1, Pass@k	+2–12% absolute accuracy
Representation	ViT-S/14 (DINOv2)	CIFAR, ImageNet, VOC, ADE20k	Accuracy, mIoU	+3.7% SR (ImageNet), +0.6% mIoU
Multi-Agent	GAT+LLM (Graph-GRPO)	MMLU, HumanEval	Task accuracy	+1–2% (edge-level credit assign)
Robotics	Flow-matching U-Net	Unicycle control	Cost reduction	50–85% lower cost
Speech ASR	Llama3 + Conformer	People’s-Speech, Voxpopuli	WER	Up to 18.4% relative reduction

In LLMs, GRPO delivers high sample efficiency, rapid convergence, and robustness to reward scaling, especially when using verifiable or rule-based rewards. In small-resource settings (few rollouts per group), MC-GRPO, which centers advantages on the median rather than the mean, reduces sign-flip errors and improves stability, closing the low-rollout–high-rollout performance gap to within 1% (Kim, 30 Jan 2026). In multimodal reasoning and RL with difficult or sparse reward tasks, DIVA-GRPO and Scaf-GRPO use global difficulty assessment, adaptive variant/hint generation, or progressive guidance to sustain nonzero advantage variances and thus continuous learning (Gao et al., 1 Mar 2026, Zhang et al., 22 Oct 2025).

In domains with computable or classical evaluation metrics (e.g., WER in ASR), GRPO optimizes the metric directly, drastically decreasing hallucinations and improving out-of-domain transfer and adaptation without the need for a learned reward model (Shivakumar et al., 2 Sep 2025).

4. Theoretical Properties, Convergence, and Limitations

Theoretical studies of GRPO have characterized its gradient estimator as operating at the old policy, with rare bias as long as the policy snapshot is refreshed frequently; trajectory-level corrections (TIC-GRPO) yield unbiased updates and provable fast convergence (Pang et al., 4 Aug 2025). Surrogate objectives can exhibit structural biases if group-weighting schemes are non-uniform, notably gradient bias in shared prefixes for sequence models. This induces systematic length or output biases unless weighting is carefully controlled (Fontana et al., 8 Jan 2026). AdamW’s normalization can mask reward-scaling effects, and optimizer momentum can circumvent intended clipping constraints, requiring careful integration of regularization and trust-region mechanisms.

Reward variance is directly linked to training speed. Pre-adjusting sampled group rewards to increase within-group variance—without altering expectations or preference orderings—accelerates RLHF training efficiency, as implemented in the GRPOVI algorithm (Yang et al., 29 May 2025).

In multi-objective settings, vanilla GRPO is especially sensitive to variance imbalances: high-variance rewards dominate updates, leading to reward hacking and failure to optimize low-variance objectives. Variance-normalized extensions (MO-GRPO, GDPO) equalize each reward’s contribution, provably achieving balanced updates (Ichihara et al., 26 Sep 2025, Liu et al., 8 Jan 2026).

5. Practical Implementation and Generalization

GRPO methods are typically critic-free (no learned state-value or Q-value baselines), compute group-normalized advantages per prompt or batch, and apply PPO-like clipping with or without a KL penalty. The generic GRPO training loop for supervised or representation models is:

1. Sample or enumerate a group of candidate outputs per data input.
2. Compute per-candidate rewards (problem-specific, possibly composite).
3. Normalize rewards to compute group-relative advantages.
4. Apply a policy-gradient or surrogate objective update with importance weighting and trust-region enforcement.
5. Optionally update the reference or old policy every few steps.

For representation models, outputs are derived from the finite label space, not a generative model; reward functions are decomposed into accuracy, uniformity, or other task-relevant structure; and the entire candidate set is used for the group (Xu et al., 19 Nov 2025).

Generalization guidelines (Xu et al., 19 Nov 2025) for adapting GRPO:

• Define a fixed candidate set (labels, retrieval items, prototypes) matching downstream task constraints.
• Attach a probabilistic or scoring head to recover the necessary $\pi_\text{old}$ and $\pi_\theta$ values per candidate.
• Design reward functions reflecting the model and application (e.g., coverage, alignment, diversity, calibration).
• Tailor normalization, clipping, and advantage computation to the output structure and computational constraints.

6. Extensions, Open Problems, and Research Directions

Several recent extensions further address the limitations and unlock new domains for GRPO:

• RLRR and Ranking RM mitigate reward-signal sparsity and instability, suggesting generalization to non-numerical or highly structured feedback scenarios (Niu et al., 30 Jan 2026).
• Median-centered MC-GRPO, difficulty-adaptive DIVA-GRPO, and scaffolded Scaf-GRPO target challenge-specific pathologies, such as high-variance baselines, reward collapse on too-hard instances, and learning cliffs that render models blind to unsolved problems.
• For multi-objective or highly imbalanced rewards, methods such as MO-GRPO and GDPO ensure each objective’s influence remains commensurate and update directions remain balanced.
• Theoretical work on the alignment objective of GRPO demonstrates its non-logarithmic, rational pooling of preferences, highlighting key differences from RLHF-style exponential-weighted updates and yielding closed-form stationary solutions under pairwise or binary reward scenarios (Vojnovic et al., 25 Feb 2025).

Open challenges include: maintaining stable learning signals in adversarial or highly imbalanced reward settings; extending relative-reward and ranking frameworks to broader domains; understanding the impact of group size and candidate selection; and integrating GRPO with value-based or critic-augmented RL for settings where temporal credit assignment is required.

In sum, Group Reward Policy Optimization constitutes a flexible, sample-efficient, and theoretically principled framework for RL with group-structured feedback, with broad applicability to LLM alignment, representation learning, multi-objective RL, structured prediction, and beyond, and continues to evolve via domain- and objective-specific methodological advances (Xu et al., 19 Nov 2025, Ichihara et al., 26 Sep 2025, Liu et al., 8 Jan 2026, Kim, 30 Jan 2026, Niu et al., 30 Jan 2026, Fontana et al., 8 Jan 2026, Pang et al., 4 Aug 2025).