Group-Normalized Advantages (GRPO) in RL

• Group-Normalized Advantages (GRPO) are variance-normalization techniques that standardize rewards within groups to ensure scale invariance and reduce learning noise.
• They extend the PPO framework by replacing critic-based advantage estimation with empirical, within-group normalization, leading to faster convergence and improved stability.
• GRPO methods are widely applied in multi-agent, multi-objective, and LLM training scenarios, though they require careful adaptation to avoid issues like advantage collapse in sparse or heterogeneous reward settings.

Group-Normalized Advantages (GRPO) are a family of variance-normalization techniques for on-policy reinforcement learning, foundational to modern post-training of LLMs and policy learning in preference-alignment, multi-agent, and multi-objective scenarios. GRPO extends the canonical Proximal Policy Optimization (PPO) framework by replacing critic-based advantage estimation with empirical, within-group normalization of scalar rewards, making the learning signal invariant to scale, reducing variance, and critically, encoding peer-relative learning dynamics. Although widely adopted for tasks requiring efficient, critic-free RL updates—such as reasoning benchmarks, text-to-image alignment, multi-agent teamwork, and heterogeneous preference alignment—group-normalized advantages exhibit subtle pathologies and require careful adaptation in settings involving reward heterogeneity, sparse feedback, or multiple objectives.

1. Formal Definition of Group-Normalized Advantage

Let $G$ denote a group of $G$ trajectories ("completions") generated by a policy $\pi_\theta$ for a common input (e.g., prompt, initial state, or task). Each trajectory $i$ receives a scalar reward $R_i$. The group-normalized advantage for trajectory $i$ is defined as

$$A^g_{i,t} = \frac{R_i - \mu_g}{\sigma_g + \epsilon}$$

where

$$\mu_g = \frac{1}{G} \sum_{j=1}^G R_j, \qquad \sigma_g = \sqrt{\,\frac{1}{G} \sum_{j=1}^G (R_j - \mu_g)^2\,}$$

and $\epsilon>0$ is a small constant for numerical stability. $A^g_{i,t}$ is typically assigned to every token position $G$0 in trajectory $G$1 (sequence-level update), but per-token refinements exist.

The GRPO policy objective utilizes these advantages in a PPO-style clipped surrogate loss, usually augmented with a per-token KL penalty to a reference policy:

$G$2

with $G$3 and final loss $G$4 averaging $G$5 over all group tokens (Wang et al., 17 Feb 2026).

2. Theoretical Properties and Motivation

The rationale for group normalization is twofold: variance reduction and scale invariance. By centering and scaling rewards within group batches, GRPO implements a form of adaptive learning rate proportional to the inverse local curvature (Fisher information or policy Hessian), thus accelerating convergence over unnormalized REINFORCE (by a factor equal to the average within-group reward standard deviation) (Ge et al., 30 Jan 2026). Empirically, this yields faster and smoother training of LLMs on mathematical reasoning, multi-agent RL, and multimodal alignment tasks.

In multi-agent and multi-objective settings, group normalization guarantees that the learning signal has comparable magnitudes across agents and objectives, preventing domination by one agent or reward component and yielding robust scaling (Feng et al., 21 Apr 2026, Ichihara et al., 26 Sep 2025). GRPO allows critic-free operation, eliminating the need for learned value functions (which can be unstable or biased in LLM and multi-agent settings).

3. Limitations: Biases, Degeneracy, and Signal Collapse

Despite its simplicity and stability, standard group-normalized advantages introduce several failure modes:

• Exchangeability and Reward Heterogeneity: Standard GRPO assumes all group samples are exchangeable. If reward distributions differ markedly across user populations (e.g., "short answer" vs. "elaborate answer" preferences), within-batch normalization yields an implicit bias toward dominant or low-variance groups, suppressing minority or high-variance groups (Wang et al., 17 Feb 2026). This statistical shrinkage yields attenuated gradients for underrepresented preferences and hinders faithful personalization.
• Advantage Collapse in Sparse/Binary Rewards: In settings with low within-group reward variance (e.g., after strong supervised finetuning or with binary rewards), most groups become degenerate (all-correct or all-incorrect), $G$6, resulting in $G$7 for all $G$8 (so-called "gradient starvation" or "advantage collapse") (Nie et al., 8 May 2026, He et al., 20 May 2026). Empirical degeneracy rates can exceed 70% for group size 4 in LLM reasoning tasks, causing most updates to vanish.
• Multi-Objective Collapse: For multiple rewards, normalizing after summing reward components leads to collapse—i.e., distinct reward combinations reduce to a handful of possible normalized advantages per group, drastically reducing gradient resolution and hindering optimization on secondary objectives (Liu et al., 8 Jan 2026, Ichihara et al., 26 Sep 2025, Lyu et al., 30 Nov 2025) [Multi-GRPO].
• Sequence-Level Uniformity and Length Bias: GRPO typically applies a trajectory-level advantage to all tokens in a sequence. This induces length bias, dilutes penalties on long bad solutions, and discards potential credit assignment to intermediate reasoning steps (Cao et al., 7 Jan 2026, Lyu et al., 30 Nov 2025).
• Exploration and Symmetry Limitations: The GRPO group-normalization enforces a strict symmetry between "good" and "bad" trajectories, leading to an exploration bottleneck (unsampled action logits remain unchanged) and bias toward medium-difficulty samples (Yu et al., 5 Feb 2026).

4. Extensions and Remedies

A range of variants address these pathologies:

Variant	Core Fix	Targeted Limitation
P-GRPO	Normalize against preference-group stats	Reward heterogeneity, personalization
AVSPO (He et al., 20 May 2026)	Injects virtual rewards in degenerate groups	Advantage collapse, binary rewards
Sign Advantage (Nie et al., 8 May 2026)	Non-relative baseline ($G$9)	Gradient starvation in binary regime
TreeAdv (Cao et al., 7 Jan 2026)	Redistributes advantage along shared trees	Credit assignment, length bias
MO-GRPO (Ichihara et al., 26 Sep 2025)	Per-reward normalization, auto-reweighting	Multi-objective collapse
Multi-GRPO (Lyu et al., 30 Nov 2025)	Temporal, reward-based grouping	Multi-objective, temporal credit
DIVA-GRPO (Gao et al., 1 Mar 2026)	Difficulty-adaptive variant balancing	Advantage vanishing, stability
EP-GRPO (Yu et al., 6 May 2026)	Entropy/progress-aligned token advantage	Token credit assignment, collapse
AMIR-GRPO (Yari et al., 7 Jan 2026)	Implicit DPO-style regularizer	Length bias, preference recall
PAPO (Tan et al., 27 Mar 2026)	Decoupled normalization (outcome/process)	Rubric reward hacking, ORM stalling
GDPO (Liu et al., 8 Jan 2026)	Reward-decoupled normalization	Multi-reward collapse

Personalized GRPO (P-GRPO)

P-GRPO (Wang et al., 17 Feb 2026) replaces within-batch normalization with running historical statistics ($\pi_\theta$0, $\pi_\theta$1) for each preference group $\pi_\theta$2 (maintained online). The advantage becomes

$\pi_\theta$3

This decouples the learning signal from instantaneously dominant reward distributions, yielding faster convergence, higher final accuracy in recommender system and generative benchmarks, and robust recovery of minority preference modes. Empirically, fine-grained clustering for historical reward statistics is crucial; randomized or coarse clusters eliminate gains.

Advantage Collapse Metrics and Remedies

The Advantage Collapse Rate (ACR) (He et al., 20 May 2026) quantifies the proportion of groups with degenerate variance ($\pi_\theta$4). ACR strongly predicts training stagnation and final performance. AVSPO injects stratified virtual reward samples into homogeneous groups, restoring advantage variance and yielding 4–9 point accuracy gains across LLM scales. Sign-advantage fixes (using non-relative $\pi_\theta$5) directly avoid degeneracy for binary rewards, aligning the gradient with pass@G and yielding dramatic gains in math-reasoning (e.g., +45pp on GSM8K test at $\pi_\theta$6).

Multi-Objective and Multi-Reward Adaptations

MO-GRPO (Ichihara et al., 26 Sep 2025) and GDPO (Liu et al., 8 Jan 2026) decouple normalization to the reward level: each objective $\pi_\theta$7 is normalized across the group, $\pi_\theta$8, and the final advantage summed over objectives. This ensures each objective contributes with balanced weight ($\pi_\theta$9 per objective for $i$0 objectives), eliminates domination by high-variance rewards, and preserves preference order. Multi-GRPO (Lyu et al., 30 Nov 2025) further orthogonalizes temporal (tree-based) and reward-based grouping to enable fine-grained credit assignment in text-to-image generation.

Sequence/Token-Level Credit Assignment

TreeAdv (Cao et al., 7 Jan 2026) and related methods build an explicit tree of shared prefixes among group rollouts and redistribute leaf-level group-normalized advantages back to tokens along shared segments, overcoming the sample inefficiency and length bias of sequence-wide uniform updates.

Entropy/Process-Guided and Difficulty-Adaptive Variants

Extensions such as EDGE-GRPO (Zhang et al., 29 Jul 2025), EP-GRPO (Yu et al., 6 May 2026), and DIVA-GRPO (Gao et al., 1 Mar 2026) leverage entropy-driven weighting, implicit policy divergence, and curriculum-style sampling to maintain gradient flow under reward sparsity, fix polarity misalignment, and adaptively target optimal correct/wrong sample balances for robust training.

5. Empirical Outcomes and Benchmarks

These group-normalized advantage techniques underpin most state-of-the-art RL fine-tuning results for LLMs, MLLMs, and generative models:

• Personalization and minority recovery: P-GRPO (MovieLens-1M, Gemma-2B, Qwen3-8B) achieves higher top-1 accuracy and faster convergence vs. GRPO. Fine-grained cluster tracking is essential (Wang et al., 17 Feb 2026).
• Binary reward regimes: Sign advantage and AVSPO yield up to +45pp gains at small group sizes (Nie et al., 8 May 2026, He et al., 20 May 2026).
• Mathematical reasoning: EP-GRPO boosts average accuracy by 26–12% (Qwen2.5-3B/7B; MATH500/AMC23/AIME24) over plain GRPO (Yu et al., 6 May 2026).
• Multi-objective settings: MO-GRPO achieves balanced optimization in machine translation and bandit/control tasks; vanilla GRPO otherwise collapses length/format constraints (Ichihara et al., 26 Sep 2025).
• Token-level assessment: TreeAdv raises Pass@1 and improves sample efficiency (–10–30% tokens per solution) (Cao et al., 7 Jan 2026).
• Multimodal reasoning: DIVA-GRPO is consistently SOTA among open 7B-scale models, with average accuracy +8.2% over backbone baselines (Gao et al., 1 Mar 2026).
• Preference supervision: AMIR-GRPO tightens decision margins and improves Pass@1/4 by 2–12pp on reasoning benchmarks (Yari et al., 7 Jan 2026).
• Process-aware optimization: PAPO continues improvement in correctness and reasoning quality after ORM baselines stall, demonstrating improved signal utilization (Tan et al., 27 Mar 2026).

6. Algorithmic Integration and Computational Aspects

The standard integration workflow for group-normalized advantages is as follows:

1. For each prompt (plus optional user preference):
   1. Sample $i$1 completions, compute scalar rewards $i$2.
   2. Compute group mean $i$3 and standard deviation $i$4.
   3. Assign normalized advantages to each trajectory (possibly via preference- or reward-based partitions).
   4. Compute importance sampling ratios and PPO-style surrogates per token.
   5. Average losses, backpropagate, and update parameters.

Extensions may incorporate group history statistics (P-GRPO), auxiliary regularizers (AMIR-GRPO), or tree-based reward backup (TreeAdv, Multi-GRPO). Most methods introduce negligible computational overhead beyond the group batch structure, and empirical results consistently support strong stability and convergence across architectures and domains.

7. Connections, Broader Implications, and Open Directions

Group-normalized advantages form the backbone of RL from human (or verifiable) feedback in LLM alignment, reasoning chain induction, cooperative multi-agent settings, and controllable generation. Their core statistical principle—anchoring learning to peer-relative reward signals—enables critic-free, scalable, and often hardware-efficient RL pipelines. However, sustained empirical experience demonstrates that their naive application yields systematic limitations in the presence of heterogeneity, reward degeneracy, and multi-objective trade-offs.

Recent research emphasizes the importance of:

• Separating advantage estimation from batch instability (e.g., P-GRPO, AVSPO).
• Adapting the normalization granularity to the relevant group (preference, objective, process step).
• Leveraging structural context (e.g., tree or temporal grouping) for credit assignment and variance reduction.
• Integrating process-level or implicit signals to remedy reward hacking and vanishing gradient scenarios.

The development of robust, general-purpose group-normalized advantage frameworks continues to structure the field’s understanding of efficient policy optimization under weak, sparse, or heterogeneous reward feedback (Wang et al., 17 Feb 2026, Nie et al., 8 May 2026, Cao et al., 7 Jan 2026, Yu et al., 5 Feb 2026, Gao et al., 1 Mar 2026, Wang et al., 28 Aug 2025, Ge et al., 30 Jan 2026, Liu et al., 8 Jan 2026, Lyu et al., 30 Nov 2025).