Group-wise REINFORCE (GPG) in Policy Gradient Methods

• Group-wise REINFORCE (GPG) is a critic-free policy-gradient estimator that normalizes rewards within groups to reduce variance and ensure effective credit assignment.
• It aggregates intra-group performance differences to simplify RL optimization and lower computational overhead compared to traditional methods like PPO and GRPO.
• Enhancements such as relative reward shaping and focal weighting stabilize training and improve data efficiency in large language models and MDP tasks.

Group-wise REINFORCE (GPG), also known as group-based policy gradient or GRPO in the literature, is a family of critic-free policy-gradient estimators designed to optimize policies by leveraging intra-group performance differences. By aggregating and normalizing rewards within groups of sampled trajectories, GPG achieves variance reduction and effective credit assignment, particularly for LLMs and general Markov Decision Processes (MDPs). Recent advances have further extended GPG through relative reward shaping and difficulty-aware weighting, providing stability and improved data-efficiency across tasks with both verifiable and open-ended feedback.

1. Core Principle and Mathematical Formulation

GPG optimizes the standard reinforcement learning objective $$\mathcal{J}(\theta) = \mathbb{E}_{\tau\sim\pi_\theta}[R(\tau)]$$ by employing a group-wise baseline strategy to estimate policy gradients without reliance on a learnable value function or critic network. For each prompt or initial state $p$, a group of $G$ trajectories ('responses' in LLM RL) $\{o_1,\ldots,o_G\}\sim\pi_\theta(p)$ is sampled, each receiving scalar reward $s_k$ from a verifier or reward model. The intra-group normalized advantage for each trajectory is computed as

$$\hat A_k = \frac{s_k - \mu_G}{\sigma_G}, \quad \mu_G = \frac{1}{G}\sum_{j=1}^G s_j, \quad \sigma_G = \mathrm{std}(\{s_j\}_{j=1}^G)$$

The policy is updated using a PPO-style clipped surrogate objective:

$$\mathcal{J}_{\text{GPG}}(\theta) = \frac{1}{G}\sum_{k=1}^G\frac{1}{|o_k|}\sum_{t=1}^{|o_k|} \min\left( \rho_{k,t}(\theta)\,\hat A_k,\;\mathrm{clip}(\rho_{k,t}(\theta),1-\epsilon,1+\epsilon)\,\hat A_k \right)$$

where $$\rho_{k,t}(\theta) = \pi_\theta(o_{k,t}|s_t)/\pi_{\theta_{\text{old}}}(o_{k,t}|s_t)$$. First-order expansion yields a group-based policy-gradient estimator:

$$\nabla_\theta \mathcal{J}(\theta) \approx \mathbb{E}_{\{o_k\}}\left[\sum_{k=1}^G\sum_{t=1}^{|o_k|} \nabla_\theta\log\pi_\theta(o_{k,t}|s_t)\;\hat A_k\right]$$

This structure enables direct optimization of the RL objective, simplifying implementation, and reducing computational overhead by eliminating the critic and reference policy dependencies (Chu et al., 3 Apr 2025, Chen et al., 4 Oct 2025, Niu et al., 30 Jan 2026).

2. Comparison to Related Baselines and Variants

GPG generalizes and subsumes a spectrum of group-based RL algorithms:

• PPO: Utilizes a value-function critic for advantage estimation and typically a clipped surrogate loss.
• GRPO (Group Relative Policy Optimization): A formulation where the group mean serves as the baseline, and policy updates are regularized using clipped importance weights and sometimes KL-divergence penalties.
• Direct Preference Optimization (DPO): As shown in "It Takes Two: Your GRPO Is Secretly DPO" (Wu et al., 1 Oct 2025), GRPO (and therefore GPG) can be reinterpreted as a contrastive learning objective, with the 2-rollout case (2-GRPO) being equivalent, in gradient structure, to DPO with a pairwise preference loss.
• F-GRPO: Extends GRPO with a focal-loss inspired scaling, down-weighting updates on easy prompts to preserve coverage of rare-correct trajectories for group-based RL with verifiable rewards (Plyusov et al., 6 Feb 2026).

Components and required elements for each method are as follows:

Method	Value Model	Reference Model	KL/Surrogate	Group Baseline
PPO	✓	✓	✓	x
GRPO	x	✓	✓	mean
GPG	x	x	x	mean/std

GPG is distinct in requiring only intra-group statistics and observed rewards, needing no reference policy or value head (Chen et al., 4 Oct 2025, Chu et al., 3 Apr 2025).

3. Theoretical Guarantees and Bias–Variance Analysis

The theoretical consistency of GPG arises from its group-based Monte Carlo baseline structure (Chen et al., 4 Oct 2025). As group size $N\to\infty$, the empirical group-based advantage estimator converges to the true on-policy gradient:

• No value function approximation bias is introduced, and variance reduction is obtained for free using parallel group sampling.
• The choice of group size and intra-group normalization (standardization within each group) facilitates a trade-off between variance reduction and potential introduction of binning bias if group granularity is increased excessively.
• The 2-rollout (2-GRPO) configuration achieves unbiased gradients and matches sample complexity and asymptotic convergence rate with larger group sizes, provided prompt batch size is scaled appropriately (Wu et al., 1 Oct 2025).

4. Reward Shaping with Relative Rankings (RLRR)

Absolute group-based normalization can be fundamentally limited in two regimes:

• Sparse feedback (verifiable tasks): When model proficiency causes groups to be uniformly correct or incorrect, $\sigma_G\rightarrow 0$ and $\hat A_k\rightarrow 0$, resulting in 'silent' updates and no learning signal.
• Unbounded/unstable reward ranges (open-ended tasks): If score magnitudes drift, the normalization introduces excessive gradient noise.

RLRR addresses these with two main approaches (Niu et al., 30 Jan 2026):

• Hybrid Relative Reward (HRR): For verifiable tasks, augment binary correctness with a rank-based, bounded bonus: $$s_k^{\text{HRR}} = s_k^{\text{rule}} + \tau \tanh(r_{\max} + 1 - 2r_k)$$.
• Pure Relative Reward (PRR): For open-ended feedback, use a normalized linear mapping from intra-group rank: $s_k^{\text{PRR}} = (r_{\max} - r_k)/(r_{\max} - 1)$.
• Ranking Reward Model (Ranking RM): Trains a listwise model to output joint group rankings, reducing sensitivity to reward-model scale and improving stability.
• Correctness-Aware Clipping: Bounds advantage magnitude for correct/incorrect responses by explicit clipping, ensuring correct responses are not disproportionately penalized.

Empirically, RLRR leads to higher data efficiency—effective gradient updates are possible on 100% of prompts versus $\lesssim$40% with traditional GRPO on easy tasks. Relative ranking ensures bounded-variance signals regardless of reward model drift, achieving consistent gains across both reasoning and open-ended generation (Niu et al., 30 Jan 2026).

5. Practical Implementation and Empirical Performance

The canonical GPG (or group-wise REINFORCE) algorithm operates as follows:

for each update step: for each prompt p in batch: sample G responses {o₁…o_G} ~ π_old(p) compute rewards {s_k} μ_G ← mean(s), σ_G ← std(s) (optionally =1) for k in 1...G: ŴA_k ← (s_k−μ_G)/σ_G # Optionally, apply correctness-aware clipping update θ using Eq.(1) with ŴA_k

When employing RLRR, groups are ranked using the ranking model, and shaped rewards are used for advantage computation with optional correctness-aware clipping (Niu et al., 30 Jan 2026).

Empirical results:

• On mathematical reasoning (e.g., DeepSeek‐Qwen‐1.5B, LLaMA-8B), GPG and RLRR consistently surpass standard GRPO on pass@k accuracy and data efficiency, with lower per-update token usage (Niu et al., 30 Jan 2026, Chu et al., 3 Apr 2025).
• On open-ended text generation, PRR and ranking models improve overall reward scores by +0.7 to +1.8 on WritingBench, and relative methods maintain stability as scalar reward models drift.
• On multimodal benchmarks, GPG outperforms both PPO and GRPO baselines, with peak GPU memory usage reduced by $\sim$30% and wall-clock training time per step lowered by $\sim$25% (Chu et al., 3 Apr 2025).
• F-GRPO controls the non-monotonic behavior in rare-mode tail-miss by down-weighting high-success prompts, preserving rare-correct solution modes at much smaller group sizes (Plyusov et al., 6 Feb 2026).

6. Guidelines, Pitfalls, and Extensions

Key practical recommendations:

• Optimal group size is hardware- and task-dependent; for LLM RL, $G=2$ suffices to match traditional $G=16$ with appropriate batch scaling (Wu et al., 1 Oct 2025).
• For stability in off-policy pipelines, regularization (clipping, KL, or secondary penalties) is more critical than raw importance weighting; extending clipping thresholds can accelerate convergence without collapses (Yao et al., 29 Sep 2025).
• Pairwise or listwise relative reward models further improve robustness to reward drift and batch skew.
• For rare-mode coverage, focal-style weighting (F-GRPO) increases pass rates for rare-correct regions without computational burden (Plyusov et al., 6 Feb 2026).
• When using fine bins for state aggregation, ensure group size is sufficient to avoid bin-scattering bias (Chen et al., 4 Oct 2025).
• Integration with more complex curricula or adaptive group sizes is plausible, but the empirical benefit depends on the specific distribution of rewards and model proficiency stage.

When to prefer GPG and its enhancements:

• RL applications where learned state-value models are unstable, expensive, or introduce additional hyperparameter complexity.
• Environments with massive parallel rollout/sample generation capacity, where leveraging observed return statistics directly is tractable.
• LLM post-training workflows with verifiable or noisy reward models, or where maintaining rare-mode coverage is crucial for generalization.
• Tasks with open-ended or drift-prone reward models, where relative ranking is necessary for stability (Niu et al., 30 Jan 2026, Chu et al., 3 Apr 2025).

7. Perspectives and Current Research Directions

The group-wise REINFORCE paradigm has catalyzed a unification of policy-gradient, contrastive, and preference-based RL for large models. Off-policy reinterpretations show GPG (and GRPO) induce implicit KL-regularized surrogates, with robust empirical performance extending to data-driven regularization and weighting schemes (Yao et al., 29 Sep 2025, Wu et al., 1 Oct 2025). Active research explores:

• Further generalization of group-based methods to richer action/state spaces and more complex group structures (Chen et al., 4 Oct 2025).
• Integration with online MIRROR-DESCENT, asymmetric baselines, and adaptive focal weighting for tailored sample efficiency and coverage (Yao et al., 29 Sep 2025, Plyusov et al., 6 Feb 2026).
• Design of ranking models that learn to produce reliable intra-group and global group orderings in dynamic or adversarial settings (Niu et al., 30 Jan 2026).
• Extensions to multiturn or memory-augmented tasks, where trajectory-wide or chain-of-thought rationales must be jointly ranked or normalized.

GPG, reinforced by relative ranking and difficulty-aware techniques, thus provides a high-performance, scalable, and theoretically-grounded framework for RL optimization in modern model reasoning and generation pipelines.