--- title: 'Group-Relative REINFORCE: Enhancing Policy Gradients' url: https://www.emergentmind.com/topics/group-relative-reinforce type: topic --- # Group-Relative REINFORCE: Enhancing Policy Gradients Group-Relative REINFORCE denotes a family of policy-gradient methods in which a policy samples multiple rollouts for the same prompt or state, computes a within-group relative reward signal, and updates the policy without relying on a learned value critic. In contemporary LLM post-training, this family is most commonly instantiated as Group Relative Policy Optimization (GRPO): a PPO-style clipped surrogate whose core gradient is a REINFORCE update with a group-relative baseline, usually the group mean or a normalized variant. Recent work has recast the method as a broader framework encompassing on-policy and off-policy training, robust advantage estimators, divergence-based objectives, ranking-based reward shaping, and application-specific adaptations in reasoning, instruction following, multimodal generation, and long-video context selection [2505.17574][2509.24203]. ## 1. Canonical formulation Classical REINFORCE optimizes the expected return \[ J(\theta)=\mathbb E_{\tau\sim \pi_\theta}[R(\tau)], \] with gradient \[ \nabla_\theta J(\theta)=\mathbb E_{\tau}\Bigl[R(\tau)\sum_t \nabla_\theta \log \pi_\theta(a_t\mid s_t)\Bigr]. \] Group-relative methods modify only the baseline construction. For a prompt \(x\), the policy samples a group of \(G\) or \(K\) rollouts \(y_1,\dots,y_G\), obtains scalar rewards \(r_i=r(x,y_i)\), and replaces a learned critic by an intra-group baseline such as the group mean \(\bar r=\frac1G\sum_i r_i\). The raw group-relative advantage is \(A_i=r_i-\bar r\); many implementations further standardize it as \[ A_i=\frac{r_i-\bar r}{\sigma+\epsilon},\qquad \sigma=\sqrt{\frac1G\sum_j (r_j-\bar r)^2}. \] The resulting REINFORCE-style update is \[ \nabla_\theta J \approx \frac1G\sum_{i=1}^G A_i \nabla_\theta \log \pi_\theta(y_i\mid x), \] or, in tokenized autoregressive form, the same scalar \(A_i\) is attached to every token of rollout \(i\) [2505.17574][2605.04960]. In deployed GRPO systems this gradient is usually embedded in a PPO-style clipped surrogate. Writing \[ \rho_{i,t}(\theta)=\frac{\pi_\theta(y_{i,t}\mid x,y_{i,0.5\), and is unbiased only at \(p_t=0.5\). For typical small group sizes \(G\le 8\), the probability of underestimation for hard prompts or overestimation for easy prompts exceeds \(0.63\), and it exceeds \(0.78\) in more extreme regimes [2601.08521]. A second issue is rare-mode omission. If \(\tau\) denotes the policy mass on a rare-correct subtype and \(\mu_{\rm pos}\) the overall success probability, then the probability that a group update is active while sampling no rare-correct trajectory is \[ \Pr[\text{active}\wedge \text{no rare}] =(1-\tau)^K-(\mu_{\rm pos}-\tau)^K-(1-\mu_{\rm pos})^K. \] This quantity is non-monotonic in group size. Under a one-step TRPO-style surrogate analysis, unsampled-but-correct mass \(Q_{u,\mathrm{pos}}\) can shrink even while total correct mass grows, implying that learning may concentrate probability on already common solutions while forgetting rare-correct ones [2602.06717]. A third line of analysis concerns what one paper terms Group Relative Advantage Estimation. Standardized group advantages satisfy \[ \sum_{i=1}^G A_i=0, \] and, for binary rewards, the total positive and negative absolute weights are exactly matched. If \(p=|\{i:r_i=1\}|/G\), then \[ \sum_{i=1}^G |A_i| = 2G\sqrt{p(1-p)}, \] which is symmetric under \(p\mapsto 1-p\) and maximized at \(p=\tfrac12\). This induces two limitations: unsampled logits are unchanged, so the update cannot actively push probability mass toward unseen correct paths, and the method implicitly prioritizes medium-difficulty groups rather than adapting difficulty focus over training [2602.05548]. Further failure modes arise in small-rollout or low-dispersion regimes. Noise in the shared mean baseline can cause advantage sign flips; relative to a \(G_{\rm ref}=128\) oracle baseline, sign-flip rates rise to \(20\)–\(30\%\) when \(G\in\{2,4\}\), and a causal injection experiment shows that even a \(5\%\) sign-flip rate can degrade final accuracy by \(\sim 4\%\) [2601.22582]. In discrete multi-constraint settings, z-score normalization additionally exhibits low-variance amplification, mean-centering blindness, and zero-variance collapse: if all group rewards are identical, \(\sigma_{\rm grp}=0\) and the policy gradient vanishes [2606.06058]. ## 4. Advantage redesign and robust variants Proposed remedies largely preserve the group-relative backbone while modifying the baseline, reward shaping, or weighting function. Several representative variants are summarized below. | Variant | Core modification | Reported effect | |---|---|---| | MC-GRPO | Median baseline, MAD scaling, \(G+1\) samples, drop zero-advantage pivot | Reduces the gap between \(G=2\) and \(G=8\) to within \(1\%\) | | F-GRPO | Focal weight \(g(x)=(1-\hat\mu_{\rm pos})^\gamma\) on group-relative advantages | Pass@256 gains of \(3\)–\(6\) points at \(K=8\) | | HA-DW | History-aware weight \(\Phi_{t,i}=\lambda_{\mathrm{scale}}\exp(D_{t,i}M_t)\) | Average gains of \(+2.2\) to \(+2.9\) on Qwen3-4B/8B GRPO | | A-GRAE | Dynamic difficulty shift and asymmetric suppression of positive advantages | Improves GRPO and variants across seven benchmarks | | RLRR | Rank-based PRR/HRR relative rewards with Ranking Reward Model | \(+1\)–\(3\) pp over absolute-score baselines | | MDP-GRPO | Multi-temperature sampling, dual-anchor advantages, prospect shaping, asymmetric KL | Up to \(5.0\%\) strict-constraint improvement | MC-GRPO replaces the mean baseline by the sample median \[ b_{\rm med}(q)=\mathrm{median}\{r_1,\dots,r_{G+1}\}, \] uses the median absolute deviation for scaling, and excludes the unique pivot sample with zero advantage from backpropagation. The stated motivation is robustness to outliers and baseline-induced sign flips under small rollout budgets [2601.22582]. F-GRPO instead retains mean-based group-relative structure but multiplies the advantage by a Focal-loss–inspired coefficient \(g(x)=(1-\hat\mu_{\rm pos})^\gamma\), down-weighting prompts with high empirical success and thereby counteracting concentration on already easy cases [2602.06717]. HA-DW introduces an evolving difficulty anchor \(C_t\) and reweights each \(\hat A_i\) by a factor that amplifies under-estimated advantages on hard prompts and suppresses over-exploited actions on easy prompts [2601.08521]. Other proposals explicitly break symmetry or move from absolute scores to relative orderings. A-GRAE combines a sample-level dynamic difficulty shift with group-level asymmetric suppression of positive advantages, motivated by the claim that asymmetrically suppressing correct trajectories encourages exploration and that learning efficiency is maximized by an easy-to-hard transition over training [2602.05548]. RLRR replaces absolute numerical rewards by relative rankings, either via Pure Relative Reward, \[ R_i^{\rm rel}=\frac{G-r_i}{G-1}, \] or Hybrid Relative Reward, \[ R_i^{\rm rel}=s_i^{\rm abs}+\tau \tanh\!\Bigl(\frac{G-r_i}{G-1}\Bigr), \] and supplements this with a listwise Ranking Reward Model that predicts intra-group orderings directly [2601.23058]. MDP-GRPO addresses discrete low-dispersion rewards through multi-temperature sampling, a mixed advantage \[ A_i=(1-\alpha)z_i+\alpha \delta_i, \qquad \delta_i=2\sqrt{C(x)}(r_i-0.5), \] prospect-theoretic shaping, and asymmetric KL penalties [2606.06058]. Two broader generalizations alter the scalar baseline paradigm itself. LambdaPO defines a pairwise preference advantage \[ A_\Lambda(\tau_i)=\sum_{j\neq i} w_{ij}\,[r(\tau_i)-r(\tau_j)], \qquad w_{ij}=\sigma((s_i-s_j)/T), \] and augments sparse binary rewards with a semantic density reward derived from ROUGE-L F\(_1\) against a ground-truth solution trace [2605.19416]. f-GRPO replaces the standard ratio-times-advantage term with a variational \(f\)-divergence estimator between above-average and below-average reward distributions, yielding a class of divergence-based on-policy objectives with theoretical average-reward improvement guarantees [2602.05946]. ## 5. Credit assignment, geometry, and rollout efficiency Another strand of work argues that the main deficiency is not only the baseline but the granularity of credit assignment. EP-GRPO identifies three failures in vanilla GRPO: uniform token-level granularity, uniform polarity, and zero-variance collapse. Empirically, randomly replacing the top \(5\%\) highest-entropy tokens in correct solutions caused a \(3.5\times\) larger accuracy drop than perturbing the bottom \(5\%\); on 500 MATH problems, \(6.6\%\) of locally incorrect steps in correct sequences were rewarded and \(51.5\%\) of locally correct steps in incorrect sequences were penalized; and in a standard run on MATH with \(G=8\), \(58.77\%\) of training steps had zero reward variance, wasting over \(46\) million token-updates. EP-GRPO adds entropy-gated outcome weighting, implicit process signals anchored by sequence outcome, and cumulative entropy mapping, combining them in a final token-level advantage \[ \hat A^{\rm final}_{i,t}=\hat A^{\rm out}_{i,t}+\hat A^{\rm prog}_{i,t}. \] The method is explicitly designed to maintain gradient flow even when sequence-level reward variance collapses [2605.04960]. AERO addresses a different failure: wasted rollout compute when fixed-size groups are homogeneous. It partitions prompts after an exploration phase into rescue, partial, and high-success subsets, performs iterative rescue sampling for hard prompts, rejection-based pruning for mixed groups, and Bayesian posterior stabilization for all-correct or all-incorrect groups. With a \(\mathrm{Beta}(1,1)\) prior, the posterior mean \[ \tilde u=\frac{c+1}{n+2} \] replaces the degenerate empirical success rate in homogeneous groups, yielding nonzero standardized advantages. Under the same total rollout budget, AERO reports about \(48\%\) lower total training compute and about \(45\%\) lower wall-clock time per step on average while matching or improving Pass@8 and Avg@8 over GRPO [2602.14338]. SALT analyzes why simply increasing the number of rollouts often fails to strengthen learning. The stated diagnosis is that per-rollout policy-gradient features concentrate into a low-rank, signed geometry, so group-relative coefficients cancel in a dominant shared subspace. SALT measures this with the participation ratio of the Gram matrix and an effective sample-size statistic \[ n_{\rm eff}(\{u_i\})=\frac{\|\sum_i u_i\|^2}{\sum_i \|u_i\|^2}, \qquad u_i=\hat A_i s_i. \] It then estimates a dominant subspace, decomposes coefficients into shared and residual channels, and reweights them as \[ a' = P a + \alpha P^\perp [a]_+, \qquad \alpha=\max(0,1-n_{\rm eff}), \] before plugging them back into the surrogate. Reported gains are \(+2\)–\(3\) percentage points across reasoning benchmarks, together with \(20\)–\(40\%\) higher participation ratio and \(2\)–\(3\times\) larger \(n_{\rm eff}\) [2606.05800]. ## 6. Applications and empirical landscape Much of the recent literature evaluates group-relative REINFORCE on verifiable mathematical reasoning, including GSM8K, MATH-500, AIME24, AIME25, AMC23, Minerva, OlympiadBench, GPQA, and multimodal math benchmarks. On Qwen2.5-7B with \(K=8\), F-GRPO improved pass@256 from \(64.1\rightarrow 70.3\) for GRPO, \(69.3\rightarrow 72.5\) for DAPO, and \(73.2\rightarrow 76.8\) for CISPO, while preserving or improving pass@1 and without increasing compute [2602.06717]. A-GRAE reported consistent gains across seven text and vision-language benchmarks, including Geo3K, MathVision, MathVerse, and HuatuoGPT-Vision [2602.05548]. MDP-GRPO was evaluated on FollowBench, IFEval, and a curated multi-constraint dataset, with up to \(5.0\%\) improvement in strict constraint satisfaction on Llama-3.2-3B while preserving MMLU and ARC performance [2606.06058]. The framework is not limited to text reasoning. InfLVG applies GRPO to inference-time context selection for long video generation. A lightweight policy \(F_\theta\) scores past tokens, Plackett–Luce sampling selects the top-\(K\) context tokens without replacement, and a hybrid reward \[ r=r_{\rm content}+r_{\rm clip}+r_{\rm artifact} \] combines identity preservation, prompt alignment, and artifact suppression. With group size \(G=10\), clipping \(\delta=0.1\), and top-\(K\) selection set to \(K=6hw\), the method reports video-length extension by up to \(9\times\), together with the introduction of the Cross-scene Video Benchmark and Event Prompt Set [2505.17574]. Open-ended generation motivates a different use of the same group-based machinery. RLRR evaluates writing tasks across Academic, Finance, Politics, Literature, Education, and Advertising, replacing unstable scalar reward-model outputs by bounded relative rankings. Reported gains are \(+0.7\)–\(1.8\) percentage points in WritingBench domains, and the fine-tuned Ranking Reward Model achieves the best scores in \(5/7\) domains [2601.23058]. Code-generation evaluations also appear in the SALT study through MBPP with HumanEval tests, indicating that the group-relative update rule and its geometric pathologies extend beyond mathematical QA [2606.05800]. Taken together, the literature presents Group-Relative REINFORCE not as a single frozen algorithm but as a family of critic-free or critic-light policy-gradient procedures organized around one core operation: compare multiple rollouts for the same prompt, transform those comparisons into relative advantages, and apply a policy-gradient update under clipping or regularization. The main research frontier has therefore shifted from defining the estimator to controlling its bias, symmetry, granularity, exploration behavior, reward representation, and compute efficiency. Source: https://www.emergentmind.com/topics/group-relative-reinforce