--- title: Group Policy Gradient (GPG) in Reinforcement Learning url: https://www.emergentmind.com/topics/group-policy-gradient-gpg type: topic --- # Group Policy Gradient (GPG) in Reinforcement Learning Searching arXiv for recent papers on Group Policy Gradient and closely related usages of the acronym GPG. Group Policy Gradient (GPG) most commonly denotes a critic-free policy-gradient approach in which a model samples a group of trajectories or responses, converts their rewards into a relative or normalized group signal, and updates the policy directly from log-probabilities rather than through a learned value function. In current arXiv usage, however, the acronym is not fully standardized: it also appears in a critic-free PPO-style estimator for general MDPs, a Transformer-specific Generalized Policy Gradient theorem, a history-aware routing framework, and the earlier Graph Policy Gradients method for multi-robot control. The group-based usage is most closely associated with reasoning-model post-training, where GPG is positioned against Group Relative Policy Optimization (GRPO) as a simpler way to optimize expected reward [2504.02546], while adjacent work places related group estimators within U-statistic theory and Transformer-native macro-action policy gradients [2603.01162] [2512.10365]. ## 1. Terminology and scope The literature uses the label “GPG” for several distinct constructions. For precision, the table below summarizes the principal usages that are directly relevant to contemporary policy-gradient research. | Usage | Core description | arXiv id | |---|---|---| | Group Policy Gradient | Minimalist RL baseline for model reasoning | [2504.02546] | | Group Policy Gradient | Critic-free PPO-style estimator for general MDPs | [2510.03679] | | Generalized Policy Gradient | Transformer-based macro-action theorem | [2512.10365] | | GPG-HT | Decision Transformer with GPG for stochastic routing | [2508.17218] | | Graph Policy Gradients | GCN-based policy gradients for robot swarms | [1907.03822] | Within the strict “Group Policy Gradient” sense, two threads dominate. The first is the reasoning-model formulation of “GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning” [2504.02546]. The second is the general-MDP formulation of “Group Policy Gradient” [2510.03679], which retains PPO’s clipped update but replaces the critic with a group-estimated Monte Carlo baseline. These share the same high-level premise: group statistics can serve as a control variate, removing the need for a learned value function. A broader theoretical usage appears in “GPG: Generalized Policy Gradient Theorem for Transformer-based Policies,” where GPG no longer means “group” but “generalized,” and the central object is a macro-action policy-gradient theorem rather than a particular estimator [2512.10365]. This suggests that “GPG” now denotes a family of partially overlapping ideas: group-normalized advantage estimation, critic-free PPO-style updates, and Transformer-native policy-gradient factorization. ## 2. Direct group-normalized policy gradients for reasoning models In “GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning,” the objective is the original RL objective \[ \mathcal{J}(\theta) = \max_{\theta} \mathbb{E}_{\pi_{\theta}} \left[ \sum_{t=0}^{T} r_t \right], \] with the standard policy-gradient forms \[ \nabla_{\theta} \mathcal{J}(\theta) = \mathbb{E}_{\pi_{\theta}} \left[ \nabla_{\theta} \log \pi_{\theta}(a_t \mid s_t) Q^{\pi_{\theta}}(s_t, a_t) \right] \] and \[ \nabla_{\theta} \mathcal{J}(\theta) = \mathbb{E}_{\pi_{\theta}} \left[ \nabla_{\theta} \log \pi_{\theta}(a_t \mid s_t) A^{\pi_{\theta}}(s_t, a_t) \right]. \] Its defining move is to eliminate the critic and compute the advantage from a group of sampled responses for the same prompt [2504.02546]. For a prompt \(q\), the method samples \(G\) responses \(\{o_i\}_{i=1}^G\), computes rewards \(\{R_i\}_{i=1}^G\), and forms the group-normalized advantage \[ \hat{A}_{i,t} = \frac{r_i - \text{mean}(\{R_i\}_{i=1}^{G})}{\text{std}(\{R_i\}_{i=1}^{G})}. \] The loss is \[ \mathcal{L}_{\text{GPG}} = - \log \pi_{\theta}(o) \cdot A, \qquad A = \frac{R(o)-\operatorname{mean}\{R(o)\}}{\operatorname{std}\{R(o)\}}. \] The same advantage is applied token-wise over the whole response trajectory. The training loop is correspondingly short: sample a group, score each response, normalize rewards within the group, and backpropagate through token log-probabilities. The paper emphasizes that GPG removes the clipped surrogate loss, the reference model, the KL penalty, any policy constraint, and the separate value model [2504.02546]. The stated motivation is that PPO requires a critic/value model and a reference model, while GRPO removes the critic but still keeps a reference model, clipping, and KL regularization. GPG is presented as direct optimization of the original RL objective rather than optimization of a surrogate objective. In the paper’s framing, this also avoids bias from learned value estimates and from ratio-clipped objectives [2504.02546]. The empirical program is broad. On mathematical reasoning with DeepSeek-R1-Distill-Qwen-1.5B, the reported averages are Base model **48.9**, GRPO **53.1**, and GPG **55.7**, with notable gains of **+3.8** on MATH-500 and **+10.0** on AMC23. On Qwen2.5-Math-7B, the reported averages are Base model **30.9**, GRPO **43.7**, and GPG **45.3**, including **+6.6** on AIME24 and **+3.6** on OlympiadBench. On multimodal tasks, the paper reports CV-Bench scores of **59.47** for +GRPO and **69.11** for +GPG on Qwen2-VL-2B, GEOQA scores of **47.48** for +GRPO and **50.80** for +GPG on Qwen2.5-VL-3B-Instruct, a classification average of **81.9** for GRPO and **86.0** for GPG, and LISA test mIoU of **37.6** for GRPO versus **51.5** for GPG [2504.02546]. The ablations in the same paper define several practical regularities. Sweeping group size \(\{2,4,8,16\}\) on Qwen2.5-Math-7B yields averages of **41.9**, **43.3**, **45.3**, and **47.3**, respectively; the authors choose **8** as a performance-cost tradeoff. Reward normalization within the group performs better than batch normalization, with **45.3** for group normalization versus **44.9** for batch normalization. Adding a KL penalty with \(\beta=0.04\) lowers the score to **43.7**, supporting the paper’s claim that KL constraints are unnecessary in its training regime [2504.02546]. ## 3. Relation to GRPO and the broader group-relative ecosystem GPG emerged in a setting already dominated by GRPO. In the GRPO meta-algorithm analyzed in “Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statistic,” for each prompt \(X^{(b)}\) one samples a group of \(G\) outputs and computes \[ \widehat g(\theta_i) = \frac{1}{BG}\sum_{b=1}^B \sum_{g=1}^G \sum_t \nabla_\theta \log \pi_{\theta_i}(Y_t^{(b,g)}\mid X^{(b)},Y_{3.5× larger accuracy drop** than perturbing the lowest-entropy tokens; that among correctly answered problems, **6.6%** of intermediate steps were locally erroneous yet got positive advantage; that among incorrectly answered problems, **51.5%** of steps were locally correct yet received negative advantage; and that with \(G=8\), **58.77%** of all training steps on MATH had zero-variance groups [2605.04960]. EP-GRPO replaces the flat sequence-level token advantage with \[ \hat{A}_{i,t}^{\text{final}}=\hat{A}_{i,t}^{\text{outcome}}+\hat{A}_{i,t}^{\text{progress}}, \] and reports average accuracy gains from **18.14** to **22.93** on Qwen2.5-3B and from **27.11** to **30.34** on Qwen2.5-7B relative to GRPO [2605.04960]. ## 4. Critic-free GPG for general MDPs The paper titled simply “Group Policy Gradient” generalizes the group-baseline idea from RLHF-style settings to ordinary MDPs while preserving PPO’s clipped-objective structure [2510.03679]. The method defines a binning function \[ f:\mathcal{S}\to \mathcal{B}, \] collects a group of \(N\) iid trajectories, computes discounted returns \[ R_t^{(i)} = R_t^\gamma(\tau_i)=\sum_{s\ge t}\gamma^{s-t}r_s, \] and sets \[ \hat A_t^{(i)}=R_t^{(i)}-\widehat b_N(s_t^{(i)}). \] The estimated baseline is the mean return of first visits to the corresponding bin: \[ \hat b_N(s)=\text{mean}\Big(\{R_t^{(i)} \mid (i,t): f(s_t^i)=f(s),\ \text{and }t\text{ is the first visit in }\tau_i\text{ to a state in bin }f(s)\}\Big). \] This formulation yields a critic-free PPO surrogate, \[ \mathcal{L}=\mathbb{E}_{\tau\sim\pi_{\theta_{\mathrm{old}}}} \left[ \min\left( \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{\mathrm{old}}}(a_t|s_t)}\hat A_t,\ \mathrm{clip}\left(\frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{\mathrm{old}}}(a_t|s_t)},1-\epsilon,1+\epsilon\right)\hat A_t \right) \right]. \] At the on-policy point \(\pi_\theta=\pi_{\theta_\text{old}}\), the clipping is inactive and \[ \nabla_\theta\mathcal{L}_N = \frac{1}{N}\sum_{n=1}^N \sum_{t=1}^T \hat A_t^n\nabla_\theta \log \pi_\theta(a_t^n|s_t^n). \] The paper proves consistency under bounded-score and bounded-reward assumptions: \[ \nabla_\theta \mathcal{L}_N \xrightarrow{\mathbb{P}} \nabla_\theta \eta(\theta). \] Its bias-variance analysis is controlled by bin granularity: universal binning is coarse, state-identity binning is fine, and time-based binning is presented as a strong default [2510.03679]. The empirical evaluation uses CartPole, CliffWalking, LunarLander, and HalfCheetah, with group size equal to the number of parallel environments. The main sweep varies parallel environments \(1,4,16,32,128\). After **200 iterations**, the reported \(128\)-rollout results are CartPole \(495.45 \pm 2.13\), CliffWalking \(-17.00 \pm 0.00\), HalfCheetah \(2773.61 \pm 222.93\), and LunarLander \(257.39 \pm 0.80\) [2510.03679]. The paper’s central empirical claim is that GPG matches or outperforms PPO when enough parallel environments are available, because larger groups directly improve the baseline estimate rather than merely increasing throughput. ## 5. Transformer-native generalization and sequence-level refinements In “GPG: Generalized Policy Gradient Theorem for Transformer-based Policies,” the acronym denotes a theorem rather than a particular group-normalized estimator. The policy is factorized over macro-actions \(MA_1,\dots,MA_K\) and macro-states \(MS_1,\dots,MS_K\), with \[ \pi_{\theta}(MA \mid MS_1) = \prod_{T=1}^{K}\pi_{\theta}(MA_T\mid MS_T), \] and the theorem states \[ \nabla_{\theta} J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}\left\{ \sum_{T=1}^{K} \left[ \nabla_{\theta}\log\pi_{\theta}(MA_T\mid MS_T)\,\Phi_T \right] \right\}. \] If each macro-action is a single token, this reduces to the standard token-level policy gradient; if the entire output sequence is one macro-action, it reduces to the sequence-level abstraction used by GRPO-style training [2512.10365]. The paper therefore treats standard policy gradient and GRPO as special cases within a single Transformer-native framework. This macro-action view directly motivates sequence-level refinements of group-based reasoning RL. “C\(^2\)GSPG” argues that GRPO and many of its variants optimize at the token level even when the reward is sequence-level, producing token-level bias. The paper cites a GPG baseline objective \[ \mathcal{J}_{\text{GPG}}(\theta) = \mathbb{E}_{\{o_i\}\sim\pi_{\theta_{\text{old}}}} \left[ \frac{1}{\sum_{j=1}^G |o_j|} \sum_{i=1}^G \sum_{t=1}^{|o_i|} A_i \log \pi_{\theta,i,t} \right], \qquad A_i=r_i-m, \] and proposes Group Sequence Policy Gradient and confidence calibration on top of normalized sequence probability [2509.23129]. For binary rewards, its confidence-calibration regularizer and policy-optimization direction are proved to share the same sign. On Knights and Knaves, the reported overall figures are **Accuracy:** \(78.97\%\pm 7.19\), **Brier Score:** \(0.136\pm 0.046\), and **ECE:** \(0.126\pm 0.053\). On the mathematical reasoning comparison table, the paper reports **C\(^2\)GSPG accuracy:** \(42.76\%\) with lower calibration errors than competitors in the reported table [2509.23129]. Taken together, these works reposition GPG from a single estimator into a segmentation question: whether credit assignment should be token-level, sequence-level, or macro-action-level. The Generalized Policy Gradient theorem supplies the formal abstraction; sequence-level variants such as GSPG and C\(^2\)GSPG modify the estimator when token-level weighting is judged to be misaligned with sequence-level rewards. ## 6. Other acronym usages and domain-specific applications The acronym also appears in domain-specific formulations that are not equivalent to Group Policy Gradient in the reasoning-RL sense. “GPG-HT: Generalized Policy Gradient with History-Aware Decision Transformer for Probabilistic Path Planning” addresses the Stochastic On-Time Arrival problem on transportation networks with stochastic travel times and dependencies [2508.17218]. Its objective is \[ J(\pi_\theta) = \mathbb{P}[G(\tau^{(j)}) \leq T], \] with Monte Carlo gradient estimator \[ \nabla_{\theta} J(\pi_\theta) \approx \frac{1}{M} \sum_{j=1}^M \mathbb{I}\{G(\tau^{(j)}) \leq T\} \nabla_{\theta} \log \pi_\theta(\tau^{(j)}). \] On the Sioux Falls Network benchmark, the paper reports that GPG-HT is generally best or tied-best across five OD pairs and three budgets, and on a synthetic network it converges to about **52.6%** versus **50.2%** for SEGAC, with a theoretical upper bound of about **53.7%** and stabilization in about **26 iterations** versus **40** [2508.17218]. A separate neighboring construction is “Policy Gradient Guidance,” whose paper states that PGG is “also referred to in the paper as a Group Policy Gradient (GPG)-style control mechanism” [2510.02148]. Its guided policy is \[ \hat{\pi}_\theta(a|s)\propto \pi_\theta(a)^{1-\gamma}\,\pi_\theta(a|s)^\gamma, \] and, under advantage estimation, the normalization term disappears, leaving \[ \nabla_\theta J(\theta) = \mathbb{E}\!\left[ A\Big( \gamma \nabla_\theta \log \pi_\theta(a|s) + (1-\gamma)\nabla_\theta \log \pi_\theta(a) \Big) \right]. \] The paper reports that conditioning dropout is helpful in discrete tasks such as CartPole and Acrobot, but can destabilize MuJoCo continuous control, whereas training with modestly larger guidance, typically \(\gamma=1.1\), improves stability and controllability in many continuous-control settings [2510.02148]. The oldest major acronym collision is “Graph Policy Gradients for Large Scale Robot Control,” where GPG denotes graph-based control of homogeneous robot swarms via graph convolutional policies [1907.03822]. The graph filter is \[ z = \sum_{k=0}^{K} h_k S^k x = H(S)x, \] with permutation-equivariance property \[ H(S)P^{T}x = P^{T}H(S)x. \] The policy-gradient objective is centralized, \[ \max_{\theta} \; \mathbb{E}_{\Pi}\left[\sum_{t=1}^{T} r_t\right], \] but the main contribution is architectural rather than group-normalization-based. The paper’s most distinctive empirical claim is successful zero-shot transfer of policies trained on just **three robots** to **over hundred robots** [1907.03822]. Across these usages, the stable encyclopedia-level conclusion is that Group Policy Gradient is not a single universally fixed algorithmic object. In its most influential recent sense, it is a critic-free, group-normalized alternative to critic-based or surrogate-heavy policy optimization, especially for reasoning-model post-training [2504.02546] [2510.03679]. In adjacent literature, the same acronym also marks theorem-level generalizations, domain-specific estimators, and older graph-based control methods. The shared theme is policy improvement from structured gradient estimators; the technical content depends on whether the relevant structure is a reward group, a macro-action decomposition, a trajectory history, or a graph. Source: https://www.emergentmind.com/topics/group-policy-gradient-gpg