Group Policy Gradient (GPG) in Reinforcement Learning

• Group Policy Gradient (GPG) is a family of reinforcement learning methods that uses group-normalized reward advantages to update policies without a critic.
• It encompasses various formulations, including reasoning model post-training, critic-free PPO for MDPs, and Transformer-based macro-action policies.
• Empirical results demonstrate that GPG improves performance on benchmarks (e.g., MATH, AMC23, multi-modal tests) by efficiently leveraging group statistics.

Searching arXiv for papers on Group Policy Gradient and closely related usages of the acronym GPG. Group Policy Gradient (GPG) most commonly denotes a critic-free policy-gradient approach in which a model samples a group of trajectories or responses, converts their rewards into a relative or normalized group signal, and updates the policy directly from log-probabilities rather than through a learned value function. In current arXiv usage, however, the acronym is not fully standardized: it also appears in a critic-free PPO-style estimator for general MDPs, a Transformer-specific Generalized Policy Gradient theorem, a history-aware routing framework, and the earlier Graph Policy Gradients method for multi-robot control. The group-based usage is most closely associated with reasoning-model post-training, where GPG is positioned against Group Relative Policy Optimization (GRPO) as a simpler way to optimize expected reward (Chu et al., 3 Apr 2025), while adjacent work places related group estimators within U-statistic theory and Transformer-native macro-action policy gradients (Zhou et al., 1 Mar 2026, Mao et al., 11 Dec 2025).

1. Terminology and scope

The literature uses the label “GPG” for several distinct constructions. For precision, the table below summarizes the principal usages that are directly relevant to contemporary policy-gradient research.

Usage	Core description	arXiv id
Group Policy Gradient	Minimalist RL baseline for model reasoning	(Chu et al., 3 Apr 2025)
Group Policy Gradient	Critic-free PPO-style estimator for general MDPs	(Chen et al., 4 Oct 2025)
Generalized Policy Gradient	Transformer-based macro-action theorem	(Mao et al., 11 Dec 2025)
GPG-HT	Decision Transformer with GPG for stochastic routing	(Wei et al., 24 Aug 2025)
Graph Policy Gradients	GCN-based policy gradients for robot swarms	(Khan et al., 2019)

Within the strict “Group Policy Gradient” sense, two threads dominate. The first is the reasoning-model formulation of “GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning” (Chu et al., 3 Apr 2025). The second is the general-MDP formulation of “Group Policy Gradient” (Chen et al., 4 Oct 2025), which retains PPO’s clipped update but replaces the critic with a group-estimated Monte Carlo baseline. These share the same high-level premise: group statistics can serve as a control variate, removing the need for a learned value function.

A broader theoretical usage appears in “GPG: Generalized Policy Gradient Theorem for Transformer-based Policies,” where GPG no longer means “group” but “generalized,” and the central object is a macro-action policy-gradient theorem rather than a particular estimator (Mao et al., 11 Dec 2025). This suggests that “GPG” now denotes a family of partially overlapping ideas: group-normalized advantage estimation, critic-free PPO-style updates, and Transformer-native policy-gradient factorization.

2. Direct group-normalized policy gradients for reasoning models

In “GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning,” the objective is the original RL objective

$$\mathcal{J}(\theta) = \max_{\theta} \mathbb{E}_{\pi_{\theta}} \left[ \sum_{t=0}^{T} r_t \right],$$

with the standard policy-gradient forms

$$\nabla_{\theta} \mathcal{J}(\theta) = \mathbb{E}_{\pi_{\theta}} \left[ \nabla_{\theta} \log \pi_{\theta}(a_t \mid s_t) Q^{\pi_{\theta}}(s_t, a_t) \right]$$

and

$$\nabla_{\theta} \mathcal{J}(\theta) = \mathbb{E}_{\pi_{\theta}} \left[ \nabla_{\theta} \log \pi_{\theta}(a_t \mid s_t) A^{\pi_{\theta}}(s_t, a_t) \right].$$

Its defining move is to eliminate the critic and compute the advantage from a group of sampled responses for the same prompt (Chu et al., 3 Apr 2025).

For a prompt $q$, the method samples $G$ responses $\{o_i\}_{i=1}^G$, computes rewards $\{R_i\}_{i=1}^G$, and forms the group-normalized advantage

$$\hat{A}_{i,t} = \frac{r_i - \text{mean}(\{R_i\}_{i=1}^{G})}{\text{std}(\{R_i\}_{i=1}^{G})}.$$

The loss is

$$\mathcal{L}_{\text{GPG}} = - \log \pi_{\theta}(o) \cdot A, \qquad A = \frac{R(o)-\operatorname{mean}\{R(o)\}}{\operatorname{std}\{R(o)\}}.$$

The same advantage is applied token-wise over the whole response trajectory. The training loop is correspondingly short: sample a group, score each response, normalize rewards within the group, and backpropagate through token log-probabilities. The paper emphasizes that GPG removes the clipped surrogate loss, the reference model, the KL penalty, any policy constraint, and the separate value model (Chu et al., 3 Apr 2025).

The stated motivation is that PPO requires a critic/value model and a reference model, while GRPO removes the critic but still keeps a reference model, clipping, and KL regularization. GPG is presented as direct optimization of the original RL objective rather than optimization of a surrogate objective. In the paper’s framing, this also avoids bias from learned value estimates and from ratio-clipped objectives (Chu et al., 3 Apr 2025).

The empirical program is broad. On mathematical reasoning with DeepSeek-R1-Distill-Qwen-1.5B, the reported averages are Base model 48.9, GRPO 53.1, and GPG 55.7, with notable gains of +3.8 on MATH-500 and +10.0 on AMC23. On Qwen2.5-Math-7B, the reported averages are Base model 30.9, GRPO 43.7, and GPG 45.3, including +6.6 on AIME24 and +3.6 on OlympiadBench. On multimodal tasks, the paper reports CV-Bench scores of 59.47 for +GRPO and 69.11 for +GPG on Qwen2-VL-2B, GEOQA scores of 47.48 for +GRPO and 50.80 for +GPG on Qwen2.5-VL-3B-Instruct, a classification average of 81.9 for GRPO and 86.0 for GPG, and LISA test mIoU of 37.6 for GRPO versus 51.5 for GPG (Chu et al., 3 Apr 2025).

The ablations in the same paper define several practical regularities. Sweeping group size $\{2,4,8,16\}$ on Qwen2.5-Math-7B yields averages of 41.9, 43.3, 45.3, and 47.3, respectively; the authors choose 8 as a performance-cost tradeoff. Reward normalization within the group performs better than batch normalization, with 45.3 for group normalization versus 44.9 for batch normalization. Adding a KL penalty with $$\nabla_{\theta} \mathcal{J}(\theta) = \mathbb{E}_{\pi_{\theta}} \left[ \nabla_{\theta} \log \pi_{\theta}(a_t \mid s_t) Q^{\pi_{\theta}}(s_t, a_t) \right]$$0 lowers the score to 43.7, supporting the paper’s claim that KL constraints are unnecessary in its training regime (Chu et al., 3 Apr 2025).

3. Relation to GRPO and the broader group-relative ecosystem

GPG emerged in a setting already dominated by GRPO. In the GRPO meta-algorithm analyzed in “Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statistic,” for each prompt $$\nabla_{\theta} \mathcal{J}(\theta) = \mathbb{E}_{\pi_{\theta}} \left[ \nabla_{\theta} \log \pi_{\theta}(a_t \mid s_t) Q^{\pi_{\theta}}(s_t, a_t) \right]$$1 one samples a group of $$\nabla_{\theta} \mathcal{J}(\theta) = \mathbb{E}_{\pi_{\theta}} \left[ \nabla_{\theta} \log \pi_{\theta}(a_t \mid s_t) Q^{\pi_{\theta}}(s_t, a_t) \right]$$2 outputs and computes

$$\nabla_{\theta} \mathcal{J}(\theta) = \mathbb{E}_{\pi_{\theta}} \left[ \nabla_{\theta} \log \pi_{\theta}(a_t \mid s_t) Q^{\pi_{\theta}}(s_t, a_t) \right]$$3

with the leave-one-out group mean baseline

$$\nabla_{\theta} \mathcal{J}(\theta) = \mathbb{E}_{\pi_{\theta}} \left[ \nabla_{\theta} \log \pi_{\theta}(a_t \mid s_t) Q^{\pi_{\theta}}(s_t, a_t) \right]$$4

The paper shows that the prompt-conditional GRPO gradient can be symmetrized into a second-order U-statistic with kernel

$$\nabla_{\theta} \mathcal{J}(\theta) = \mathbb{E}_{\pi_{\theta}} \left[ \nabla_{\theta} \log \pi_{\theta}(a_t \mid s_t) Q^{\pi_{\theta}}(s_t, a_t) \right]$$5

and that its kernel mean equals the true policy gradient (Zhou et al., 1 Mar 2026).

That analysis derives a sharp MSE expansion,

$$\nabla_{\theta} \mathcal{J}(\theta) = \mathbb{E}_{\pi_{\theta}} \left[ \nabla_{\theta} \log \pi_{\theta}(a_t \mid s_t) Q^{\pi_{\theta}}(s_t, a_t) \right]$$6

establishes asymptotic equivalence to an oracle-baseline gradient algorithm, and gives the universal scaling law

$$\nabla_{\theta} \mathcal{J}(\theta) = \mathbb{E}_{\pi_{\theta}} \left[ \nabla_{\theta} \log \pi_{\theta}(a_t \mid s_t) Q^{\pi_{\theta}}(s_t, a_t) \right]$$7

This does not redefine GPG itself, but it provides the main statistical reference point for group-based critic-free estimators more generally. A plausible implication is that GPG-style methods can be assessed not only by empirical simplicity, but also by how closely their group baseline approximates oracle variance reduction.

The same comparison class includes newer GRPO variants. “Pave-GRPO” addresses flow-based generative models, where group rollouts are costly because each rollout is a multi-step denoising chain. It keeps few-step group rollouts, computes the standard group-relative advantage

$$\nabla_{\theta} \mathcal{J}(\theta) = \mathbb{E}_{\pi_{\theta}} \left[ \nabla_{\theta} \log \pi_{\theta}(a_t \mid s_t) Q^{\pi_{\theta}}(s_t, a_t) \right]$$8

and augments the clipped GRPO objective with a decomposition term that reuses each coarse transition under multiple equivalent hybrid decompositions, yielding what the paper calls zero-cost horizon expansion and comprehensive temporal supervision (Ling et al., 1 Jun 2026). Under HPS-v2 reward, it reports HPS-v2 = 36.40 versus Flow-GRPO’s 34.63, and on UniGenBench it reports 63.96 versus Flow-GRPO’s 62.65 (Ling et al., 1 Jun 2026).

“EP-GRPO” diagnoses three GRPO credit-assignment failures: uniform token-level granularity, uniform polarity, and zero-variance collapse. Its motivation experiments report that perturbing the top 5% highest-entropy tokens causes a >3.5× larger accuracy drop than perturbing the lowest-entropy tokens; that among correctly answered problems, 6.6% of intermediate steps were locally erroneous yet got positive advantage; that among incorrectly answered problems, 51.5% of steps were locally correct yet received negative advantage; and that with $$\nabla_{\theta} \mathcal{J}(\theta) = \mathbb{E}_{\pi_{\theta}} \left[ \nabla_{\theta} \log \pi_{\theta}(a_t \mid s_t) Q^{\pi_{\theta}}(s_t, a_t) \right]$$9, 58.77% of all training steps on MATH had zero-variance groups (Yu et al., 6 May 2026). EP-GRPO replaces the flat sequence-level token advantage with

$$\nabla_{\theta} \mathcal{J}(\theta) = \mathbb{E}_{\pi_{\theta}} \left[ \nabla_{\theta} \log \pi_{\theta}(a_t \mid s_t) A^{\pi_{\theta}}(s_t, a_t) \right].$$0

and reports average accuracy gains from 18.14 to 22.93 on Qwen2.5-3B and from 27.11 to 30.34 on Qwen2.5-7B relative to GRPO (Yu et al., 6 May 2026).

4. Critic-free GPG for general MDPs

The paper simply “Group Policy Gradient” generalizes the group-baseline idea from RLHF-style settings to ordinary MDPs while preserving PPO’s clipped-objective structure (Chen et al., 4 Oct 2025). The method defines a binning function

$$\nabla_{\theta} \mathcal{J}(\theta) = \mathbb{E}_{\pi_{\theta}} \left[ \nabla_{\theta} \log \pi_{\theta}(a_t \mid s_t) A^{\pi_{\theta}}(s_t, a_t) \right].$$1

collects a group of $$\nabla_{\theta} \mathcal{J}(\theta) = \mathbb{E}_{\pi_{\theta}} \left[ \nabla_{\theta} \log \pi_{\theta}(a_t \mid s_t) A^{\pi_{\theta}}(s_t, a_t) \right].$$2 iid trajectories, computes discounted returns

$$\nabla_{\theta} \mathcal{J}(\theta) = \mathbb{E}_{\pi_{\theta}} \left[ \nabla_{\theta} \log \pi_{\theta}(a_t \mid s_t) A^{\pi_{\theta}}(s_t, a_t) \right].$$3

and sets

$$\nabla_{\theta} \mathcal{J}(\theta) = \mathbb{E}_{\pi_{\theta}} \left[ \nabla_{\theta} \log \pi_{\theta}(a_t \mid s_t) A^{\pi_{\theta}}(s_t, a_t) \right].$$4

The estimated baseline is the mean return of first visits to the corresponding bin: $$\nabla_{\theta} \mathcal{J}(\theta) = \mathbb{E}_{\pi_{\theta}} \left[ \nabla_{\theta} \log \pi_{\theta}(a_t \mid s_t) A^{\pi_{\theta}}(s_t, a_t) \right].$$5

This formulation yields a critic-free PPO surrogate,

$$\nabla_{\theta} \mathcal{J}(\theta) = \mathbb{E}_{\pi_{\theta}} \left[ \nabla_{\theta} \log \pi_{\theta}(a_t \mid s_t) A^{\pi_{\theta}}(s_t, a_t) \right].$$6

At the on-policy point $$\nabla_{\theta} \mathcal{J}(\theta) = \mathbb{E}_{\pi_{\theta}} \left[ \nabla_{\theta} \log \pi_{\theta}(a_t \mid s_t) A^{\pi_{\theta}}(s_t, a_t) \right].$$7, the clipping is inactive and

$$\nabla_{\theta} \mathcal{J}(\theta) = \mathbb{E}_{\pi_{\theta}} \left[ \nabla_{\theta} \log \pi_{\theta}(a_t \mid s_t) A^{\pi_{\theta}}(s_t, a_t) \right].$$8

The paper proves consistency under bounded-score and bounded-reward assumptions: $$\nabla_{\theta} \mathcal{J}(\theta) = \mathbb{E}_{\pi_{\theta}} \left[ \nabla_{\theta} \log \pi_{\theta}(a_t \mid s_t) A^{\pi_{\theta}}(s_t, a_t) \right].$$9 Its bias-variance analysis is controlled by bin granularity: universal binning is coarse, state-identity binning is fine, and time-based binning is presented as a strong default (Chen et al., 4 Oct 2025).

The empirical evaluation uses CartPole, CliffWalking, LunarLander, and HalfCheetah, with group size equal to the number of parallel environments. The main sweep varies parallel environments $q$0. After 200 iterations, the reported $q$1-rollout results are CartPole $q$2, CliffWalking $q$3, HalfCheetah $q$4, and LunarLander $q$5 (Chen et al., 4 Oct 2025). The paper’s central empirical claim is that GPG matches or outperforms PPO when enough parallel environments are available, because larger groups directly improve the baseline estimate rather than merely increasing throughput.

5. Transformer-native generalization and sequence-level refinements

In “GPG: Generalized Policy Gradient Theorem for Transformer-based Policies,” the acronym denotes a theorem rather than a particular group-normalized estimator. The policy is factorized over macro-actions $q$6 and macro-states $q$7, with

$q$8

and the theorem states

$q$9

If each macro-action is a single token, this reduces to the standard token-level policy gradient; if the entire output sequence is one macro-action, it reduces to the sequence-level abstraction used by GRPO-style training (Mao et al., 11 Dec 2025). The paper therefore treats standard policy gradient and GRPO as special cases within a single Transformer-native framework.

This macro-action view directly motivates sequence-level refinements of group-based reasoning RL. “C$G$0GSPG” argues that GRPO and many of its variants optimize at the token level even when the reward is sequence-level, producing token-level bias. The paper cites a GPG baseline objective

$G$1

and proposes Group Sequence Policy Gradient and confidence calibration on top of normalized sequence probability (Liu et al., 27 Sep 2025). For binary rewards, its confidence-calibration regularizer and policy-optimization direction are proved to share the same sign. On Knights and Knaves, the reported overall figures are Accuracy: $G$2, Brier Score: $G$3, and ECE: $G$4. On the mathematical reasoning comparison table, the paper reports C$G$5GSPG accuracy: $G$6 with lower calibration errors than competitors in the reported table (Liu et al., 27 Sep 2025).

Taken together, these works reposition GPG from a single estimator into a segmentation question: whether credit assignment should be token-level, sequence-level, or macro-action-level. The Generalized Policy Gradient theorem supplies the formal abstraction; sequence-level variants such as GSPG and C$G$7GSPG modify the estimator when token-level weighting is judged to be misaligned with sequence-level rewards.

6. Other acronym usages and domain-specific applications

The acronym also appears in domain-specific formulations that are not equivalent to Group Policy Gradient in the reasoning-RL sense. “GPG-HT: Generalized Policy Gradient with History-Aware Decision Transformer for Probabilistic Path Planning” addresses the Stochastic On-Time Arrival problem on transportation networks with stochastic travel times and dependencies (Wei et al., 24 Aug 2025). Its objective is

$G$8

with Monte Carlo gradient estimator

$G$9

On the Sioux Falls Network benchmark, the paper reports that GPG-HT is generally best or tied-best across five OD pairs and three budgets, and on a synthetic network it converges to about 52.6% versus 50.2% for SEGAC, with a theoretical upper bound of about 53.7% and stabilization in about 26 iterations versus 40 (Wei et al., 24 Aug 2025).

A separate neighboring construction is “Policy Gradient Guidance,” whose paper states that PGG is “also referred to in the paper as a Group Policy Gradient (GPG)-style control mechanism” (Qi et al., 2 Oct 2025). Its guided policy is

$\{o_i\}_{i=1}^G$0

and, under advantage estimation, the normalization term disappears, leaving

$\{o_i\}_{i=1}^G$1

The paper reports that conditioning dropout is helpful in discrete tasks such as CartPole and Acrobot, but can destabilize MuJoCo continuous control, whereas training with modestly larger guidance, typically $\{o_i\}_{i=1}^G$2, improves stability and controllability in many continuous-control settings (Qi et al., 2 Oct 2025).

The oldest major acronym collision is “Graph Policy Gradients for Large Scale Robot Control,” where GPG denotes graph-based control of homogeneous robot swarms via graph convolutional policies (Khan et al., 2019). The graph filter is

$\{o_i\}_{i=1}^G$3

with permutation-equivariance property

$\{o_i\}_{i=1}^G$4

The policy-gradient objective is centralized,

$\{o_i\}_{i=1}^G$5

but the main contribution is architectural rather than group-normalization-based. The paper’s most distinctive empirical claim is successful zero-shot transfer of policies trained on just three robots to over hundred robots (Khan et al., 2019).

Across these usages, the stable encyclopedia-level conclusion is that Group Policy Gradient is not a single universally fixed algorithmic object. In its most influential recent sense, it is a critic-free, group-normalized alternative to critic-based or surrogate-heavy policy optimization, especially for reasoning-model post-training (Chu et al., 3 Apr 2025, Chen et al., 4 Oct 2025). In adjacent literature, the same acronym also marks theorem-level generalizations, domain-specific estimators, and older graph-based control methods. The shared theme is policy improvement from structured gradient estimators; the technical content depends on whether the relevant structure is a reward group, a macro-action decomposition, a trajectory history, or a graph.