---
title: Group Sequence Policy Gradient (GSPG)
url: https://www.emergentmind.com/topics/group-sequence-policy-gradient-gspg
type: topic
---

# Group Sequence Policy Gradient (GSPG)

Searching arXiv for the primary papers on GSPG, GSPO, and related group-based sequence RL methods.
Group Sequence Policy Gradient (GSPG) denotes a **sequence-level policy gradient framework** for reasoning large language models in which an entire generated reasoning chain is treated as the optimization unit, rather than assigning policy updates token by token. In the formulation introduced in “C$^2$GSPG: Confidence-calibrated Group Sequence Policy Gradient towards Self-aware Reasoning” [2509.23129], GSPG is designed to **eliminate the token-level bias commonly appearing in GRPO and its variants** by using a **single confidence/probability scalar per sequence**. Closely related work on “Group Sequence Policy Optimization” (GSPO) presents a near-synonymous sequence-level, group-based clipped policy objective, but explicitly states that **GSPG is not the paper’s name** and that the correct term there is **GSPO** [2507.18071]. Taken together, these works situate GSPG within a broader family of critic-free or critic-light, group-based reinforcement-learning methods for autoregressive policies.

## 1. Terminological scope and historical placement

The term **Group Sequence Policy Gradient** appears explicitly in the 2025 paper on confidence-calibrated reasoning models, which proposes both a base **GSPG framework** and its calibrated extension **C$^2$GSPG** [2509.23129]. In that work, GSPG is presented as a framework for **learning reasoning models** that moves optimization from the token level to the sequence level in order to remove token-level bias and to support confidence-aware training.

A distinct but closely allied line of work introduces **Group Sequence Policy Optimization (GSPO)** rather than GSPG. That paper states unambiguously that **GSPG is not the paper’s name** and that GSPO should be understood as the formal method name; it also notes that a phrase such as “Group Sequence Policy Gradient” would be **closely related** and is best interpreted as a **gradient-based variant/description of GSPO**, rather than as a separate algorithm in that paper [2507.18071].

The broader methodological context is a 2025–2026 transition from **token-level** RL objectives such as GRPO toward **group-based** and **sequence-aware** objectives. This transition includes: critic-free grouped baselines for general MDPs in **GPG** [2510.03679]; sequence-level communication optimization in **Agent-GSPO** [2510.22477]; and structure-aware **sub-sequence** optimization in **GSsPO** for multi-turn Think-Action workflows [2602.01202]. This suggests a family resemblance rather than a single canonical definition: GSPG names one concrete sequence-level framework, while GSPO, GPG, and GSsPO instantiate adjacent design choices at different optimization granularities.

## 2. Core formulation of GSPG

In the GSPG formulation, for a prompt \(q\) and generated sequence \(o_i = \{o_{i,t}\}_{t=1}^{|o_i|}\), the model confidence for the entire sequence is defined as the **normalized sequence-level probability**
\[
c_{\theta,i} := \pi_\theta(o_i|q)^{\frac{1}{|o_i|}}
= \exp\!\Big(\frac{1}{|o_i|}\sum_{t=1}^{|o_i|}\log \pi_{\theta,i,t}\Big),
\]
where \(\pi_{\theta,i,t} = \pi_\theta(o_{i,t}\mid o_{i,<t}, q)\) [2509.23129]. The normalization by \(\frac{1}{|o_i|}\) removes length dependence.

The policy term is sequence-level. In the calibrated presentation, it takes the form
\[
\tilde{A}_i \cdot \log c_{\theta,i},
\]
where \(\tilde{A}_i\) is a confidence-modulated advantage [2509.23129]. For **binary rewards**, the paper defines
\[
\tilde{A}_i = \frac{r_i - m}{1 - c_{\theta_{\text{old},i}}},
\]
with \(r_i \in \{0,1\}\) the sequence reward, \(m = \frac{1}{G}\sum_{i=1}^G r_i\) the group mean reward, and \(c_{\theta_{\text{old},i}}\) the confidence under the old policy [2509.23129].

This differs from GRPO’s variance-normalized group advantage
\[
\hat{A}_i = \frac{r_i - m}{\sigma},
\]
and the paper argues that GSPG’s **confidence-based denominator** gives **less suppression of low-confidence sequences** and avoids the **gradient diminishing** effect on those examples [2509.23129]. The same paper explicitly contrasts this with GSPO, stating that **GSPO uses a sequence-level ratio \(s_{\theta,i}\)**, whereas **GSPG uses the confidence-modulated advantage \((r_i-m)/(1-c_{\theta_{\text{old},i}})\)** and **does not rely on variance normalization** [2509.23129].

A related sequence-level objective from GSPO uses a **length-normalized sequence likelihood ratio**
\[
s_i(\theta)=\left(\frac{\pi_\theta(y_i\mid x)}{\pi_{\theta_\text{old}}(y_i\mid x)}\right)^{\frac{1}{|y_i|}}
\]
and a PPO-like clipped surrogate at the **whole-sequence** level
\[
\mathcal{J}_\text{GSPO}(\theta)= \mathbb{E}\left[ \frac{1}{G}\sum_{i=1}^{G} \min\left( s_i(\theta)\widehat{A}_i,\, \mathrm{clip}(s_i(\theta),1-\varepsilon,1+\varepsilon)\widehat{A}_i \right) \right]
\]
[2507.18071]. This provides the closest neighboring formalism to GSPG.

## 3. Motivation: token-level bias and sequence-level credit assignment

The central motivation for GSPG is the claim that **existing token-level RL objectives for reasoning models—especially GRPO and related methods—tend to create two problems**: **token-level bias** and **overconfidence / miscalibration** [2509.23129]. In the paper’s diagnosis, GRPO applies policy gradients through **per-token probability ratios**, which can cause **low-probability tokens to dominate training**, making updates uneven across a sequence. The remedy is to optimize using a single sequence-level scalar, so that an entire reasoning trajectory is treated as a coherent unit.

The closely related GSPO paper frames a similar diagnosis in different terms. It argues that GRPO’s **token-level importance ratios** are unstable, especially for **long responses**, **large models**, **Mixture-of-Experts (MoE) models**, and settings with **mini-batch reuse** from rollout data [2507.18071]. Its core claim is that **reward is sequence-level**, so the **importance ratio should also be sequence-level**, and clipping should also be **sequence-level**. That paper further states that token-level ratios can introduce **high-variance gradient noise**, accumulate instability over **long sequences**, and can ultimately cause **catastrophic model collapse** [2507.18071].

In a more general MDP setting, **Group Policy Gradient (GPG)** presents the same family of ideas in critic-free form: it is described as **PPO with a group-estimated baseline**, replacing a learned value function with a **group-based Monte Carlo baseline** while preserving the clipped-policy-update structure [2510.03679]. That work does not present GPG as sequence-level by definition, but it explicitly connects grouped baselines to **GRPO** and related RLHF formulations. A plausible implication is that GSPG can be read as the sequence-specific specialization of a broader group-baseline principle: critic-free grouping can replace critic learning when relative comparisons among samples are informative.

## 4. Confidence calibration and the C$^2$GSPG extension

C$^2$GSPG extends GSPG by adding a **cross-entropy / binary cross-entropy regularizer** that aligns sequence confidence with reward [2509.23129]. For binary rewards, the objective is
\[
\mathcal{J}_{\text{C}^2\text{GSPG}}(\theta) = \mathbb{E}_{\{o_i\}\sim \pi_{\theta_{\text{old}}}}
\frac{1}{G}\sum_{i=1}^G \Big[
\tilde{A}_i \log c_{\theta,i}
+ \beta\big(r_i\log c_{\theta,i} + (1-r_i)\log(1-c_{\theta,i})\big)
\Big],
\]
where \(\beta>0\) controls regularization strength [2509.23129].

The interpretation given in the paper is direct. If \(r_i=1\), the BCE term encourages \(c_{\theta,i}\to 1\); if \(r_i=0\), it encourages \(c_{\theta,i}\to 0\). The stated purpose is to make the model’s confidence behave like a calibrated probability of correctness [2509.23129].

A central theoretical claim of the paper is that, for **binary rewards**, the policy-gradient term and the confidence-calibration term **always share the same sign**. Using
\[
\nabla_\theta c_{\theta,i} = c_{\theta,i}\cdot \Big(\frac{1}{|o_i|}\sum_{t=1}^{|o_i|}\nabla_\theta \log \pi_{\theta,i,t}\Big),
\]
the paper derives a gradient in which the GSPG term and BCE term are collaborative, not adversarial [2509.23129]. The stated consequence is that correct sequences become more confident and incorrect sequences become less confident.

For **non-binary rewards**, the paper states that this alignment does **not automatically extend**. It therefore introduces two mechanisms: **nonlinear reward normalization** with
\[
\hat{r}_i = \sigma(\alpha r_i)=\frac{1}{1+\exp(-\alpha r_i)},
\]
and **adaptive regularizer clipping**
\[
\mathbb{I}_i(\beta)=
\begin{cases}
\beta, & \text{if } \operatorname{sign}(\hat{r}_i - \hat{m}) = \operatorname{sign}(\hat{r}_i - c_{\theta,i})\\
0, & \text{otherwise}.
\end{cases}
\]
The full non-binary objective applies the BCE-style term only when it agrees with the policy signal [2509.23129]. This is presented as a method for mitigating objective conflict in graded-reward settings such as logic tasks with rewards like \(\{-3,-1,-0.5,3\}\).

## 5. Empirical findings and benchmarked behavior

The GSPG paper reports experiments on **logical reasoning** and **mathematical reasoning** tasks, comparing **C$^2$GSPG** to **GRPO**, **GPG**, **GSPO**, and **AR-Lopti** [2509.23129]. On **K\(_{paper}\)K / Knights and Knaves** with **Qwen2.5-3B-Instruct**, the paper reports for **C$^2$GSPG**: **Accuracy: 78.97 ± 7.19**, **BS: 0.136 ± 0.046**, and **ECE: 0.126 ± 0.053**. The best listed baseline is **AR-Lopti** with **Accuracy 56.61**, **BS 0.282**, and **ECE 0.343** [2509.23129]. The same paper emphasizes reliability diagrams close to the perfect diagonal, a more bimodal confidence distribution, and ablations showing that **sigmoid normalization**, **adaptive clipping**, and the **full method** are best [2509.23129].

Related sequence-level results from **GSPO** reinforce the broader empirical case for sequence-level optimization. The GSPO paper reports stable training, continuous performance improvement with more training compute, updated query sets, and longer generation lengths; it also states that GSPO beats GRPO in **training accuracy**, **benchmark performance**, and **efficiency per compute and per query**, and that GSPO **eliminates the need for Routing Replay** in MoE training [2507.18071].

An application-specific extension appears in **Agent-GSPO**, which adapts GSPO to multi-agent communication by treating an utterance as a sequence-level action and optimizing a communication-aware reward that penalizes verbosity [2510.22477]. Across **seven reasoning benchmarks**, the paper reports the best results on **MMLU, MultiArith, GSM8K, SVAMP, AQuA, HumanEval, and MATH-500**. It gives concrete examples: **96.02\% on GSM8K**, **90.70\% pass@1 on HumanEval**, and **83.10\% on MATH-500** [2510.22477]. It also reports token reductions such as **\(9.20 \times 10^5\)** tokens on MMLU versus **\(1.50 \times 10^6\)** for LLM-Debate, **\(2.60 \times 10^6\)** for PHP, and **\(1.20 \times 10^6\)** for DyLAN, as well as **\(7.20 \times 10^6\)** tokens on GSM8K versus **\(2.20 \times 10^7\)**, **\(2.60 \times 10^7\)**, and **\(1.40 \times 10^7\)**, respectively [2510.22477]. This does not demonstrate GSPG directly, but it does demonstrate the practical efficacy of the closely related sequence-level group-optimization family.

## 6. Relations to adjacent methods and granularity variants

GSPG is most naturally contrasted with **GRPO**, **GSPO**, **GPG**, and **GSsPO**.

| Method | Optimization unit | Distinctive feature |
|---|---|---|
| GRPO | token level | per-token ratios with group-relative advantage |
| GSPG | sequence level | normalized sequence-level probability as confidence |
| GSPO | sequence level | sequence-level likelihood ratio and sequence-level clipping |
| GPG | grouped trajectories in general MDPs | critic-free group-estimated baseline |
| GSsPO | sub-sequence level | Think-Action pair as atomic unit |

Within this family, GSPO is the closest formal neighbor. Its defining move is to compute a **sequence-level importance ratio**, use **group-relative reward normalization**, and apply **sequence-level clipping** rather than token-level clipping [2507.18071]. GSPG differs in that its calibrated version is centered on the sequence confidence \(c_{\theta,i}\) and a confidence-dependent denominator \(1-c_{\theta_{\text{old},i}}\), rather than on variance-normalized advantages and sequence likelihood ratios [2509.23129].

GPG, by contrast, is not tied to language sequences per se. It is a **critic-free policy-gradient method for general MDPs** in which grouped trajectories and a **binning function** provide the baseline, and the paper proves a **consistency theorem** as group size \(N\to\infty\) [2510.03679]. This suggests a general principle: group baselines can substitute for learned critics in settings with sufficient parallel samples.

GSsPO further refines the granularity choice for multi-turn agentic workflows. That method argues that token-level optimization is too fine and full-sequence optimization too coarse for interleaved **Think-Action** interactions; it therefore optimizes a **sub-sequence**, defined as one contiguous Think-Action pair per turn [2602.01202]. A plausible implication is that GSPG belongs to a wider search for the semantically appropriate unit of credit assignment: token, sequence, or structured sub-sequence.

A broader theoretical framing appears in “GPG: Generalized Policy Gradient Theorem for Transformer-based Policies,” which states that both standard Policy Gradient Theorem and **GRPO emerge as special cases** of a macro-action policy-gradient theorem for Transformer policies [2512.10365]. That work does not define GSPG by name, but it formalizes a continuum between **token-level** and **sequence-level/group-like** policy gradients. This suggests that GSPG can be viewed as one point in a general segmentation space for autoregressive policy optimization.

## 7. Interpretation, misconceptions, and significance

A recurrent misconception is that **GSPG** and **GSPO** are interchangeable names for a single algorithm. The record in the cited papers is more precise. **GSPG** is explicitly named in the confidence-calibration paper [2509.23129], whereas the 2025 sequence-level clipping paper explicitly says that **the correct term is GSPO**, and that “Group Sequence Policy Gradient” is best understood there as a **descriptive misnaming or near-synonym for the gradient view of GSPO**, not a distinct algorithm [2507.18071].

Another misconception is that these methods are merely token-level policy gradients with sequence rewards attached after the fact. The cited papers argue the opposite: the optimization unit itself is changed. In GSPG, the update uses a **sequence-level confidence scalar** [2509.23129]. In GSPO, it uses a **whole-sequence likelihood ratio** and **whole-sequence clipping** [2507.18071]. In GSsPO, it uses a **sub-sequence-level importance ratio** for Think-Action units [2602.01202]. The technical claim common to these methods is that the granularity of optimization should match the semantic or reward granularity of the task.

The significance of GSPG therefore lies not only in its particular confidence-modulated objective, but also in what it represents within post-training RL for reasoning models: a move away from token-wise correction toward group-relative optimization over semantically meaningful units. The empirical record reported for C$^2$GSPG—improvements in both **reasoning accuracy** and **confidence calibration** [2509.23129]—and the allied results for GSPO in sequence-level stability, MoE training, and communication efficiency [2507.18071; 2510.22477] indicate that sequence-level group-based optimization has become a distinct and expanding research direction.

Source: https://www.emergentmind.com/topics/group-sequence-policy-gradient-gspg