Group Sequence Policy Gradient (GSPG)

• GSPG is a sequence-level policy gradient framework that treats the entire generated reasoning chain as one optimization unit to eliminate token-level bias.
• It uses a normalized confidence scalar derived from sequence probabilities to modulate policy updates without variance normalization.
• Empirical results show that GSPG, especially its C²GSPG extension, improves reasoning accuracy and calibration on benchmark tasks compared to token-based methods.

Searching arXiv for the primary papers on GSPG, GSPO, and related group-based sequence RL methods. Group Sequence Policy Gradient (GSPG) denotes a sequence-level policy gradient framework for reasoning LLMs in which an entire generated reasoning chain is treated as the optimization unit, rather than assigning policy updates token by token. In the formulation introduced in “C$^2$GSPG: Confidence-calibrated Group Sequence Policy Gradient towards Self-aware Reasoning” (Liu et al., 27 Sep 2025), GSPG is designed to eliminate the token-level bias commonly appearing in GRPO and its variants by using a single confidence/probability scalar per sequence. Closely related work on “Group Sequence Policy Optimization” (GSPO) presents a near-synonymous sequence-level, group-based clipped policy objective, but explicitly states that GSPG is not the paper’s name and that the correct term there is GSPO (Zheng et al., 24 Jul 2025). Taken together, these works situate GSPG within a broader family of critic-free or critic-light, group-based reinforcement-learning methods for autoregressive policies.

1. Terminological scope and historical placement

The term Group Sequence Policy Gradient appears explicitly in the 2025 paper on confidence-calibrated reasoning models, which proposes both a base GSPG framework and its calibrated extension C$^2$GSPG (Liu et al., 27 Sep 2025). In that work, GSPG is presented as a framework for learning reasoning models that moves optimization from the token level to the sequence level in order to remove token-level bias and to support confidence-aware training.

A distinct but closely allied line of work introduces Group Sequence Policy Optimization (GSPO) rather than GSPG. That paper states unambiguously that GSPG is not the paper’s name and that GSPO should be understood as the formal method name; it also notes that a phrase such as “Group Sequence Policy Gradient” would be closely related and is best interpreted as a gradient-based variant/description of GSPO, rather than as a separate algorithm in that paper (Zheng et al., 24 Jul 2025).

The broader methodological context is a 2025–2026 transition from token-level RL objectives such as GRPO toward group-based and sequence-aware objectives. This transition includes: critic-free grouped baselines for general MDPs in GPG (Chen et al., 4 Oct 2025); sequence-level communication optimization in Agent-GSPO (Fan et al., 26 Oct 2025); and structure-aware sub-sequence optimization in GSsPO for multi-turn Think-Action workflows (Kong et al., 1 Feb 2026). This suggests a family resemblance rather than a single canonical definition: GSPG names one concrete sequence-level framework, while GSPO, GPG, and GSsPO instantiate adjacent design choices at different optimization granularities.

2. Core formulation of GSPG

In the GSPG formulation, for a prompt $q$ and generated sequence $o_i = \{o_{i,t}\}_{t=1}^{|o_i|}$, the model confidence for the entire sequence is defined as the normalized sequence-level probability

$$c_{\theta,i} := \pi_\theta(o_i|q)^{\frac{1}{|o_i|}} = \exp\!\Big(\frac{1}{|o_i|}\sum_{t=1}^{|o_i|}\log \pi_{\theta,i,t}\Big),$$

where $$\pi_{\theta,i,t} = \pi_\theta(o_{i,t}\mid o_{i,<t}, q)$$ (Liu et al., 27 Sep 2025). The normalization by $\frac{1}{|o_i|}$ removes length dependence.

The policy term is sequence-level. In the calibrated presentation, it takes the form

$\tilde{A}_i \cdot \log c_{\theta,i},$

where $\tilde{A}_i$ is a confidence-modulated advantage (Liu et al., 27 Sep 2025). For binary rewards, the paper defines

$$\tilde{A}_i = \frac{r_i - m}{1 - c_{\theta_{\text{old},i}}},$$

with $^2$0 the sequence reward, $^2$1 the group mean reward, and $^2$2 the confidence under the old policy (Liu et al., 27 Sep 2025).

This differs from GRPO’s variance-normalized group advantage

$^2$3

and the paper argues that GSPG’s confidence-based denominator gives less suppression of low-confidence sequences and avoids the gradient diminishing effect on those examples (Liu et al., 27 Sep 2025). The same paper explicitly contrasts this with GSPO, stating that GSPO uses a sequence-level ratio $^2$4, whereas GSPG uses the confidence-modulated advantage $^2$5 and does not rely on variance normalization (Liu et al., 27 Sep 2025).

A related sequence-level objective from GSPO uses a length-normalized sequence likelihood ratio

$^2$6

and a PPO-like clipped surrogate at the whole-sequence level

$^2$7

(Zheng et al., 24 Jul 2025). This provides the closest neighboring formalism to GSPG.

3. Motivation: token-level bias and sequence-level credit assignment

The central motivation for GSPG is the claim that existing token-level RL objectives for reasoning models—especially GRPO and related methods—tend to create two problems: token-level bias and overconfidence / miscalibration (Liu et al., 27 Sep 2025). In the paper’s diagnosis, GRPO applies policy gradients through per-token probability ratios, which can cause low-probability tokens to dominate training, making updates uneven across a sequence. The remedy is to optimize using a single sequence-level scalar, so that an entire reasoning trajectory is treated as a coherent unit.

The closely related GSPO paper frames a similar diagnosis in different terms. It argues that GRPO’s token-level importance ratios are unstable, especially for long responses, large models, Mixture-of-Experts (MoE) models, and settings with mini-batch reuse from rollout data (Zheng et al., 24 Jul 2025). Its core claim is that reward is sequence-level, so the importance ratio should also be sequence-level, and clipping should also be sequence-level. That paper further states that token-level ratios can introduce high-variance gradient noise, accumulate instability over long sequences, and can ultimately cause catastrophic model collapse (Zheng et al., 24 Jul 2025).

In a more general MDP setting, Group Policy Gradient (GPG) presents the same family of ideas in critic-free form: it is described as PPO with a group-estimated baseline, replacing a learned value function with a group-based Monte Carlo baseline while preserving the clipped-policy-update structure (Chen et al., 4 Oct 2025). That work does not present GPG as sequence-level by definition, but it explicitly connects grouped baselines to GRPO and related RLHF formulations. A plausible implication is that GSPG can be read as the sequence-specific specialization of a broader group-baseline principle: critic-free grouping can replace critic learning when relative comparisons among samples are informative.

4. Confidence calibration and the C$^2$8GSPG extension

C$^2$9GSPG extends GSPG by adding a cross-entropy / binary cross-entropy regularizer that aligns sequence confidence with reward (Liu et al., 27 Sep 2025). For binary rewards, the objective is

$q$0

where $q$1 controls regularization strength (Liu et al., 27 Sep 2025).

The interpretation given in the paper is direct. If $q$2, the BCE term encourages $q$3; if $q$4, it encourages $q$5. The stated purpose is to make the model’s confidence behave like a calibrated probability of correctness (Liu et al., 27 Sep 2025).

A central theoretical claim of the paper is that, for binary rewards, the policy-gradient term and the confidence-calibration term always share the same sign. Using

$q$6

the paper derives a gradient in which the GSPG term and BCE term are collaborative, not adversarial (Liu et al., 27 Sep 2025). The stated consequence is that correct sequences become more confident and incorrect sequences become less confident.

For non-binary rewards, the paper states that this alignment does not automatically extend. It therefore introduces two mechanisms: nonlinear reward normalization with

$q$7

and adaptive regularizer clipping

$q$8

The full non-binary objective applies the BCE-style term only when it agrees with the policy signal (Liu et al., 27 Sep 2025). This is presented as a method for mitigating objective conflict in graded-reward settings such as logic tasks with rewards like $q$9.

5. Empirical findings and benchmarked behavior

The GSPG paper reports experiments on logical reasoning and mathematical reasoning tasks, comparing C$o_i = \{o_{i,t}\}_{t=1}^{|o_i|}$0GSPG to GRPO, GPG, GSPO, and AR-Lopti (Liu et al., 27 Sep 2025). On K$o_i = \{o_{i,t}\}_{t=1}^{|o_i|}$1K / Knights and Knaves with Qwen2.5-3B-Instruct, the paper reports for C$o_i = \{o_{i,t}\}_{t=1}^{|o_i|}$2GSPG: Accuracy: 78.97 ± 7.19, BS: 0.136 ± 0.046, and ECE: 0.126 ± 0.053. The best listed baseline is AR-Lopti with Accuracy 56.61, BS 0.282, and ECE 0.343 (Liu et al., 27 Sep 2025). The same paper emphasizes reliability diagrams close to the perfect diagonal, a more bimodal confidence distribution, and ablations showing that sigmoid normalization, adaptive clipping, and the full method are best (Liu et al., 27 Sep 2025).

Related sequence-level results from GSPO reinforce the broader empirical case for sequence-level optimization. The GSPO paper reports stable training, continuous performance improvement with more training compute, updated query sets, and longer generation lengths; it also states that GSPO beats GRPO in training accuracy, benchmark performance, and efficiency per compute and per query, and that GSPO eliminates the need for Routing Replay in MoE training (Zheng et al., 24 Jul 2025).

An application-specific extension appears in Agent-GSPO, which adapts GSPO to multi-agent communication by treating an utterance as a sequence-level action and optimizing a communication-aware reward that penalizes verbosity (Fan et al., 26 Oct 2025). Across seven reasoning benchmarks, the paper reports the best results on MMLU, MultiArith, GSM8K, SVAMP, AQuA, HumanEval, and MATH-500. It gives concrete examples: 96.02\% on GSM8K, 90.70\% pass@1 on HumanEval, and 83.10\% on MATH-500 (Fan et al., 26 Oct 2025). It also reports token reductions such as $o_i = \{o_{i,t}\}_{t=1}^{|o_i|}$3 tokens on MMLU versus $o_i = \{o_{i,t}\}_{t=1}^{|o_i|}$4 for LLM-Debate, $o_i = \{o_{i,t}\}_{t=1}^{|o_i|}$5 for PHP, and $o_i = \{o_{i,t}\}_{t=1}^{|o_i|}$6 for DyLAN, as well as $o_i = \{o_{i,t}\}_{t=1}^{|o_i|}$7 tokens on GSM8K versus $o_i = \{o_{i,t}\}_{t=1}^{|o_i|}$8, $o_i = \{o_{i,t}\}_{t=1}^{|o_i|}$9, and $$c_{\theta,i} := \pi_\theta(o_i|q)^{\frac{1}{|o_i|}} = \exp\!\Big(\frac{1}{|o_i|}\sum_{t=1}^{|o_i|}\log \pi_{\theta,i,t}\Big),$$0, respectively (Fan et al., 26 Oct 2025). This does not demonstrate GSPG directly, but it does demonstrate the practical efficacy of the closely related sequence-level group-optimization family.

6. Relations to adjacent methods and granularity variants

GSPG is most naturally contrasted with GRPO, GSPO, GPG, and GSsPO.

Method	Optimization unit	Distinctive feature
GRPO	token level	per-token ratios with group-relative advantage
GSPG	sequence level	normalized sequence-level probability as confidence
GSPO	sequence level	sequence-level likelihood ratio and sequence-level clipping
GPG	grouped trajectories in general MDPs	critic-free group-estimated baseline
GSsPO	sub-sequence level	Think-Action pair as atomic unit

Within this family, GSPO is the closest formal neighbor. Its defining move is to compute a sequence-level importance ratio, use group-relative reward normalization, and apply sequence-level clipping rather than token-level clipping (Zheng et al., 24 Jul 2025). GSPG differs in that its calibrated version is centered on the sequence confidence $$c_{\theta,i} := \pi_\theta(o_i|q)^{\frac{1}{|o_i|}} = \exp\!\Big(\frac{1}{|o_i|}\sum_{t=1}^{|o_i|}\log \pi_{\theta,i,t}\Big),$$1 and a confidence-dependent denominator $$c_{\theta,i} := \pi_\theta(o_i|q)^{\frac{1}{|o_i|}} = \exp\!\Big(\frac{1}{|o_i|}\sum_{t=1}^{|o_i|}\log \pi_{\theta,i,t}\Big),$$2, rather than on variance-normalized advantages and sequence likelihood ratios (Liu et al., 27 Sep 2025).

GPG, by contrast, is not tied to language sequences per se. It is a critic-free policy-gradient method for general MDPs in which grouped trajectories and a binning function provide the baseline, and the paper proves a consistency theorem as group size $$c_{\theta,i} := \pi_\theta(o_i|q)^{\frac{1}{|o_i|}} = \exp\!\Big(\frac{1}{|o_i|}\sum_{t=1}^{|o_i|}\log \pi_{\theta,i,t}\Big),$$3 (Chen et al., 4 Oct 2025). This suggests a general principle: group baselines can substitute for learned critics in settings with sufficient parallel samples.

GSsPO further refines the granularity choice for multi-turn agentic workflows. That method argues that token-level optimization is too fine and full-sequence optimization too coarse for interleaved Think-Action interactions; it therefore optimizes a sub-sequence, defined as one contiguous Think-Action pair per turn (Kong et al., 1 Feb 2026). A plausible implication is that GSPG belongs to a wider search for the semantically appropriate unit of credit assignment: token, sequence, or structured sub-sequence.

A broader theoretical framing appears in “GPG: Generalized Policy Gradient Theorem for Transformer-based Policies,” which states that both standard Policy Gradient Theorem and GRPO emerge as special cases of a macro-action policy-gradient theorem for Transformer policies (Mao et al., 11 Dec 2025). That work does not define GSPG by name, but it formalizes a continuum between token-level and sequence-level/group-like policy gradients. This suggests that GSPG can be viewed as one point in a general segmentation space for autoregressive policy optimization.

7. Interpretation, misconceptions, and significance

A recurrent misconception is that GSPG and GSPO are interchangeable names for a single algorithm. The record in the cited papers is more precise. GSPG is explicitly named in the confidence-calibration paper (Liu et al., 27 Sep 2025), whereas the 2025 sequence-level clipping paper explicitly says that the correct term is GSPO, and that “Group Sequence Policy Gradient” is best understood there as a descriptive misnaming or near-synonym for the gradient view of GSPO, not a distinct algorithm (Zheng et al., 24 Jul 2025).

Another misconception is that these methods are merely token-level policy gradients with sequence rewards attached after the fact. The cited papers argue the opposite: the optimization unit itself is changed. In GSPG, the update uses a sequence-level confidence scalar (Liu et al., 27 Sep 2025). In GSPO, it uses a whole-sequence likelihood ratio and whole-sequence clipping (Zheng et al., 24 Jul 2025). In GSsPO, it uses a sub-sequence-level importance ratio for Think-Action units (Kong et al., 1 Feb 2026). The technical claim common to these methods is that the granularity of optimization should match the semantic or reward granularity of the task.

The significance of GSPG therefore lies not only in its particular confidence-modulated objective, but also in what it represents within post-training RL for reasoning models: a move away from token-wise correction toward group-relative optimization over semantically meaningful units. The empirical record reported for C$$c_{\theta,i} := \pi_\theta(o_i|q)^{\frac{1}{|o_i|}} = \exp\!\Big(\frac{1}{|o_i|}\sum_{t=1}^{|o_i|}\log \pi_{\theta,i,t}\Big),$$4GSPG—improvements in both reasoning accuracy and confidence calibration (Liu et al., 27 Sep 2025)—and the allied results for GSPO in sequence-level stability, MoE training, and communication efficiency (Zheng et al., 24 Jul 2025, Fan et al., 26 Oct 2025) indicate that sequence-level group-based optimization has become a distinct and expanding research direction.