Stable GRPO (S-GRPO) Methods

• S-GRPO is a reinforcement learning approach that enhances GRPO stability using noise-aware reweighting, stratified normalization, and robust divergence control.
• It employs probabilistic conflict resolution and scalarized advantage construction to ensure faithful credit assignment and smooth convergence.
• Empirical evaluations demonstrate that S-GRPO improves accuracy and stability under adversarial noise and heterogeneous reward settings in diverse applications.

Stable Group-Relative Policy Optimization (S-GRPO) encompasses a family of algorithmic advances designed to overcome the instability and brittleness of Group-Relative Policy Optimization (GRPO)—a class of critic-free reinforcement learning methods underpinning large-scale reasoning, generation, and alignment tasks across LLMs and multimodal generative models. S-GRPO mechanisms primarily address credit assignment pathologies arising from noisy or structurally heterogeneous rewards, adversarial statistic skew, policy divergence, or unconstrained gradient dynamics. By introducing noise-aware reweighting, stratified normalization, probabilistic conflict arbitration, and robust divergence constraints, S-GRPO establishes consistent convergence, improved noise tolerance, and more faithful credit assignment across modalities and RL settings (Shen et al., 8 Aug 2025, Girgis et al., 5 Feb 2026, Qiang et al., 6 Feb 2026).

1. Noise-Aware Advantage Reweighting

S-GRPO was initially motivated by the “Think-Answer Mismatch” in GRPO-based LLM reasoning. Here, group-wise standardization of binary outcome rewards allows noisy or flawed reasoning chains—yielding correct answers only by chance—to disproportionately contaminate the batch statistics. Under moderate symmetric label noise (e.g., 20%), this mismatch inflates normalized advantages and can stall learning, especially in unbalanced groups (e.g., 1 correct in 8) (Shen et al., 8 Aug 2025).

To counteract this, S-GRPO introduces analytic, group-specific advantage weights based on a closed-form minimization of the expected squared error between observed (noisy) and latent (true) advantages under the assumed label noise model:

$$w^\star(N, k, p) = \frac{ (1-2p) t (1-t) }{ \sqrt{ \bar r (1 - \bar r) + \epsilon } \sqrt{ t (1 - t) + \epsilon } },\quad t = \frac{k/N - p}{1-2p}$$

where $k$ is the number of observed correct responses in group of size $N$, $p$ is the noise rate, and $t$ the inferred true rate clipped to $[0,1]$. This weight attenuates updates from highly imbalanced or noisy groups, achieves consensus-amplified confidence in balanced batches, and provably regularizes training towards minimal mean-squared advantage risk. The full surrogate loss for each sample becomes:

$$\mathcal{L}_i(\theta) = \min \left[ \operatorname{ratio}_i(\theta) w^\star a_i, \operatorname{clip}(\operatorname{ratio}_i(\theta), 1 - \epsilon, 1 + \epsilon) w^\star a_i \right]$$

Extensive empirical validation on math reasoning models (Qwen2.5-Math-1.5B, Llama-3.2-3B) demonstrated S-GRPO’s robust stability and superior pass@1 accuracy, especially under adversarial noise (up to 20% flip), where standard GRPO failed to make learning progress (Shen et al., 8 Aug 2025).

2. Enhanced Credit Assignment via Stratified and Scalarized Normalization

S-GRPO generalizes advantage normalization by accounting for structural heterogeneity. In agents operating under multi-objective or constrained tasks—or with stratified trajectory structure (e.g., number/placement of search tool use)—global normalization induces “cross-stratum bias”: advantages computed globally mix trajectories with incommensurate statistics, distorting credit assignment and harming exploration (Zhu et al., 7 Oct 2025).

Stratified S-GRPO (Editors' term: SAN-GRPO) groups trajectories into homogeneous strata (e.g., by number of external tool calls, constraint types, etc.), computes empirical mean and std locally per stratum, and normalizes returns strictly within these partitions. The estimator

$$A_{\text{SAN}}(\tau) = \frac{R(\tau) - \hat{\mu}_k}{\hat{\sigma}_k + \epsilon},\quad \tau \in \text{stratum } k$$

eliminates cross-stratum bias, guarantees per-stratum unbiasedness and unit variance, and retains the global unbiasedness property of standard normalization. Empirically, this correction enables S-GRPO to achieve up to +11.3 EM over baseline GRPO on multi-hop QA, with improvements dominant on structurally diverse tasks (Zhu et al., 7 Oct 2025).

For constrained policy optimization (e.g., CMDPs), S-GRPO employs “scalarized advantage construction”: each component (reward, constraint cost) is standardized, and the single pre-clipped advantage is their weighted sum. This preserves Lagrange multiplier semantics and optimization control, as opposed to naive multi-component normalization that distorts constraint trade-offs due to variance mismatch (Girgis et al., 5 Feb 2026).

3. Probabilistic Conflict Resolution for Plasticity–Stability Dilemma

A fundamental instability in GRPO for LLMs arises from geometric conflict between the task-specific “plasticity” gradient and the KL-constrained “stability” gradient. Deterministic projection remedies (e.g., PCGrad) are suboptimal due to gradient estimation noise. S-GRPO formalizes this as a Bayesian inference problem: gradients are modeled as random variables, with their orthogonal decomposition resolved via uncertainty-aware “soft projection” (Qiang et al., 6 Feb 2026):

$$g_{\text{final}} = \mu_{\text{pla}} - \alpha \frac{ \mu_{\text{pla}} \cdot \mu_{\text{sta}} }{ \|\mu_{\text{sta}}\|^2 } \mu_{\text{sta}}$$

where $$\alpha = \sigma_{\text{sta}}^2 / (\sigma_{\text{pla}}^2 + \sigma_{\text{sta}}^2)$$ dynamically interpolates between full projection and naive gradient addition, optimizing the mean-squared update error (MMSE). This results in provably lower update noise, empirically smoother training curves, and improved retention of both reasoning competence and pre-trained generalization (Qiang et al., 6 Feb 2026).

4. Robust Policy Divergence Control

Policy divergence control in GRPO is typically achieved through clipping of the importance ratio. S-GRPO, under the “ATR-GRPO” or “unified clipping” framework, interprets this as a special case of imposing a divergence constraint via a sample-level estimator—specifically, the KL₃ estimator:

$k$0

with $k$1. Imposing the per-step constraint $k$2 leads to asymmetric adaptive ratio clipping, which both stabilizes updates and encourages exploration beyond the rigid boundaries of traditional clipping (Wu et al., 5 Feb 2026).

These divergence-aware surrogates bound single-step update magnitude, guarantee trust-region behavior, and empirically raise convergence speed and final accuracy in mathematical reasoning tasks (Wu et al., 5 Feb 2026).

5. S-GRPO in Diverse Generative and RLHF Applications

S-GRPO principles have been instantiated across multiple generative settings, including:

• Intrinsic, annotation-free reward design: SR-GRPO leverages the stable rank of hidden state representations as a geometric reward, yielding high-quality, unsupervised preference alignment in LLMs that outperforms learned reward models on diverse STEM, math, and conversational domains. This method achieves robust group-wise learning signals and in-distribution proxy accuracies exceeding 84% (Tang et al., 2 Dec 2025).
• Decaying early exit for chain-of-thought RL: S-GRPO variants in reasoning combine a serial-group rollout structure with exponentially decaying reward for earlier correct exits, directly incentivizing concise and efficient thought generation while preserving or improving task accuracy (Dai et al., 12 May 2025).
• Branching and self-paced curricula in diffusion models: In diffusion language and image models, S-GRPO applies branch sampling, tree-based reward fusion, depth normalization, and curriculum-driven reward blending to stabilize training under sparse or saturated external rewards. These methods demonstrably reduce computational cost, gradient variance, and reward collapse (Li et al., 7 Sep 2025, Li et al., 24 Nov 2025).

6. Algorithmic Structure and Implementation

The S-GRPO family is characterized by the following core structural elements:

• Group-wise advantage normalization, optionally stratified or noise-attenuated
• Variance-minimizing or noise-corrected advantage reweighting (analytic closed form or empirical attenuation)
• Principled policy divergence control (ratio-based, KL₃, or self-normalization in gradient estimation)
• Flexible surrogate objectives retaining PPO-style clipped importance weights, but augmented with calculated scaling or projection
• Empirical batch-level operations (stratification, per-stratum statistics, group size adaptation)
• Adaptive policy refresh and step-size scheduling (where applicable)

A canonical S-GRPO training loop (noise-aware variant) as per (Shen et al., 8 Aug 2025):

$k$3

This structure is adapted to the specific application domain, reward signal, and normalization scheme.

7. Limitations, Open Problems, and Extensions

Theoretical analyses and empirical investigations consistently affirm S-GRPO’s stabilization, yet key limitations remain:

• Most noise-aware variants assume symmetric, known label noise; online adaptive estimation or handling asymmetric noise remains open (Shen et al., 8 Aug 2025).
• Finite-sample regret and convergence bounds are not generally available for group-normalized or stratified settings, though partial results under smoothness and bounded-variance are established (Pang et al., 4 Aug 2025, Girgis et al., 5 Feb 2026).
• Extensions to off-policy, multi-agent, and nonstationary or adversarial reward environments pose unresolved methodological challenges.
• Integration with process-level supervision, step-by-step feedback, or hybrid reward heuristics is hypothesized to further enhance sample efficiency and credit assignment but requires systematic study (Shen et al., 8 Aug 2025, Tang et al., 2 Dec 2025, Li et al., 24 Nov 2025).

Nevertheless, S-GRPO provides a robust, theoretically principled, and empirically validated framework for stable, efficient RL-based optimization of language and generative models across a range of application domains and optimization paradigms (Shen et al., 8 Aug 2025, Girgis et al., 5 Feb 2026, Qiang et al., 6 Feb 2026, Tang et al., 2 Dec 2025).