---
title: Stable Group-Relative Policy Optimization (S-GRPO)
url: https://www.emergentmind.com/topics/stable-group-relative-policy-optimization-s-grpo
type: topic
---

# Stable Group-Relative Policy Optimization (S-GRPO)

Stable Group-Relative Policy Optimization (S-GRPO) is a modification of Group-Relative Policy Optimization (GRPO) for RL-style post-training of reasoning language models with binary outcome rewards. It keeps GRPO’s critic-free, PPO-style clipped update, but inserts a closed-form, noise-aware group weight \(w^\star\) that rescales standardized group advantages under an explicit symmetric reward-noise model. The method is designed for the setting in which final-answer correctness is only a noisy proxy for reasoning quality, a discrepancy formalized as the **Think-Answer Mismatch**. In that setting, standard GRPO can amplify reward corruption precisely on highly imbalanced response groups, where its normalized signal would otherwise be strongest; S-GRPO suppresses those unreliable updates while preserving GRPO’s group-relative comparison structure [2508.05928].

## 1. Problem formulation and the Think-Answer Mismatch

S-GRPO is introduced in the standard post-training regime for reasoning LLMs. For each query \(q\), the policy \(\pi_\theta\) samples a group of responses \(\{o_i\}_{i=1}^N\). Each response receives a binary observed reward \(r_i \in \{0,1\}\), typically indicating whether the final answer is correct. Standard GRPO computes a relative advantage from the reward statistics of that sampled group and uses a PPO-style clipped objective, thereby avoiding a learned critic [2508.05928].

The method’s motivating diagnosis is that outcome reward is often an imperfect proxy for reasoning quality. A response can contain flawed reasoning but still end at the correct answer, or contain largely good reasoning but finish with an incorrect answer. The paper formalizes this by introducing an unobserved latent true reward \(r_i^* \in \{0,1\}\), while GRPO observes only \(r_i\). This mismatch is not treated as mild label noise. Because GRPO standardizes rewards within a group, a single corrupted reward changes not only one sample’s label, but also the group mean and the group scale, thereby perturbing the normalized advantages of all samples in that group [2508.05928].

For a group of size \(N\) with \(k\) observed positive rewards, standard GRPO assigns the same standardized advantage to all positives and another to all negatives:
\[
a_{\text{pos}} = \frac{N-k}{\sqrt{k(N-k)}}, \qquad
a_{\text{neg}} = \frac{-k}{\sqrt{k(N-k)}}.
\]
If one of the \(k\) positives is in fact a mismatch, the true group composition is different, and all positive and negative advantages shift accordingly. The paper’s central qualitative finding is that the resulting total deviation is **U-shaped** in group composition and is worst for highly imbalanced groups, such as \(1/7\), rather than balanced groups such as \(4/4\). This means that standard GRPO tends to trust most strongly exactly those groups that are least reliable under outcome noise [2508.05928].

This diagnosis places S-GRPO in a narrower niche than generic “stable RL” methods. Its target is neither generic variance reduction nor trust-region control in the abstract; it is specifically the corruption of group-relative normalization by noisy outcome labels in reasoning tasks.

## 2. Standard GRPO and the S-GRPO reformulation

The starting point is the GRPO group-relative advantage for binary rewards:
\[
a_i = \frac{r_i - \bar r}{\sqrt{\bar r(1-\bar r)+\epsilon}},
\qquad
\bar r = \frac{1}{N}\sum_{i=1}^N r_i.
\]
With a PPO-style ratio
\[
\mathrm{ratio}_i(\theta)=\frac{\pi_\theta(o_i\mid q)}{\pi_{\theta_{\text{old}}}(o_i\mid q)},
\]
standard GRPO uses the per-sample clipped surrogate
\[
\mathcal{L}_i^{\text{GRPO}}(\theta)=
\min\Big(
\mathrm{ratio}_i(\theta)\, a_i,\;
\mathrm{clip}(\mathrm{ratio}_i(\theta),1-\epsilon,1+\epsilon)\, a_i
\Big).
\]
S-GRPO preserves this objective form but multiplies the group-relative advantage by a scalar group weight \(w^\star\):
\[
\mathcal{L}_i^{\text{S-GRPO}}(\theta)=
\min\Big(
\mathrm{ratio}_i(\theta)\, w^\star a_i,\;
\mathrm{clip}(\mathrm{ratio}_i(\theta),1-\epsilon,1+\epsilon)\, w^\star a_i
\Big).
\]
Thus the change is not architectural and not token-specific. It is a group-level rescaling of the observed standardized advantage [2508.05928].

The derivation assumes symmetric independent label noise:
\[
\mathbb{P}(r_i \ne r_i^*) = p, \qquad 0 \le p < 0.5.
\]
If \(t=\mathbb{E}[r_i^*]\) is the latent clean mean reward within the group, then the observed mean obeys
\[
\bar r = (1-p)t + p(1-t),
\]
so the latent mean is estimated by
\[
t = \frac{\bar r - p}{1-2p},
\]
with \(t\) clipped to \([0,1]\). The paper then defines observed and latent standardized advantages:
\[
a_i = \frac{r_i-\bar r}{\sigma_r}, \qquad
a_i^* = \frac{r_i^*-t}{\sigma_t},
\]
with
\[
\sigma_r^2 = \bar r(1-\bar r)+\epsilon, \qquad
\sigma_t^2 = t(1-t)+\epsilon.
\]

The scalar weight is chosen by a mean-squared approximation principle:
\[
w^\star = \arg\min_w \mathbb{E}\big[(w a_i - a_i^*)^2\big].
\]
Because both \(a_i\) and \(a_i^*\) are standardized, the minimizer becomes the covariance, equivalently the correlation, between observed and latent standardized rewards. Under the symmetric flip model, the final closed form is
\[
w^\star(N,k,p)
=
\frac{(1-2p)t(1-t)}
{\sqrt{\bar r(1-\bar r)+\epsilon}\sqrt{t(1-t)+\epsilon}}.
\]
S-GRPO then uses the reweighted advantage
\[
\tilde a_i = w^\star a_i.
\]
The paper is explicit that this “optimality” is narrow: \(w^\star\) is optimal only in the sense of minimizing \(\mathbb{E}[(w a_i-a_i^*)^2]\) under the chosen noise model; it is not presented as a convergence theorem or a likelihood-ratio correction [2508.05928].

## 3. Interpretation of the weight and algorithmic behavior

The weight \(w^\star\) has three behaviors that define the algorithm’s practical character. First, it is **noise-adaptive attenuation**: since it is bounded by \(1-2p\), larger assumed noise reduces update magnitudes globally. In the noiseless case \(p=0\), the paper states that \(w^\star=1\), so S-GRPO reduces exactly to standard GRPO. Second, it expresses **confidence through consensus**: the weight is largest for balanced groups and smallest for highly imbalanced groups. Third, it creates a **noise-gating effect**: when the observed success rate is too extreme relative to the assumed noise floor, the clipped latent mean collapses toward \(0\) or \(1\), and the group’s update can be suppressed entirely [2508.05928].

The paper gives concrete gating examples. For \(N=16\) and \(p=0.20\), groups with \(k \le 3\) or \(k \ge 13\) receive zero weight. For the common training setting \(N=8\), at \(p=0.15\), extreme groups \(k \in \{1,7\}\) receive zero weight. This is the opposite of standard GRPO’s bias: GRPO’s z-score becomes large on extreme groups, while S-GRPO interprets those same groups as unreliable under outcome noise and attenuates or removes their contribution [2508.05928].

Algorithmically, the procedure is lightweight. For each prompt, sample \(N\) responses, compute binary rewards, form \(\bar r\), estimate \(t\), compute \(\sigma_r\) and \(\sigma_t\), construct the standard GRPO advantage \(a_i\), compute \(w^\star\), and insert \(w^\star a_i\) into the usual PPO-style clipped surrogate. The only new input beyond standard GRPO is the assumed mismatch rate \(p\). The paper states that the computational overhead is negligible because the extra work consists only of a few scalar operations per group [2508.05928].

A common misreading is that S-GRPO attempts per-sample label correction. It does not. The method does not infer corrected binary labels for individual responses; it estimates how much the *group-relative advantage* should be trusted as a proxy for the latent clean advantage. This distinction is central to its design.

## 4. Empirical results and robustness claims

The empirical setting is mathematical reasoning RL. Training uses 8,500 MATH problems of difficulty 3–5. Evaluation is on AMC, MATH500, Minerva, and OlympiadBench with Pass@1 under greedy decoding. The reported models are Qwen2.5-Math-7B-Base, Qwen2.5-Math-1.5B-Instruct, and Llama-3.2-3B-Base [2508.05928].

Across those models, S-GRPO consistently improves over standard GRPO and Dr. GRPO. On Qwen2.5-Math-1.5B-Instruct, average benchmark accuracy rises from \(46.3\%\) with GRPO to \(47.3\%\) with Dr. GRPO and \(49.7\%\) with S-GRPO. On Llama-3.2-3B-Base, the average rises from \(8.3\%\) with GRPO to \(9.8\%\) with S-GRPO, which the paper describes as \(+1.5\) over GRPO and \(+2.2\) over reported Dr. GRPO. On Qwen2.5-Math-7B-Base, average accuracy rises from \(51.5\%\) with GRPO and \(53.5\%\) with Dr. GRPO to \(56.0\%\) with S-GRPO; the headline gains versus Dr. GRPO are \(+2.5\%\) on Qwen-Math-7B-Base, \(+2.2\%\) on Llama-3.2-3B-Base, and \(+2.4\%\) on Qwen-Math-1.5B-Instruct [2508.05928].

The robustness experiments are the paper’s strongest evidence. Under \(10\%\) synthetic reward noise, standard GRPO learns much more slowly than S-GRPO. Under \(20\%\) synthetic noise, the paper states that standard GRPO “fails to learn under 20% synthetic reward noise” and can collapse entirely during the first 100 training steps, whereas S-GRPO still shows stable progress. This result is presented as direct evidence that the noise-aware weighting is doing what it is intended to do: suppressing unreliable updates from corrupted outcome rewards [2508.05928].

The ablations on \(p\) expose a stability-speed tradeoff. On Qwen2.5-Math-7B-Base, \(p=0.10\) reaches roughly \(54\%\) within 50 steps but shows a temporary later dip, while \(p=0.15\) starts lower and improves more monotonically. The paper interprets this via the gating mechanism: larger \(p\) suppresses more extreme groups, sacrificing early aggressiveness for stability. It also reports model-size dependence: the smaller 1.5B model performs best at \(p=0.15\), while the 7B model performs best at \(p=0.10\), consistent with the claim that weaker models exhibit higher effective mismatch rates [2508.05928].

Additional analyses are supportive but indirect. Entropy trajectories under S-GRPO are smoother than under \(p=0\) training. The paper also reports increased frequency of self-reflection keywords and longer responses under S-GRPO, and qualitative case studies in which GRPO-trained outputs exhibit incoherent or hallucinated mathematics while S-GRPO-trained outputs preserve a more sensible reasoning scaffold even when imperfect [2508.05928].

## 5. Assumptions, scope, and limitations

S-GRPO is derived under a specific probabilistic model: binary latent rewards, symmetric independent flips with a single scalar noise rate \(p\), and group-wise standardized advantages. The paper is explicit that real Think-Answer Mismatch may be more complicated. In particular, mismatch may be asymmetric: flawed reasoning may still end with a correct answer more often than good reasoning ends with an incorrect one, or error rates may depend on question difficulty, response length, or model confidence. The authors identify asymmetric noise models and integration with process supervision as future work [2508.05928].

The method also introduces a new hyperparameter-selection problem. If \(p\) is too small, robustness gains are limited; if \(p\) is too large, informative extreme groups are over-suppressed. This is not a secondary issue, because S-GRPO’s core behavior—attenuation, balance preference, and gating—is all mediated by \(p\). The ablations indicate that the preferred value can depend on model scale [2508.05928].

A second limitation is theoretical scope. The paper provides a closed-form estimator under a chosen denoising objective, not a general convergence guarantee. It does not present concentration bounds, a variance decomposition for the full policy gradient, or a theorem establishing optimality beyond the MSE projection from \(a_i\) to \(a_i^*\). Accordingly, the strongest claims are empirical robustness and mechanistic plausibility, not full RL theory [2508.05928].

A third boundary is task scope. The target regime is verifier-like outcome reward corruption. The paper states that the logic extends naturally to “any binary reward channel subject to symmetric flips,” but its experiments and derivation are centered on outcome-level correctness signals, not on process reward models or dense reward functions. S-GRPO should therefore be understood as a specialized stabilization method for noisy binary group-relative training rather than a universal replacement for GRPO in all regimes [2508.05928].

## 6. Position within the broader GRPO stabilization literature

S-GRPO belongs to a broader family of GRPO modifications that target different instability sources rather than a single unified “stable GRPO” recipe. In the closest neighboring line, **F-GRPO** studies finite-group RLVR and shows that standard group-relative updates can bias learning toward already-common correct trajectories while shrinking unsampled rare-correct mass; it introduces a difficulty-aware scaling coefficient, inspired by focal loss, to down-weight updates on high-success prompts without increasing group size [2602.06717]. This addresses a different failure mode from S-GRPO: finite-group mode concentration rather than noisy label corruption.

Other works stabilize GRPO by changing the update geometry rather than the reward model. **MHPO** keeps GRPO’s group-relative advantages but replaces hard clipping with a Log-Fidelity Modulator and a Decoupled Hazard Penalty, arguing that importance-ratio control is the central source of instability in GRPO-based reasoning RL [2603.16929]. **StableDRL** makes a related move for diffusion language models, where importance ratios are only proxy-estimated: it uses unconditional clipping and self-normalization to keep updates inside the convex hull of per-sample gradients and to break the self-reinforcing instability loop caused by noisy ratio estimates [2603.06743].

A separate class of papers addresses instability in the advantage signal itself. **AGPO** handles the zero-variance corner case of GRPO in binary-reward reasoning tasks by assigning fixed \(+1/-1\) advantages to all-correct and all-wrong groups, thereby restoring signal when standard group normalization degenerates [2503.15952]. **MDP-GRPO** analyzes low-variance amplification, mean-centering blindness, and zero-variance collapse in multi-constraint instruction following, and combines multi-temperature sampling, dual-anchor advantages, prospect-theoretic shaping, and asymmetric KL regularization to stabilize discrete, low-dispersion reward regimes [2606.06058]. Compared with these methods, S-GRPO is distinctive in that it leaves the ratio control and the basic GRPO objective unchanged, and instead introduces a denoising weight derived from a reward-corruption model.

Stability can also be undermined by reward heterogeneity rather than noise. **P-GRPO** replaces immediate batch normalization with preference-group-specific historical normalization, arguing that standard GRPO’s exchangeability assumption destabilizes learning when batches mix users or contexts with different reward distributions [2603.10009]. In a more theoretical direction, **“Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statistic”** shows that the GRPO gradient is a second-order U-statistic, derives MSE and suboptimality results, and gives the scaling law
\[
G^* = \sqrt{\frac{c_3}{c_1}},
\]
which frames stability through estimator covariance and group-size tradeoffs rather than only through heuristic design choices [2603.01162].

This broader literature suggests that “stable GRPO” is not a single method family but a cluster of interventions aimed at different structural weaknesses: noisy outcome labels, rare-mode forgetting, hard clipping, heterogeneous reward distributions, low-dispersion group rewards, or proxy-estimated importance ratios. Within that landscape, S-GRPO is the version specialized to **Think-Answer Mismatch** and **binary reward corruption**. Its central idea is narrow but precise: when the group-relative signal itself becomes unreliable, scale the whole group update by an estimate of how much the observed standardized advantage can be trusted as a proxy for the latent clean one [2508.05928].

Source: https://www.emergentmind.com/topics/stable-group-relative-policy-optimization-s-grpo