Stable Group-Relative Policy Optimization (S-GRPO)

• The paper’s main contribution is introducing S-GRPO, which modifies GRPO by scaling group-relative advantages with a noise-aware weight to counteract reward corruption.
• S-GRPO employs a closed-form weight derived from a symmetric noise model, enabling noise-adaptive attenuation and reliable group comparisons.
• Empirical evaluations show that S-GRPO improves accuracy and stability over standard GRPO, especially under high levels of synthetic reward noise.

Stable Group-Relative Policy Optimization (S-GRPO) is a modification of Group-Relative Policy Optimization (GRPO) for RL-style post-training of reasoning LLMs with binary outcome rewards. It keeps GRPO’s critic-free, PPO-style clipped update, but inserts a closed-form, noise-aware group weight $w^\star$ that rescales standardized group advantages under an explicit symmetric reward-noise model. The method is designed for the setting in which final-answer correctness is only a noisy proxy for reasoning quality, a discrepancy formalized as the Think-Answer Mismatch. In that setting, standard GRPO can amplify reward corruption precisely on highly imbalanced response groups, where its normalized signal would otherwise be strongest; S-GRPO suppresses those unreliable updates while preserving GRPO’s group-relative comparison structure (Shen et al., 8 Aug 2025).

1. Problem formulation and the Think-Answer Mismatch

S-GRPO is introduced in the standard post-training regime for reasoning LLMs. For each query $q$, the policy $\pi_\theta$ samples a group of responses $\{o_i\}_{i=1}^N$. Each response receives a binary observed reward $r_i \in \{0,1\}$, typically indicating whether the final answer is correct. Standard GRPO computes a relative advantage from the reward statistics of that sampled group and uses a PPO-style clipped objective, thereby avoiding a learned critic (Shen et al., 8 Aug 2025).

The method’s motivating diagnosis is that outcome reward is often an imperfect proxy for reasoning quality. A response can contain flawed reasoning but still end at the correct answer, or contain largely good reasoning but finish with an incorrect answer. The paper formalizes this by introducing an unobserved latent true reward $r_i^* \in \{0,1\}$, while GRPO observes only $r_i$. This mismatch is not treated as mild label noise. Because GRPO standardizes rewards within a group, a single corrupted reward changes not only one sample’s label, but also the group mean and the group scale, thereby perturbing the normalized advantages of all samples in that group (Shen et al., 8 Aug 2025).

For a group of size $N$ with $k$ observed positive rewards, standard GRPO assigns the same standardized advantage to all positives and another to all negatives: $$a_{\text{pos}} = \frac{N-k}{\sqrt{k(N-k)}}, \qquad a_{\text{neg}} = \frac{-k}{\sqrt{k(N-k)}}.$$ If one of the $q$0 positives is in fact a mismatch, the true group composition is different, and all positive and negative advantages shift accordingly. The paper’s central qualitative finding is that the resulting total deviation is U-shaped in group composition and is worst for highly imbalanced groups, such as $q$1, rather than balanced groups such as $q$2. This means that standard GRPO tends to trust most strongly exactly those groups that are least reliable under outcome noise (Shen et al., 8 Aug 2025).

This diagnosis places S-GRPO in a narrower niche than generic “stable RL” methods. Its target is neither generic variance reduction nor trust-region control in the abstract; it is specifically the corruption of group-relative normalization by noisy outcome labels in reasoning tasks.

2. Standard GRPO and the S-GRPO reformulation

The starting point is the GRPO group-relative advantage for binary rewards: $q$3 With a PPO-style ratio

$q$4

standard GRPO uses the per-sample clipped surrogate

$q$5

S-GRPO preserves this objective form but multiplies the group-relative advantage by a scalar group weight $q$6: $q$7 Thus the change is not architectural and not token-specific. It is a group-level rescaling of the observed standardized advantage (Shen et al., 8 Aug 2025).

The derivation assumes symmetric independent label noise: $q$8 If $q$9 is the latent clean mean reward within the group, then the observed mean obeys

$\pi_\theta$0

so the latent mean is estimated by

$\pi_\theta$1

with $\pi_\theta$2 clipped to $\pi_\theta$3. The paper then defines observed and latent standardized advantages: $\pi_\theta$4 with

$\pi_\theta$5

The scalar weight is chosen by a mean-squared approximation principle: $\pi_\theta$6 Because both $\pi_\theta$7 and $\pi_\theta$8 are standardized, the minimizer becomes the covariance, equivalently the correlation, between observed and latent standardized rewards. Under the symmetric flip model, the final closed form is

$\pi_\theta$9

S-GRPO then uses the reweighted advantage

$\{o_i\}_{i=1}^N$0

The paper is explicit that this “optimality” is narrow: $\{o_i\}_{i=1}^N$1 is optimal only in the sense of minimizing $\{o_i\}_{i=1}^N$2 under the chosen noise model; it is not presented as a convergence theorem or a likelihood-ratio correction (Shen et al., 8 Aug 2025).

3. Interpretation of the weight and algorithmic behavior

The weight $\{o_i\}_{i=1}^N$3 has three behaviors that define the algorithm’s practical character. First, it is noise-adaptive attenuation: since it is bounded by $\{o_i\}_{i=1}^N$4, larger assumed noise reduces update magnitudes globally. In the noiseless case $\{o_i\}_{i=1}^N$5, the paper states that $\{o_i\}_{i=1}^N$6, so S-GRPO reduces exactly to standard GRPO. Second, it expresses confidence through consensus: the weight is largest for balanced groups and smallest for highly imbalanced groups. Third, it creates a noise-gating effect: when the observed success rate is too extreme relative to the assumed noise floor, the clipped latent mean collapses toward $\{o_i\}_{i=1}^N$7 or $\{o_i\}_{i=1}^N$8, and the group’s update can be suppressed entirely (Shen et al., 8 Aug 2025).

The paper gives concrete gating examples. For $\{o_i\}_{i=1}^N$9 and $r_i \in \{0,1\}$0, groups with $r_i \in \{0,1\}$1 or $r_i \in \{0,1\}$2 receive zero weight. For the common training setting $r_i \in \{0,1\}$3, at $r_i \in \{0,1\}$4, extreme groups $r_i \in \{0,1\}$5 receive zero weight. This is the opposite of standard GRPO’s bias: GRPO’s z-score becomes large on extreme groups, while S-GRPO interprets those same groups as unreliable under outcome noise and attenuates or removes their contribution (Shen et al., 8 Aug 2025).

Algorithmically, the procedure is lightweight. For each prompt, sample $r_i \in \{0,1\}$6 responses, compute binary rewards, form $r_i \in \{0,1\}$7, estimate $r_i \in \{0,1\}$8, compute $r_i \in \{0,1\}$9 and $r_i^* \in \{0,1\}$0, construct the standard GRPO advantage $r_i^* \in \{0,1\}$1, compute $r_i^* \in \{0,1\}$2, and insert $r_i^* \in \{0,1\}$3 into the usual PPO-style clipped surrogate. The only new input beyond standard GRPO is the assumed mismatch rate $r_i^* \in \{0,1\}$4. The paper states that the computational overhead is negligible because the extra work consists only of a few scalar operations per group (Shen et al., 8 Aug 2025).

A common misreading is that S-GRPO attempts per-sample label correction. It does not. The method does not infer corrected binary labels for individual responses; it estimates how much the group-relative advantage should be trusted as a proxy for the latent clean advantage. This distinction is central to its design.

4. Empirical results and robustness claims

The empirical setting is mathematical reasoning RL. Training uses 8,500 MATH problems of difficulty 3–5. Evaluation is on AMC, MATH500, Minerva, and OlympiadBench with Pass@1 under greedy decoding. The reported models are Qwen2.5-Math-7B-Base, Qwen2.5-Math-1.5B-Instruct, and Llama-3.2-3B-Base (Shen et al., 8 Aug 2025).

Across those models, S-GRPO consistently improves over standard GRPO and Dr. GRPO. On Qwen2.5-Math-1.5B-Instruct, average benchmark accuracy rises from $r_i^* \in \{0,1\}$5 with GRPO to $r_i^* \in \{0,1\}$6 with Dr. GRPO and $r_i^* \in \{0,1\}$7 with S-GRPO. On Llama-3.2-3B-Base, the average rises from $r_i^* \in \{0,1\}$8 with GRPO to $r_i^* \in \{0,1\}$9 with S-GRPO, which the paper describes as $r_i$0 over GRPO and $r_i$1 over reported Dr. GRPO. On Qwen2.5-Math-7B-Base, average accuracy rises from $r_i$2 with GRPO and $r_i$3 with Dr. GRPO to $r_i$4 with S-GRPO; the headline gains versus Dr. GRPO are $r_i$5 on Qwen-Math-7B-Base, $r_i$6 on Llama-3.2-3B-Base, and $r_i$7 on Qwen-Math-1.5B-Instruct (Shen et al., 8 Aug 2025).

The robustness experiments are the paper’s strongest evidence. Under $r_i$8 synthetic reward noise, standard GRPO learns much more slowly than S-GRPO. Under $r_i$9 synthetic noise, the paper states that standard GRPO “fails to learn under 20% synthetic reward noise” and can collapse entirely during the first 100 training steps, whereas S-GRPO still shows stable progress. This result is presented as direct evidence that the noise-aware weighting is doing what it is intended to do: suppressing unreliable updates from corrupted outcome rewards (Shen et al., 8 Aug 2025).

The ablations on $N$0 expose a stability-speed tradeoff. On Qwen2.5-Math-7B-Base, $N$1 reaches roughly $N$2 within 50 steps but shows a temporary later dip, while $N$3 starts lower and improves more monotonically. The paper interprets this via the gating mechanism: larger $N$4 suppresses more extreme groups, sacrificing early aggressiveness for stability. It also reports model-size dependence: the smaller 1.5B model performs best at $N$5, while the 7B model performs best at $N$6, consistent with the claim that weaker models exhibit higher effective mismatch rates (Shen et al., 8 Aug 2025).

Additional analyses are supportive but indirect. Entropy trajectories under S-GRPO are smoother than under $N$7 training. The paper also reports increased frequency of self-reflection keywords and longer responses under S-GRPO, and qualitative case studies in which GRPO-trained outputs exhibit incoherent or hallucinated mathematics while S-GRPO-trained outputs preserve a more sensible reasoning scaffold even when imperfect (Shen et al., 8 Aug 2025).

5. Assumptions, scope, and limitations

S-GRPO is derived under a specific probabilistic model: binary latent rewards, symmetric independent flips with a single scalar noise rate $N$8, and group-wise standardized advantages. The paper is explicit that real Think-Answer Mismatch may be more complicated. In particular, mismatch may be asymmetric: flawed reasoning may still end with a correct answer more often than good reasoning ends with an incorrect one, or error rates may depend on question difficulty, response length, or model confidence. The authors identify asymmetric noise models and integration with process supervision as future work (Shen et al., 8 Aug 2025).

The method also introduces a new hyperparameter-selection problem. If $N$9 is too small, robustness gains are limited; if $k$0 is too large, informative extreme groups are over-suppressed. This is not a secondary issue, because S-GRPO’s core behavior—attenuation, balance preference, and gating—is all mediated by $k$1. The ablations indicate that the preferred value can depend on model scale (Shen et al., 8 Aug 2025).

A second limitation is theoretical scope. The paper provides a closed-form estimator under a chosen denoising objective, not a general convergence guarantee. It does not present concentration bounds, a variance decomposition for the full policy gradient, or a theorem establishing optimality beyond the MSE projection from $k$2 to $k$3. Accordingly, the strongest claims are empirical robustness and mechanistic plausibility, not full RL theory (Shen et al., 8 Aug 2025).

A third boundary is task scope. The target regime is verifier-like outcome reward corruption. The paper states that the logic extends naturally to “any binary reward channel subject to symmetric flips,” but its experiments and derivation are centered on outcome-level correctness signals, not on process reward models or dense reward functions. S-GRPO should therefore be understood as a specialized stabilization method for noisy binary group-relative training rather than a universal replacement for GRPO in all regimes (Shen et al., 8 Aug 2025).

6. Position within the broader GRPO stabilization literature

S-GRPO belongs to a broader family of GRPO modifications that target different instability sources rather than a single unified “stable GRPO” recipe. In the closest neighboring line, F-GRPO studies finite-group RLVR and shows that standard group-relative updates can bias learning toward already-common correct trajectories while shrinking unsampled rare-correct mass; it introduces a difficulty-aware scaling coefficient, inspired by focal loss, to down-weight updates on high-success prompts without increasing group size (Plyusov et al., 6 Feb 2026). This addresses a different failure mode from S-GRPO: finite-group mode concentration rather than noisy label corruption.

Other works stabilize GRPO by changing the update geometry rather than the reward model. MHPO keeps GRPO’s group-relative advantages but replaces hard clipping with a Log-Fidelity Modulator and a Decoupled Hazard Penalty, arguing that importance-ratio control is the central source of instability in GRPO-based reasoning RL (Wang et al., 14 Mar 2026). StableDRL makes a related move for diffusion LLMs, where importance ratios are only proxy-estimated: it uses unconditional clipping and self-normalization to keep updates inside the convex hull of per-sample gradients and to break the self-reinforcing instability loop caused by noisy ratio estimates (Zhong et al., 6 Mar 2026).

A separate class of papers addresses instability in the advantage signal itself. AGPO handles the zero-variance corner case of GRPO in binary-reward reasoning tasks by assigning fixed $k$4 advantages to all-correct and all-wrong groups, thereby restoring signal when standard group normalization degenerates (Li et al., 20 Mar 2025). MDP-GRPO analyzes low-variance amplification, mean-centering blindness, and zero-variance collapse in multi-constraint instruction following, and combines multi-temperature sampling, dual-anchor advantages, prospect-theoretic shaping, and asymmetric KL regularization to stabilize discrete, low-dispersion reward regimes (Salmani-Zarchi et al., 4 Jun 2026). Compared with these methods, S-GRPO is distinctive in that it leaves the ratio control and the basic GRPO objective unchanged, and instead introduces a denoising weight derived from a reward-corruption model.

Stability can also be undermined by reward heterogeneity rather than noise. P-GRPO replaces immediate batch normalization with preference-group-specific historical normalization, arguing that standard GRPO’s exchangeability assumption destabilizes learning when batches mix users or contexts with different reward distributions (Wang et al., 17 Feb 2026). In a more theoretical direction, “Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statistic” shows that the GRPO gradient is a second-order U-statistic, derives MSE and suboptimality results, and gives the scaling law

$k$5

which frames stability through estimator covariance and group-size tradeoffs rather than only through heuristic design choices (Zhou et al., 1 Mar 2026).

This broader literature suggests that “stable GRPO” is not a single method family but a cluster of interventions aimed at different structural weaknesses: noisy outcome labels, rare-mode forgetting, hard clipping, heterogeneous reward distributions, low-dispersion group rewards, or proxy-estimated importance ratios. Within that landscape, S-GRPO is the version specialized to Think-Answer Mismatch and binary reward corruption. Its central idea is narrow but precise: when the group-relative signal itself becomes unreliable, scale the whole group update by an estimate of how much the observed standardized advantage can be trusted as a proxy for the latent clean one (Shen et al., 8 Aug 2025).