Group-Standard-Deviation Identity in RLVR

• Group-Standard-Deviation Identity is an exact formulation that relates the gradient update magnitude to the group reward standard deviation.
• It unifies GRPO, Dr. GRPO, and DAPO by showing that different normalization steps manipulate the same scalar, clarifying update dynamics.
• Empirical results confirm that prompts with mixed correct and incorrect samples yield strong learning signals, while unanimous groups contribute no gradient.

The Group-Standard-Deviation Identity is the exact finite-group statement that, in binary-reward RLVR training, the size of a GRPO update for one prompt is determined by the standard deviation of the group’s right/wrong rewards. In this setting, the same scalar that appears in the denominator of GRPO’s standardized advantage is also the actual magnitude of the learning signal. The identity is presented as a unifying result for Group Relative Policy Optimization (GRPO), GRPO Done Right (Dr. GRPO), and Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO), which are treated as three operations on one number: the group reward standard deviation (Bay et al., 30 Jun 2026).

1. Formal setting and the group reward standard deviation

The setting is one prompt $x$ during training. The policy $\pi_\theta$ samples a group of $G$ responses,

$y_1,\dots,y_G \sim \pi_\theta(\cdot\mid x),$

and a verifier assigns each sampled answer a binary reward,

$R_i \in \{0,1\},$

where $1$ means correct and $0$ means incorrect. Let

$k=\sum_{i=1}^G R_i$

be the number of correct samples in the group. The group mean reward and standard deviation are

$$\mu=\frac1G\sum_j R_j=\frac{k}{G},\qquad \sigma=\sqrt{\frac1G\sum_j (R_j-\mu)^2}.$$

Because rewards are Bernoulli,

$\sigma=\sqrt{\mu(1-\mu)}=\frac{\sqrt{k(G-k)}}{G}.$

GRPO uses the standardized advantage

$\pi_\theta$0

This advantage is broadcast to all tokens of response $\pi_\theta$1, and the policy update uses the usual score function term $\pi_\theta$2 (Bay et al., 30 Jun 2026).

The construction makes the prompt-local training signal depend only on within-group disagreement. When all sampled answers agree, $\pi_\theta$3; when the group splits between correct and incorrect answers, $\pi_\theta$4 is positive and quantifies the degree of disagreement. In the paper’s formulation, this is the scalar that organizes the entire prompt-level update geometry.

2. Exact finite-group identity

Define the score of response $\pi_\theta$5 as

$\pi_\theta$6

Let the mean score among correct samples and incorrect samples be

$\pi_\theta$7

Then the per-prompt GRPO update is exactly

$\pi_\theta$8

This is the Group-Standard-Deviation Identity. It states that the direction of the update is the contrast between correct and incorrect rollouts, $\pi_\theta$9, while the magnitude of the update is exactly the group’s reward standard deviation $G$0 (Bay et al., 30 Jun 2026).

The derivation proceeds by observing that, for binary rewards, there are only two possible standardized advantages: $G$1 Splitting the update across correct and incorrect samples gives

$G$2

which simplifies to

$G$3

Using

$G$4

one obtains

$G$5

Two further consequences are emphasized. First, in the scalar Bernoulli-logit special case,

$G$6

since $G$7 in that one-dimensional case. Second, if the group is unanimous, $G$8 or $G$9, then

$y_1,\dots,y_G \sim \pi_\theta(\cdot\mid x),$0

Thus unanimous groups contribute no gradient at all.

The identity is also baseline-free. If any reward baseline $y_1,\dots,y_G \sim \pi_\theta(\cdot\mid x),$1 is added, it shifts all advantages equally, but since the standardized advantages sum to zero, the shift cancels. The paper further notes that group-mean centering is just leave-one-out centering up to a constant: $y_1,\dots,y_G \sim \pi_\theta(\cdot\mid x),$2 This is used to argue that the truly objective-changing step is not subtracting the mean; it is dividing by $y_1,\dots,y_G \sim \pi_\theta(\cdot\mid x),$3.

3. Unification of GRPO, Dr. GRPO, and DAPO

The identity is used to unify three methods as three operations on the same scalar. GRPO uses

$y_1,\dots,y_G \sim \pi_\theta(\cdot\mid x),$4

so it divides by $y_1,\dots,y_G \sim \pi_\theta(\cdot\mid x),$5. Because the update magnitude becomes $y_1,\dots,y_G \sim \pi_\theta(\cdot\mid x),$6, GRPO effectively gives more weight to prompts whose sampled answers split between right and wrong. Dr. GRPO removes the division and instead uses

$y_1,\dots,y_G \sim \pi_\theta(\cdot\mid x),$7

In the paper’s table, this corresponds to

$y_1,\dots,y_G \sim \pi_\theta(\cdot\mid x),$8

so the update no longer has the same difficulty reweighting. DAPO’s dynamic sampling discards groups with

$y_1,\dots,y_G \sim \pi_\theta(\cdot\mid x),$9

that is, all-correct or all-wrong groups. For mixed groups, DAPO does not change the formula; it filters out the zero-variance groups before updating (Bay et al., 30 Jun 2026).

This yields a precise algorithmic taxonomy. GRPO divides by $R_i \in \{0,1\},$0, Dr. GRPO removes that division, and DAPO discards groups where $R_i \in \{0,1\},$1. The unifying claim is that the apparent methodological differences are not independent fixes but distinct manipulations of the same prompt-level standard-deviation term. The paper’s conceptual point is that what looks like a harmless normalization step is the dial that decides where learning happens and how strongly.

4. Learning dynamics, difficulty weighting, and group size

The identity immediately implies a structural account of where training signal comes from. Since

$R_i \in \{0,1\},$2

one has $R_i \in \{0,1\},$3 if $R_i \in \{0,1\},$4 or $R_i \in \{0,1\},$5, so unanimous groups produce no update, while $R_i \in \{0,1\},$6 is largest when $R_i \in \{0,1\},$7, so groups split evenly between right and wrong produce the strongest update. The paper interprets the most informative prompt as one where the model is uncertain enough to produce a mix of correct and incorrect samples. The training signal is therefore the within-group disagreement (Bay et al., 30 Jun 2026).

The same analysis yields a population-level difficulty-weighting description. In the large-group limit, the expected GRPO gradient becomes the arcsine/variance-stabilized form

$R_i \in \{0,1\},$8

where $R_i \in \{0,1\},$9 is the underlying success probability for the prompt. Equivalently, GRPO ascends

$1$0

whose derivative is

$1$1

This is the paper’s “difficulty bias”: GRPO places more marginal weight on very easy and very hard prompts, because those are where $1$2 is small and the normalization $1$3 is large. Dr. GRPO removes that bias by dropping the division and therefore aligns with the raw success-rate objective $1$4.

The group-size analysis is stated as a design rule. Defining fidelity as the fraction of the large-group gradient realized at finite $1$5, the asymptotic expansion

$1$6

gives the approximate requirement

$1$7

Mid-difficulty prompts with $1$8 need relatively few samples, whereas very easy or very hard prompts need many more samples. The paper also defines the silent-group probability,

$1$9

which is the fraction of groups with no right/wrong contrast.

5. Empirical validation

The identity is validated on the Big-Math corpus using $0$0 Big-Math problems with empirical solve rates from Llama-3.1-8B over 64 rollouts. The reported difficulty distribution is bimodal, with many very easy and very hard problems. Under GRPO’s standardization, more gradient mass shifts toward the extremes than under Dr. GRPO: extreme prompts move from $0$1 to $0$2 of total gradient mass, while medium prompts move from $0$3 to $0$4. Silent-group rates are also large at moderate $0$5; at $0$6, the paper reports about $0$7 silent groups, and the closed form

$0$8

matches subsampling closely (Bay et al., 30 Jun 2026).

A controlled training run is then reported with $0$9 Bernoulli-logit prompts, Big-Math-derived initial difficulties, $k=\sum_{i=1}^G R_i$0, 150 steps, and three methods: GRPO, Dr. GRPO, and DAPO. The measured silent-group fraction tracks the predicted $k=\sum_{i=1}^G R_i$1 with $k=\sum_{i=1}^G R_i$2. Realized gradient mass by difficulty matches the closed-form finite-$k=\sum_{i=1}^G R_i$3 prediction. GRPO lifts hard prompts more strongly than Dr. GRPO, while DAPO is fastest on hard prompts but does so by oversampling and discarding silent groups.

These results function as an empirical confirmation of the finite-group identity rather than as a replacement for it. The core statement remains exact at the per-prompt level; the experiments show that the induced weighting and silence phenomena persist in large corpora and controlled optimization settings.

6. Scope, interpretation, and non-equivalent usages

The identity is specific to binary verifier rewards in grouped RLVR training. The paper emphasizes that the analysis is about the advantage construction itself; clipping and KL regularization are standard but do not change the first-step identity being proved. The term “group” refers here to the sampled group of $k=\sum_{i=1}^G R_i$4 responses for one prompt, not to algebraic groups, grouped summary statistics, or generic dispersion identities (Bay et al., 30 Jun 2026).

This suggests a need for terminological separation from other standard-deviation results. In particular, “On the Order Between the Standard Deviation and Gini Mean Difference” does not establish an identity of the form “SD = GMD.” Instead, it develops a comparison or ordering theory, showing conditions under which one dominates the other; for the query “Group-Standard-Deviation Identity,” its own synthesized account states that the answer is inequalities and dominance relations, not an identity (Mohammed, 18 Jan 2026). A different usage appears in allometric scaling, where the exact theoretical identity is

$k=\sum_{i=1}^G R_i$5

with the empirical form

$k=\sum_{i=1}^G R_i$6

there the identity concerns scaling exponents as ratios of standard deviations of log-transformed measures, not grouped RLVR updates (Chen, 2020).

Within its own domain, however, the Group-Standard-Deviation Identity is exact and operational: for binary grouped rewards,

$k=\sum_{i=1}^G R_i$7

Its significance lies in converting what might appear to be a normalization choice into a structural statement about prompt-level learning. Split groups teach the most, unanimous groups teach nothing, and the standard deviation of binary reward disagreement is both the normalizer and the size of the update.