GRPO Done Right (Dr. GRPO)

• GRPO Done Right is a refined approach to group policy optimization that removes per-group standard-deviation normalization for raw success-rate optimization.
• It improves stability and efficiency in reinforcement learning by leveraging group-relative baselines and contrastive pairwise updates.
• The method features rigorous theoretical grounding with U-statistics analysis and optimal group-size scaling, informing various RL applications.

“GRPO Done Right” (“Dr. GRPO”) names a specific revision of Group Relative Policy Optimization and, in more expansive recent usage, a broader program for repairing GRPO’s normalization, credit-assignment, and stability pathologies in reasoning-heavy reinforcement learning. In the narrow sense formalized by “GRPO, Dr. GRPO, and DAPO Are Three Operations on One Number: The Group-Standard-Deviation Identity” (Bay et al., 30 Jun 2026), Dr. GRPO keeps GRPO’s group-mean baseline but removes division by the group reward standard deviation, thereby changing the effective objective from a variance-stabilized, difficulty-reweighted one to raw success-rate optimization. In the surrounding literature, the phrase is also used more loosely for principled GRPO design choices involving group size, token aggregation, contrastive structure, diversity, and self-correction (Zhou et al., 1 Mar 2026).

1. Baseline GRPO and the narrow definition of Dr. GRPO

GRPO is a critic-free, PPO-style policy-gradient method for LLM post-training. In the standard formulation, prompts $X \sim f(X)$ are sampled, a policy $\pi_\theta$ generates sequences $Y$, and a bounded terminal reward $Z$ is used to optimize

$J(\theta)=\mathbb{E}^{\pi_\theta}[Z].$

A batch-level GRPO update samples $B$ prompts and $G$ outputs per prompt, computes rewards $Z^{(b,g)}$, and forms a gradient estimate with a group-relative baseline rather than a learned value function (Zhou et al., 1 Mar 2026).

The canonical GRPO baseline for sample $(b,g)$ is the leave-one-out group mean

$$\bar{Z}^{(b,-g)}=\frac{1}{G-1}\sum_{k\neq g} Z^{(b,k)},$$

or equivalently the full group mean up to a learning-rate adjustment. In the binary-reward RLVR setting emphasized in later work, GRPO standardizes the centered reward within each prompt-specific group: $\pi_\theta$0 Dr. GRPO removes the division by $\pi_\theta$1 and uses only mean-centering,

$\pi_\theta$2

This makes it equivalent to a leave-one-out REINFORCE baseline up to a constant factor,

$\pi_\theta$3

so the distinctive change is not the baseline itself but the elimination of per-group standard-deviation normalization (Bay et al., 30 Jun 2026).

This narrower definition should be distinguished from later papers that discuss Dr. GRPO chiefly through token-aggregation choices or use “doing GRPO right” as a broader design label for improved GRPO practice (Wang et al., 8 Oct 2025).

2. The group-standard-deviation identity and what Dr. GRPO changes

For binary rewards $\pi_\theta$4, if a prompt’s group contains $\pi_\theta$5 correct answers out of $\pi_\theta$6, then

$\pi_\theta$7

The central result of (Bay et al., 30 Jun 2026) is that this $\pi_\theta$8 is not just a normalization constant: it is exactly the size of the GRPO gradient on that prompt. Writing $\pi_\theta$9, and $Y$0 and $Y$1 for the mean scores of correct and incorrect responses, respectively, the per-prompt GRPO update is

$Y$2

For Dr. GRPO, the corresponding update is

$Y$3

This yields three immediate consequences. First, unanimous groups are silent: if $Y$4 or $Y$5, then $Y$6 and the gradient is zero. Second, split groups are maximally informative: the update is largest when correct and incorrect samples are balanced. Third, GRPO, Dr. GRPO, and DAPO differ by how they act on the same scalar $Y$7: GRPO divides by it in the advantage, Dr. GRPO removes that division, and DAPO discards groups where it is zero (Bay et al., 30 Jun 2026).

At the objective level, the paper states that GRPO’s normalization induces an arcsine-transformed, difficulty-biased objective, whereas Dr. GRPO ascends raw success rate. The corresponding marginal difficulty weight for GRPO is

$Y$8

while Dr. GRPO applies flat weighting. On Big-Math, this changes where gradient mass goes: under the large-group analysis, Dr. GRPO places $Y$9 of gradient mass on extreme prompts with $Z$0 or $Z$1, whereas GRPO places $Z$2 there. For $Z$3, the closed-form silent-group fraction is $Z$4, closely matched by empirical subsampling at $Z$5 (Bay et al., 30 Jun 2026).

3. GRPO theory: U-statistics, oracle equivalence, and scaling laws

The modern theory of GRPO makes clear that the algorithm’s grouped structure is not incidental. “Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statistic” proves that, for a fixed prompt, the GRPO gradient estimator is a second-order U-statistic with symmetric kernel

$Z$6

Its Hoeffding decomposition shows that GRPO equals an oracle estimator using the exact value baseline $Z$7 plus a degenerate second-order residual. As $Z$8, the residual is $Z$9 while the first-order term is $J(\theta)=\mathbb{E}^{\pi_\theta}[Z].$0, so GRPO is asymptotically equivalent to the oracle estimator (Zhou et al., 1 Mar 2026).

The same analysis gives finite-sample MSE decompositions and a universal scaling law for group size. With minibatches of prompts and groups, the gradient MSE decomposes into a prompt-variance term $J(\theta)=\mathbb{E}^{\pi_\theta}[Z].$1, an oracle-like term $J(\theta)=\mathbb{E}^{\pi_\theta}[Z].$2, and a U-statistic residual $J(\theta)=\mathbb{E}^{\pi_\theta}[Z].$3. Under a fixed per-step sample budget $J(\theta)=\mathbb{E}^{\pi_\theta}[Z].$4, the suboptimality-controlling quantity becomes

$J(\theta)=\mathbb{E}^{\pi_\theta}[Z].$5

with optimal group size

$J(\theta)=\mathbb{E}^{\pi_\theta}[Z].$6

The paper emphasizes that this optimum is universal with respect to budget and iterations, but model- and task-dependent (Zhou et al., 1 Mar 2026).

This suggests that Dr. GRPO alters the objective weighting induced by standard-deviation normalization rather than abolishing GRPO’s core grouped, critic-free structure. The grouped-sampling logic, and much of the variance-reduction story, remain intact.

4. Contrastive reinterpretations and pairwise preference structure

A parallel line of work argues that GRPO is best understood as a contrastive method. “It Takes Two: Your GRPO Is Secretly DPO” shows that, in RLVR with binary rewards, GRPO’s gradient is a contrastive gradient over positives and negatives within each prompt group. In that view, the prompt is the anchor, correct trajectories are positives, incorrect ones are negatives, and the GRPO gradient directly increases average log-probability of positives while decreasing that of negatives. This analysis motivates 2-GRPO, and the paper reports that 2-GRPO is on par with 16-GRPO while using only $J(\theta)=\mathbb{E}^{\pi_\theta}[Z].$7 of the rollouts and reducing training time by over $J(\theta)=\mathbb{E}^{\pi_\theta}[Z].$8 (Wu et al., 1 Oct 2025).

Other papers exploit the same hidden pairwise structure more explicitly. AMIR-GRPO mines intra-group reward rankings to build an implicit DPO-style regularizer, thereby converting each rollout group into a denser set of contrastive constraints and amplifying suppression of low-reward trajectories (Yari et al., 7 Jan 2026). BiCC and RCC go further: Bilateral Context Conditioning feeds correct solutions the context of failed ones and vice versa, while Reward-Confidence Correction replaces the raw group-mean baseline with

$J(\theta)=\mathbb{E}^{\pi_\theta}[Z].$9

where $B$0 is a log-ratio confidence term, yielding a first-order approximation to the variance-minimizing baseline (Li et al., 13 Mar 2026).

Taken together, these results recast Dr. GRPO as one point in a broader movement away from treating the group merely as a source of a scalar baseline and toward treating it as a structured contrastive object.

5. Broader “doing GRPO right”: length bias, diversity, self-correction, and stability

Several later papers use “Dr. GRPO” or “GRPO done right” more broadly to denote principled fixes for known GRPO pathologies. One recurring theme is length bias. DRPO argues that naively adding length penalties to GRPO can assign negative advantages to correct but long rollouts because correct and incorrect samples are normalized together; it therefore decouples the length-based learning signal for correct rollouts from incorrect ones. On GSM8K with a 1.5B model, the paper reports $B$1 length reduction with only $B$2 performance loss, compared with a follow-up baseline that sacrifices $B$3 for $B$4 length reduction (Li et al., 6 Oct 2025).

A second theme is token aggregation. The $B$5-GRPO framework rewrites GRPO, DAPO, and Dr. GRPO as instances of a single objective with sequence-level weighting function $B$6, then replaces fixed heuristic weighting by a learnable scalar $B$7 that expresses a token preference over longer or shorter responses. On Qwen2.5 models, it improves average accuracy by $B$8, $B$9, and $G$0 over vanilla GRPO for 1.5B, 3B, and 7B scales, respectively (Wang et al., 8 Oct 2025).

A third theme is richer supervision without external critics. DRA-GRPO observes that scalar solution-level rewards do not track semantic diversity and adds Diversity-aware Reward Adjustment via graph-cut Submodular Mutual Information, reaching an average accuracy of $G$1 on five mathematical reasoning benchmarks using only 7,000 fine-tuning samples and a total training cost of approximately $G$2 (Chen et al., 14 May 2025). MGRPO introduces a second GRPO layer for self-correction; on MATH, GSM8K, Minerva Math, and OlympiadBench, its Acc.@t2 exceeds Acc.@t1′ by $G$3, $G$4, $G$5, and $G$6 points, respectively (Ding et al., 5 Jun 2025).

A fourth theme is estimator robustness. For diffusion LLMs, StableDRL argues that vanilla GRPO fails because importance ratios must be estimated, not computed exactly; noisy ratios can bypass conditional clipping and create a self-reinforcing instability loop. StableDRL therefore replaces standard GRPO’s conditional clipping with unconditional clipping and fixed group-size averaging with self-normalization, constraining updates to the convex hull of per-sample gradients (Zhong et al., 6 Mar 2026).

6. Scope, applications, and unresolved questions

The influence of GRPO and its “done right” revisions extends beyond mathematical reasoning. In radiology-report classification, a two-stage SFT-plus-GRPO pipeline improves micro-F1 in 8 of 9 model–dataset cohorts and increases reasoning recall and comprehensiveness without explicit reasoning supervision (Wei et al., 21 Apr 2026). GRPO-style ideas have also been adapted to representation models by defining a fixed discrete output set and group-relative advantages over class probabilities (Xu et al., 19 Nov 2025), and to test-time adaptation in vision-LLMs by treating top-$G$7 class prompts as the group and optimizing alignment and dispersion rewards with GRPO at inference time (Li et al., 5 May 2026).

At the same time, the literature suggests that the label “Dr. GRPO” is not fully uniform. Some papers reserve it for the precise removal of group-standard-deviation division (Bay et al., 30 Jun 2026), others analyze it mainly through token aggregation (Wang et al., 8 Oct 2025), and still others use “doing GRPO right” as a general design slogan for better group size, diversity handling, or reward shaping (Chen et al., 14 May 2025). A plausible implication is that Dr. GRPO is becoming both a technical term and a research agenda.

Two limitations remain especially salient. First, the group-standard-deviation identity is exact in the binary-reward, grouped RLVR setting; its clean form does not automatically extend to graded rewards or noisy reward models (Bay et al., 30 Jun 2026). Second, the best group size, normalization, and conditioning strategy remain task- and model-dependent even when a universal law exists within a fixed setting (Zhou et al., 1 Mar 2026). The ongoing disputes over length normalization, pairwise versus scalar supervision, confidence-aware baselines, and ratio estimation indicate that “GRPO done right” is less a settled recipe than a rapidly consolidating set of principles for making group-based policy optimization faithful to the actual structure of reasoning problems.