---
title: Guided Reward Policy Optimization (GRPO)
url: https://www.emergentmind.com/topics/guided-reward-policy-optimization-grpo
type: topic
---

# Guided Reward Policy Optimization (GRPO)

GRPO is a reinforcement learning method used to train advanced artificial intelligence models such as DeepSeek-R1-Zero and DeepSeekMath, and it was used successfully to train DeepSeek-R1 models for promoting reasoning capabilities of LLMs using verifiable or binary rewards [2502.18548][2503.06639]. In the cited literature, the acronym is predominantly expanded as **Group Relative Policy Optimization**, while one analytical note refers to **Guided Reward Policy Optimization** [2502.18548]. Its defining mechanism is group-relative normalization: for each prompt or state, multiple outputs are sampled, scored, standardized within the sampled group, and then optimized with a PPO-style surrogate together with a KL penalty to a reference policy, thereby removing the need for a learned critic or value model in the standard formulation [2509.01939][2606.06058].

## 1. Core algorithm and optimization structure

The standard GRPO workflow is organized around grouped rollouts. A batch of prompts or initial states is sampled; for each prompt, a group of \(G\) candidate outputs is generated; each output is scored by one or more reward functions; and the policy is updated to increase the probability of outputs with above-average reward relative to others in the same group [2509.22047][2509.01939]. In the single-reward setting, the central learning signal is the group-normalized advantage,
\[
A_i = \frac{r_i - \mu_{\text{group}}}{\sigma_{\text{group}} + \epsilon},
\]
or, equivalently in one formulation,
\[
\hat{A}_{i,t} = \frac{\mathbb{R}(i) - \mathbb{E}[\mathbb{R}(i)]}{\sigma(\mathbb{R}(i))},
\]
with the same normalized signal applied across the tokens of a completion [2509.01939][2606.06058].

The policy objective is PPO-like rather than value-based. One representative formulation writes
\[
\mathcal{L}_{\text{GRPO}}(\theta)
=
-\frac{1}{\sum_{i=1}^{G} |o_i|}
\sum_{i=1}^{G}\sum_{t=1}^{|o_i|}
\Big[
\min\bigl(w_{i,t}\hat{A}_{i,t}, \mathrm{clip}(w_{i,t},1-\epsilon,1+\epsilon)\hat{A}_{i,t}\bigr)
-\beta \mathcal{D}_{\mathrm{KL}}(\pi_\theta \| \pi_{\mathrm{ref}})
\Big],
\]
where \(w_{i,t}\) is the ratio between current and old policy probabilities and \(\beta\) controls regularization toward a reference policy [2508.13023]. This architecture explains why GRPO is routinely described as critic-free: advantage estimation is derived from within-group reward statistics rather than from a separately learned value function [2509.01939][2503.06639].

A recurrent design choice is how rewards are aggregated before normalization. In vanilla multi-objective GRPO, one common construction sums the rewards across objectives and then normalizes the resulting scalar total over the sampled group [2509.22047][2601.05242]. Much of the later GRPO literature consists of modifications to this normalization step, or to the sampling process that feeds it.

## 2. Alignment objective and theoretical characterizations

Analytical work on GRPO has emphasized that its alignment objective is not identical to the objective induced by standard RLHF. One note characterizes the stationary policies of GRPO and argues that its aggregation of preferences differs fundamentally from standard logarithmic pooling; the precise form of aggregation arises from the reward preference model and from a penalty that essentially corresponds to the reverse KL divergence between the aggregation policy and the reference policy [2502.18548]. The same analysis shows that for groups of size two, the reward preference model corresponds to pairwise comparison preferences, and it provides explicit characterizations for binary questions, for group size two, and in the limit of large group size [2502.18548].

For verifiable binary rewards, GRPO has also been rewritten as a KL-regularized contrastive loss in which the contrastive samples are synthetic data sampled from the old policy [2503.06639]. In that analysis, the optimal GRPO policy can be expressed explicitly in terms of the binary reward together with first- and second-order statistics of the old policy and the reference policy, and the success probability \(p_n\) follows a recurrence that converges to a fixed point \(p^*\) satisfying \(p^* > p_{\mathrm{ref}}\) under the stated conditions [2503.06639]. The paper therefore describes GRPO as inherently success-amplifying in the verifiable-reward regime [2503.06639].

A separate line of work argues that GRPO induces a non-trivial process reward model under assumptions regarding within-group overlap of token sequences across completions [2509.21154]. Under that view, the standard per-group GRPO loss can be rewritten as a PRM-aware objective over shared prefixes, and empirical measurements reported that almost all groups, about \(99.8\%\), had non-trivial process-set structure for group size 6, while none were trivial for group size 36 [2509.21154]. This provides a formal counterpoint to the common view that GRPO is purely outcome-level; the paper’s claim is instead that step-level reward structure is already latent in the grouped objective [2509.21154].

## 3. Failure modes, pathologies, and common points of contention

The most prominent criticism of vanilla GRPO concerns its behavior when reward scales, variances, or distributions are poorly matched to the normalization scheme. In multi-objective settings, GRPO was shown to be vulnerable to reward hacking: when reward functions differ in variance or scale, the composite advantage can become dominated by high-variance objectives, so that the agent optimizes one objective at the cost of others [2509.22047]. The concrete failure mode described in machine translation was maximizing a readability metric by never using the target language at all [2509.22047].

A related multi-reward critique targets the order of aggregation and normalization. When distinct rewards are first summed and only then group-normalized, many distinct reward combinations can collapse into identical advantage values, reducing the resolution of the training signal and sometimes leading to early training failure [2601.05242]. This is not a merely stylistic concern: the cited experiments report degraded convergence in tool calling, math reasoning, and coding reasoning when GRPO is applied naively in this summed-reward form [2601.05242].

Discrete, low-dispersion rewards create another class of pathologies. In multi-constraint instruction following, z-score group normalization was formalized as suffering from **low-variance amplification**, **mean-centering blindness**, and **zero-variance collapse** [2606.06058]. Low-variance amplification inflates tiny reward differences into large advantages; mean-centering blindness removes information about absolute reward level, so uniformly good and uniformly bad groups can look the same after normalization; and zero-variance collapse yields no learning signal when all rewards in a group are identical [2606.06058].

Further work argues that critic-free optimization does not, by itself, solve credit assignment. GRPO-VPS identifies indiscriminate credit assignment for intermediate steps and links it to overthinking, because every token in a trajectory receives the same trajectory-level signal [2604.20659]. EP-GRPO similarly identifies uniform token-level granularity, uniform polarity, and zero-variance collapse, and reports that in failed sequences over half of correct intermediate steps were penalized while some erroneous steps in correct sequences were rewarded [2605.04960]. In diffusion and flow-based generative models, an additional inefficiency appears as **reward clustering**, where many sampled trajectories collapse toward the group-mean reward and contribute little optimization value [2512.15347].

## 4. Multi-objective, heterogeneous-preference, and ordinal variants

The most direct response to multi-objective reward hacking is **MO-GRPO**, which computes a normalized advantage for each reward function individually and then sums those normalized advantages for the policy update [2509.22047]. The paper shows analytically that, under mild assumptions, every objective contributes equally to the policy update regardless of its variance, and it further proves affine invariance under positive affine transformations \(R_i' = a_i R_i + b_i\), \(a_i > 0\), a property it states is not true for vanilla GRPO [2509.22047]. This removes the need for manual reweighting of reward scales in the stated setting [2509.22047].

A closely related repair is **GDPO**, which decouples normalization across individual rewards before aggregation rather than normalizing only the summed reward. The paper’s criticism is that sum-then-normalize collapses distinct rollout reward combinations into identical advantages; its remedy is normalize-then-sum followed by batch-wise normalization, which is reported to produce substantially improved training stability and better results on tool calling, math reasoning, and coding reasoning [2601.05242]. The conceptual difference between MO-GRPO and GDPO is small enough that the two papers can be read as addressing the same structural weakness from slightly different formal angles.

Where rewards are heterogeneous across users rather than across objectives, **Personalized GRPO (P-GRPO)** replaces concurrent group statistics with preference-group-specific reward histories [2603.10009]. Instead of assuming that all samples in a group are exchangeable, P-GRPO maintains running mean and standard deviation for each preference group and normalizes a sample against those historical statistics, which the paper reports yields faster convergence and higher rewards than standard GRPO across diverse tasks [2603.10009]. This directly challenges the homogeneous-preference assumption built into standard within-group normalization.

When rewards are non-verifiable and only rankings are considered trustworthy, **GOPO** replaces standardized reward magnitudes with rank-based advantages [2602.03876]. Its stated motivation is that reward models are optimized to capture relative preferences, whereas GRPO relies on absolute reward magnitudes during policy optimization; GOPO therefore uses only the ranking of rewards and discards their magnitudes [2602.03876]. The reported consequences are consistently higher training and validation reward trajectories, improved LLM-as-judge evaluations across most intermediate training steps, and reaching comparable policy quality in substantially fewer training steps than GRPO [2602.03876].

In subjective alignment regimes such as social bias mitigation, **BiasGRPO** adapts GRPO to a high-variance reward landscape by using a custom bias reward model together with group-relative normalization [2606.04807]. The paper reports that BiasGRPO outperforms DPO and PPO across multiple benchmarks, and it gives a training-stability comparison in which average reward standard deviation was about \(0.14\) for PPO and about \(0.067\) for GRPO [2606.04807]. A plausible implication is that GRPO’s relative baseline is particularly attractive when there is no single ground truth but meaningful within-prompt comparisons remain possible.

## 5. Process-aware, guided, and token-level credit assignment

One major branch of GRPO research introduces explicit guidance into the rollout or the reward. **G\(^2\)RPO-A** injects ground-truth reasoning steps into a subset of rollouts and adaptively adjusts guidance length on the basis of recent reward signals [2508.13023]. The paper reports that naively adding guidance delivers limited gains, whereas the adaptive controller improves both mathematical reasoning and code generation relative to vanilla GRPO, especially for small language models that otherwise suffer from vanishing advantages [2508.13023].

Another branch makes the latent process structure explicit. **\(\lambda\)-GRPO** begins from the claim that standard GRPO is secretly a process reward model and identifies a flaw in the induced objective: the contribution of a process step is scaled by its process-set size, which can hinder both exploration and exploitation [2509.21154]. The proposed correction divides each token’s loss contribution by the size of its process set. In the reported experiments, all four \(\lambda\)-GRPO models reached higher maximum validation accuracy than their GRPO counterparts and did so in fewer training steps, with the paper summarizing this as a greater than \(10\%\) average increase in maximum validation accuracy and peak performance reached in less than half the number of training steps [2509.21154].

Teacher-guided process supervision appears in several forms. **PM4GRPO** augments answer and format rewards with a process-mining conformance reward computed as the F1 score of fitness and precision between student and teacher reasoning traces [2510.25065]. **GRPO-VPS** instead probes the model’s conditional probability of the correct answer at segment boundaries, using the increment in that probability as a verifiable process signal; it reports up to \(2.6\)-point accuracy improvements and \(13.7\%\) reasoning-length reductions on math tasks, and up to \(2.4\) points and \(4\%\) on general-domain tasks [2604.20659]. These methods share a common premise: trajectory-level correctness alone is too coarse for long-chain reasoning.

Self-supervised token-level credit assignment is the focus of entropy-based variants. **GTPO** and **GRPO-S** use dynamic entropy weighting to shape rewards at the token or sequence level, on the premise that high-entropy tokens in correct responses indicate critical decision points [2508.04349]. **EP-GRPO** combines entropy-gated modulation with implicit process signals derived from policy divergence and cumulative entropy mapping; on Qwen2.5-3B it reports average accuracy improving from \(18.1\%\) to \(22.9\%\) across five mathematical datasets, and on 7B from \(27.1\%\) to \(30.3\%\) [2605.04960]. Taken together, these papers argue that GRPO’s original uniform within-trajectory credit assignment is workable but not information-efficient for deep reasoning.

## 6. Variance engineering, sampling control, and computational efficiency

Because GRPO depends strongly on within-group dispersion, several works modify the reward distribution or the sampling process rather than the policy objective alone. **GRPOVI** proposes a reward adjustment model that provably increases reward variance while preserving reward expectation and relative preferences, solves the resulting nonconvex problem with an \(O(n \log n)\) algorithm, and integrates the method into GRPO to accelerate RLHF training [2505.23247]. The same paper offers an indirect explanation for the empirical effectiveness of GRPO with rule-based reward in systems such as DeepSeek-R1: rule-based reward schemes can inherently increase reward variance [2505.23247].

**MDP-GRPO** addresses discrete low-dispersion rewards with four mechanisms: multi-temperature sampling, dual-anchor advantages, prospect-theoretic shaping, and asymmetric KL regularization [2606.06058]. Its abstract reports improvement in strict constraint satisfaction by up to \(5.0\%\) on Llama-3.2-3B, while the detailed summary emphasizes stable convergence with small group sizes and preservation of general capabilities on MMLU and ARC [2606.06058]. **RC-GRPO** tackles the same low-variance problem in multi-turn tool calling by conditioning rollouts on discrete reward tokens such as `<|high_reward|>` and `<|low_reward|>` [2602.03025]. The full method, RCTP-FT plus RC-GRPO, is reported to reach \(85.00\%\) overall accuracy on BFCLv4 with Qwen2.5-7B-Instruct, exceeding all listed closed-source API baselines in that benchmark summary [2602.03025].

A distinct efficiency line studies over-sampling and pruning. **Pro-GRPO** begins from the observation that many trajectories are reward-clustered and therefore carry little optimization value; it introduces Optimal Variance Filtering and a dynamic **Expand-and-Prune** framework that prunes trajectories during sampling rather than after full generation [2512.15347]. The reported outcomes include \(1.26\times\) speedup for standard Pro-GRPO, \(1.41\times\) for Pro-GRPO-Flash, and total FLOPs reduced by up to \(41\%\) while improving task performance on diffusion-based and flow-based models [2512.15347].

Reward-model integration has also been reworked for scalability. **IRPO** replaces \(O(n^2)\) pairwise reward computations with a Bradley-Terry-inspired pointwise scoring scheme, enabling efficient evaluation of arbitrarily many candidates during RL training while preserving fine-grained reward signals [2601.00677]. **URPO** removes the separation between policy and reward models altogether by unifying instruction following and reward modeling in a single model and a single GRPO training phase; the same model acts as both player and referee [2507.17515]. In the reported Qwen2.5-7B experiments, URPO improved AlpacaEval from \(42.24\) to \(44.84\), the composite reasoning average from \(32.66\) to \(35.66\), and RewardBench from \(83.55\) to \(85.15\) relative to the stated baseline it replaces [2507.17515].

## 7. Application domains and empirical record

GRPO and its variants have been deployed across an unusually broad range of tasks. In speech recognition, GRPO with rule-based rewards produced up to \(18.4\%\) relative reduction in word error rate, together with reduction in hallucinations, increased robustness on out-of-domain datasets, and effectiveness in domain adaptation [2509.01939]. In multi-objective RL, MO-GRPO was evaluated on multi-armed bandits, simulated control in Mo-Gymnasium, WMT machine translation for En-Ja and En-Zh, and instruction following; the reported findings were stable learning, avoidance of reward hacking, and the highest human-judged translation quality among the compared GRPO-style methods in the translation setting [2509.22047].

Instruction following and tool use form another major deployment area. MDP-GRPO was evaluated on FollowBench, IFEval, and a curated multi-constraint dataset, where it outperformed standard GRPO in strict constraint satisfaction while maintaining general capabilities [2606.06058]. RC-GRPO targeted BFCLv4 multi-turn tool calling, where sparse trajectory-level rewards and low within-group variation make standard SFT-plus-GRPO brittle; the reward-conditioned approach was reported to improve performance consistently over baselines [2602.03025]. In personalized alignment, P-GRPO was evaluated on content recommendation, synthesis, and review generation, and the paper reports higher rewards and faster convergence without loss of general capability on MMLU [2603.10009].

Reasoning remains the most visible arena for GRPO research. The method has been used in mathematical reasoning, code generation, and verifiable-reward post-training more broadly, and a large fraction of later GRPO variants are expressly designed to improve reasoning credit assignment or exploration [2503.06639][2508.13023]. PM4GRPO, GRPO-VPS, EP-GRPO, GTPO, and GRPO-S all report gains on mathematical benchmarks such as MATH500, OlympiadBench, Minerva, AIME24, and AIME25, though they do so through markedly different mechanisms: teacher conformance, segment-wise answer-belief probing, entropy-progress alignment, and token-level entropy weighting, respectively [2510.25065][2604.20659][2605.04960][2508.04349].

The scope of GRPO has also extended beyond text-only language modeling. Pro-GRPO reports improvements for diffusion-based and flow-based text-to-image generation on rewards such as HPS-v2.1, CLIP, and PickScore, while BiasGRPO adapts the method to subjective social-bias mitigation across BOLD, RealToxicityPrompts, BBQ, and TruthfulQA [2512.15347][2606.04807]. This suggests that GRPO is best understood not as a single fixed recipe, but as a family of group-normalized policy optimization methods whose practical behavior depends crucially on how rewards are scaled, decomposed, guided, and sampled.

Source: https://www.emergentmind.com/topics/guided-reward-policy-optimization-grpo