Guided Reward Policy Optimization (GRPO)

• GRPO is a reinforcement learning method that leverages group-relative normalization by sampling multiple outputs and standardizing rewards within each group.
• It uses a PPO-like objective with KL penalties to remove the need for a learned critic, thereby simplifying policy optimization using within-group reward statistics.
• GRPO and its variants address challenges such as reward scaling, credit assignment, and efficiency, enhancing reasoning and performance across diverse AI applications.

GRPO is a reinforcement learning method used to train advanced artificial intelligence models such as DeepSeek-R1-Zero and DeepSeekMath, and it was used successfully to train DeepSeek-R1 models for promoting reasoning capabilities of LLMs using verifiable or binary rewards (Vojnovic et al., 25 Feb 2025, Mroueh, 9 Mar 2025). In the cited literature, the acronym is predominantly expanded as Group Relative Policy Optimization, while one analytical note refers to Guided Reward Policy Optimization (Vojnovic et al., 25 Feb 2025). Its defining mechanism is group-relative normalization: for each prompt or state, multiple outputs are sampled, scored, standardized within the sampled group, and then optimized with a PPO-style surrogate together with a KL penalty to a reference policy, thereby removing the need for a learned critic or value model in the standard formulation (Shivakumar et al., 2 Sep 2025, Salmani-Zarchi et al., 4 Jun 2026).

1. Core algorithm and optimization structure

The standard GRPO workflow is organized around grouped rollouts. A batch of prompts or initial states is sampled; for each prompt, a group of $G$ candidate outputs is generated; each output is scored by one or more reward functions; and the policy is updated to increase the probability of outputs with above-average reward relative to others in the same group (Ichihara et al., 26 Sep 2025, Shivakumar et al., 2 Sep 2025). In the single-reward setting, the central learning signal is the group-normalized advantage,

$$A_i = \frac{r_i - \mu_{\text{group}}}{\sigma_{\text{group}} + \epsilon},$$

or, equivalently in one formulation,

$$\hat{A}_{i,t} = \frac{\mathbb{R}(i) - \mathbb{E}[\mathbb{R}(i)]}{\sigma(\mathbb{R}(i))},$$

with the same normalized signal applied across the tokens of a completion (Shivakumar et al., 2 Sep 2025, Salmani-Zarchi et al., 4 Jun 2026).

The policy objective is PPO-like rather than value-based. One representative formulation writes

$$\mathcal{L}_{\text{GRPO}}(\theta) = -\frac{1}{\sum_{i=1}^{G} |o_i|} \sum_{i=1}^{G}\sum_{t=1}^{|o_i|} \Big[ \min\bigl(w_{i,t}\hat{A}_{i,t}, \mathrm{clip}(w_{i,t},1-\epsilon,1+\epsilon)\hat{A}_{i,t}\bigr) -\beta \mathcal{D}_{\mathrm{KL}}(\pi_\theta \| \pi_{\mathrm{ref}}) \Big],$$

where $w_{i,t}$ is the ratio between current and old policy probabilities and $\beta$ controls regularization toward a reference policy (Guo et al., 18 Aug 2025). This architecture explains why GRPO is routinely described as critic-free: advantage estimation is derived from within-group reward statistics rather than from a separately learned value function (Shivakumar et al., 2 Sep 2025, Mroueh, 9 Mar 2025).

A recurrent design choice is how rewards are aggregated before normalization. In vanilla multi-objective GRPO, one common construction sums the rewards across objectives and then normalizes the resulting scalar total over the sampled group (Ichihara et al., 26 Sep 2025, Liu et al., 8 Jan 2026). Much of the later GRPO literature consists of modifications to this normalization step, or to the sampling process that feeds it.

2. Alignment objective and theoretical characterizations

Analytical work on GRPO has emphasized that its alignment objective is not identical to the objective induced by standard RLHF. One note characterizes the stationary policies of GRPO and argues that its aggregation of preferences differs fundamentally from standard logarithmic pooling; the precise form of aggregation arises from the reward preference model and from a penalty that essentially corresponds to the reverse KL divergence between the aggregation policy and the reference policy (Vojnovic et al., 25 Feb 2025). The same analysis shows that for groups of size two, the reward preference model corresponds to pairwise comparison preferences, and it provides explicit characterizations for binary questions, for group size two, and in the limit of large group size (Vojnovic et al., 25 Feb 2025).

For verifiable binary rewards, GRPO has also been rewritten as a KL-regularized contrastive loss in which the contrastive samples are synthetic data sampled from the old policy (Mroueh, 9 Mar 2025). In that analysis, the optimal GRPO policy can be expressed explicitly in terms of the binary reward together with first- and second-order statistics of the old policy and the reference policy, and the success probability $p_n$ follows a recurrence that converges to a fixed point $p^*$ satisfying $p^* > p_{\mathrm{ref}}$ under the stated conditions (Mroueh, 9 Mar 2025). The paper therefore describes GRPO as inherently success-amplifying in the verifiable-reward regime (Mroueh, 9 Mar 2025).

A separate line of work argues that GRPO induces a non-trivial process reward model under assumptions regarding within-group overlap of token sequences across completions (Sullivan, 25 Sep 2025). Under that view, the standard per-group GRPO loss can be rewritten as a PRM-aware objective over shared prefixes, and empirical measurements reported that almost all groups, about $99.8\%$, had non-trivial process-set structure for group size 6, while none were trivial for group size 36 (Sullivan, 25 Sep 2025). This provides a formal counterpoint to the common view that GRPO is purely outcome-level; the paper’s claim is instead that step-level reward structure is already latent in the grouped objective (Sullivan, 25 Sep 2025).

3. Failure modes, pathologies, and common points of contention

The most prominent criticism of vanilla GRPO concerns its behavior when reward scales, variances, or distributions are poorly matched to the normalization scheme. In multi-objective settings, GRPO was shown to be vulnerable to reward hacking: when reward functions differ in variance or scale, the composite advantage can become dominated by high-variance objectives, so that the agent optimizes one objective at the cost of others (Ichihara et al., 26 Sep 2025). The concrete failure mode described in machine translation was maximizing a readability metric by never using the target language at all (Ichihara et al., 26 Sep 2025).

A related multi-reward critique targets the order of aggregation and normalization. When distinct rewards are first summed and only then group-normalized, many distinct reward combinations can collapse into identical advantage values, reducing the resolution of the training signal and sometimes leading to early training failure (Liu et al., 8 Jan 2026). This is not a merely stylistic concern: the cited experiments report degraded convergence in tool calling, math reasoning, and coding reasoning when GRPO is applied naively in this summed-reward form (Liu et al., 8 Jan 2026).

Discrete, low-dispersion rewards create another class of pathologies. In multi-constraint instruction following, z-score group normalization was formalized as suffering from low-variance amplification, mean-centering blindness, and zero-variance collapse (Salmani-Zarchi et al., 4 Jun 2026). Low-variance amplification inflates tiny reward differences into large advantages; mean-centering blindness removes information about absolute reward level, so uniformly good and uniformly bad groups can look the same after normalization; and zero-variance collapse yields no learning signal when all rewards in a group are identical (Salmani-Zarchi et al., 4 Jun 2026).

Further work argues that critic-free optimization does not, by itself, solve credit assignment. GRPO-VPS identifies indiscriminate credit assignment for intermediate steps and links it to overthinking, because every token in a trajectory receives the same trajectory-level signal (Wang et al., 22 Apr 2026). EP-GRPO similarly identifies uniform token-level granularity, uniform polarity, and zero-variance collapse, and reports that in failed sequences over half of correct intermediate steps were penalized while some erroneous steps in correct sequences were rewarded (Yu et al., 6 May 2026). In diffusion and flow-based generative models, an additional inefficiency appears as reward clustering, where many sampled trajectories collapse toward the group-mean reward and contribute little optimization value (Ge et al., 17 Dec 2025).

4. Multi-objective, heterogeneous-preference, and ordinal variants

The most direct response to multi-objective reward hacking is MO-GRPO, which computes a normalized advantage for each reward function individually and then sums those normalized advantages for the policy update (Ichihara et al., 26 Sep 2025). The paper shows analytically that, under mild assumptions, every objective contributes equally to the policy update regardless of its variance, and it further proves affine invariance under positive affine transformations $$A_i = \frac{r_i - \mu_{\text{group}}}{\sigma_{\text{group}} + \epsilon},$$0, $$A_i = \frac{r_i - \mu_{\text{group}}}{\sigma_{\text{group}} + \epsilon},$$1, a property it states is not true for vanilla GRPO (Ichihara et al., 26 Sep 2025). This removes the need for manual reweighting of reward scales in the stated setting (Ichihara et al., 26 Sep 2025).

A closely related repair is GDPO, which decouples normalization across individual rewards before aggregation rather than normalizing only the summed reward. The paper’s criticism is that sum-then-normalize collapses distinct rollout reward combinations into identical advantages; its remedy is normalize-then-sum followed by batch-wise normalization, which is reported to produce substantially improved training stability and better results on tool calling, math reasoning, and coding reasoning (Liu et al., 8 Jan 2026). The conceptual difference between MO-GRPO and GDPO is small enough that the two papers can be read as addressing the same structural weakness from slightly different formal angles.

Where rewards are heterogeneous across users rather than across objectives, Personalized GRPO (P-GRPO) replaces concurrent group statistics with preference-group-specific reward histories (Wang et al., 17 Feb 2026). Instead of assuming that all samples in a group are exchangeable, P-GRPO maintains running mean and standard deviation for each preference group and normalizes a sample against those historical statistics, which the paper reports yields faster convergence and higher rewards than standard GRPO across diverse tasks (Wang et al., 17 Feb 2026). This directly challenges the homogeneous-preference assumption built into standard within-group normalization.

When rewards are non-verifiable and only rankings are considered trustworthy, GOPO replaces standardized reward magnitudes with rank-based advantages (Choi et al., 1 Feb 2026). Its stated motivation is that reward models are optimized to capture relative preferences, whereas GRPO relies on absolute reward magnitudes during policy optimization; GOPO therefore uses only the ranking of rewards and discards their magnitudes (Choi et al., 1 Feb 2026). The reported consequences are consistently higher training and validation reward trajectories, improved LLM-as-judge evaluations across most intermediate training steps, and reaching comparable policy quality in substantially fewer training steps than GRPO (Choi et al., 1 Feb 2026).

In subjective alignment regimes such as social bias mitigation, BiasGRPO adapts GRPO to a high-variance reward landscape by using a custom bias reward model together with group-relative normalization (Reddy et al., 3 Jun 2026). The paper reports that BiasGRPO outperforms DPO and PPO across multiple benchmarks, and it gives a training-stability comparison in which average reward standard deviation was about $$A_i = \frac{r_i - \mu_{\text{group}}}{\sigma_{\text{group}} + \epsilon},$$2 for PPO and about $$A_i = \frac{r_i - \mu_{\text{group}}}{\sigma_{\text{group}} + \epsilon},$$3 for GRPO (Reddy et al., 3 Jun 2026). A plausible implication is that GRPO’s relative baseline is particularly attractive when there is no single ground truth but meaningful within-prompt comparisons remain possible.

5. Process-aware, guided, and token-level credit assignment

One major branch of GRPO research introduces explicit guidance into the rollout or the reward. G$$A_i = \frac{r_i - \mu_{\text{group}}}{\sigma_{\text{group}} + \epsilon},$$4RPO-A injects ground-truth reasoning steps into a subset of rollouts and adaptively adjusts guidance length on the basis of recent reward signals (Guo et al., 18 Aug 2025). The paper reports that naively adding guidance delivers limited gains, whereas the adaptive controller improves both mathematical reasoning and code generation relative to vanilla GRPO, especially for small LLMs that otherwise suffer from vanishing advantages (Guo et al., 18 Aug 2025).

Another branch makes the latent process structure explicit. $$A_i = \frac{r_i - \mu_{\text{group}}}{\sigma_{\text{group}} + \epsilon},$$5-GRPO begins from the claim that standard GRPO is secretly a process reward model and identifies a flaw in the induced objective: the contribution of a process step is scaled by its process-set size, which can hinder both exploration and exploitation (Sullivan, 25 Sep 2025). The proposed correction divides each token’s loss contribution by the size of its process set. In the reported experiments, all four $$A_i = \frac{r_i - \mu_{\text{group}}}{\sigma_{\text{group}} + \epsilon},$$6-GRPO models reached higher maximum validation accuracy than their GRPO counterparts and did so in fewer training steps, with the paper summarizing this as a greater than $$A_i = \frac{r_i - \mu_{\text{group}}}{\sigma_{\text{group}} + \epsilon},$$7 average increase in maximum validation accuracy and peak performance reached in less than half the number of training steps (Sullivan, 25 Sep 2025).

Teacher-guided process supervision appears in several forms. PM4GRPO augments answer and format rewards with a process-mining conformance reward computed as the F1 score of fitness and precision between student and teacher reasoning traces (Park et al., 29 Oct 2025). GRPO-VPS instead probes the model’s conditional probability of the correct answer at segment boundaries, using the increment in that probability as a verifiable process signal; it reports up to $$A_i = \frac{r_i - \mu_{\text{group}}}{\sigma_{\text{group}} + \epsilon},$$8-point accuracy improvements and $$A_i = \frac{r_i - \mu_{\text{group}}}{\sigma_{\text{group}} + \epsilon},$$9 reasoning-length reductions on math tasks, and up to $$\hat{A}_{i,t} = \frac{\mathbb{R}(i) - \mathbb{E}[\mathbb{R}(i)]}{\sigma(\mathbb{R}(i))},$$0 points and $$\hat{A}_{i,t} = \frac{\mathbb{R}(i) - \mathbb{E}[\mathbb{R}(i)]}{\sigma(\mathbb{R}(i))},$$1 on general-domain tasks (Wang et al., 22 Apr 2026). These methods share a common premise: trajectory-level correctness alone is too coarse for long-chain reasoning.

Self-supervised token-level credit assignment is the focus of entropy-based variants. GTPO and GRPO-S use dynamic entropy weighting to shape rewards at the token or sequence level, on the premise that high-entropy tokens in correct responses indicate critical decision points (Tan et al., 6 Aug 2025). EP-GRPO combines entropy-gated modulation with implicit process signals derived from policy divergence and cumulative entropy mapping; on Qwen2.5-3B it reports average accuracy improving from $$\hat{A}_{i,t} = \frac{\mathbb{R}(i) - \mathbb{E}[\mathbb{R}(i)]}{\sigma(\mathbb{R}(i))},$$2 to $$\hat{A}_{i,t} = \frac{\mathbb{R}(i) - \mathbb{E}[\mathbb{R}(i)]}{\sigma(\mathbb{R}(i))},$$3 across five mathematical datasets, and on 7B from $$\hat{A}_{i,t} = \frac{\mathbb{R}(i) - \mathbb{E}[\mathbb{R}(i)]}{\sigma(\mathbb{R}(i))},$$4 to $$\hat{A}_{i,t} = \frac{\mathbb{R}(i) - \mathbb{E}[\mathbb{R}(i)]}{\sigma(\mathbb{R}(i))},$$5 (Yu et al., 6 May 2026). Taken together, these papers argue that GRPO’s original uniform within-trajectory credit assignment is workable but not information-efficient for deep reasoning.

6. Variance engineering, sampling control, and computational efficiency

Because GRPO depends strongly on within-group dispersion, several works modify the reward distribution or the sampling process rather than the policy objective alone. GRPOVI proposes a reward adjustment model that provably increases reward variance while preserving reward expectation and relative preferences, solves the resulting nonconvex problem with an $$\hat{A}_{i,t} = \frac{\mathbb{R}(i) - \mathbb{E}[\mathbb{R}(i)]}{\sigma(\mathbb{R}(i))},$$6 algorithm, and integrates the method into GRPO to accelerate RLHF training (Yang et al., 29 May 2025). The same paper offers an indirect explanation for the empirical effectiveness of GRPO with rule-based reward in systems such as DeepSeek-R1: rule-based reward schemes can inherently increase reward variance (Yang et al., 29 May 2025).

MDP-GRPO addresses discrete low-dispersion rewards with four mechanisms: multi-temperature sampling, dual-anchor advantages, prospect-theoretic shaping, and asymmetric KL regularization (Salmani-Zarchi et al., 4 Jun 2026). Its abstract reports improvement in strict constraint satisfaction by up to $$\hat{A}_{i,t} = \frac{\mathbb{R}(i) - \mathbb{E}[\mathbb{R}(i)]}{\sigma(\mathbb{R}(i))},$$7 on Llama-3.2-3B, while the detailed summary emphasizes stable convergence with small group sizes and preservation of general capabilities on MMLU and ARC (Salmani-Zarchi et al., 4 Jun 2026). RC-GRPO tackles the same low-variance problem in multi-turn tool calling by conditioning rollouts on discrete reward tokens such as <|high_reward|> and <|low_reward|> (Zhong et al., 3 Feb 2026). The full method, RCTP-FT plus RC-GRPO, is reported to reach $$\hat{A}_{i,t} = \frac{\mathbb{R}(i) - \mathbb{E}[\mathbb{R}(i)]}{\sigma(\mathbb{R}(i))},$$8 overall accuracy on BFCLv4 with Qwen2.5-7B-Instruct, exceeding all listed closed-source API baselines in that benchmark summary (Zhong et al., 3 Feb 2026).

A distinct efficiency line studies over-sampling and pruning. Pro-GRPO begins from the observation that many trajectories are reward-clustered and therefore carry little optimization value; it introduces Optimal Variance Filtering and a dynamic Expand-and-Prune framework that prunes trajectories during sampling rather than after full generation (Ge et al., 17 Dec 2025). The reported outcomes include $$\hat{A}_{i,t} = \frac{\mathbb{R}(i) - \mathbb{E}[\mathbb{R}(i)]}{\sigma(\mathbb{R}(i))},$$9 speedup for standard Pro-GRPO, $$\mathcal{L}_{\text{GRPO}}(\theta) = -\frac{1}{\sum_{i=1}^{G} |o_i|} \sum_{i=1}^{G}\sum_{t=1}^{|o_i|} \Big[ \min\bigl(w_{i,t}\hat{A}_{i,t}, \mathrm{clip}(w_{i,t},1-\epsilon,1+\epsilon)\hat{A}_{i,t}\bigr) -\beta \mathcal{D}_{\mathrm{KL}}(\pi_\theta \| \pi_{\mathrm{ref}}) \Big],$$0 for Pro-GRPO-Flash, and total FLOPs reduced by up to $$\mathcal{L}_{\text{GRPO}}(\theta) = -\frac{1}{\sum_{i=1}^{G} |o_i|} \sum_{i=1}^{G}\sum_{t=1}^{|o_i|} \Big[ \min\bigl(w_{i,t}\hat{A}_{i,t}, \mathrm{clip}(w_{i,t},1-\epsilon,1+\epsilon)\hat{A}_{i,t}\bigr) -\beta \mathcal{D}_{\mathrm{KL}}(\pi_\theta \| \pi_{\mathrm{ref}}) \Big],$$1 while improving task performance on diffusion-based and flow-based models (Ge et al., 17 Dec 2025).

Reward-model integration has also been reworked for scalability. IRPO replaces $$\mathcal{L}_{\text{GRPO}}(\theta) = -\frac{1}{\sum_{i=1}^{G} |o_i|} \sum_{i=1}^{G}\sum_{t=1}^{|o_i|} \Big[ \min\bigl(w_{i,t}\hat{A}_{i,t}, \mathrm{clip}(w_{i,t},1-\epsilon,1+\epsilon)\hat{A}_{i,t}\bigr) -\beta \mathcal{D}_{\mathrm{KL}}(\pi_\theta \| \pi_{\mathrm{ref}}) \Big],$$2 pairwise reward computations with a Bradley-Terry-inspired pointwise scoring scheme, enabling efficient evaluation of arbitrarily many candidates during RL training while preserving fine-grained reward signals (Song et al., 2 Jan 2026). URPO removes the separation between policy and reward models altogether by unifying instruction following and reward modeling in a single model and a single GRPO training phase; the same model acts as both player and referee (Lu et al., 23 Jul 2025). In the reported Qwen2.5-7B experiments, URPO improved AlpacaEval from $$\mathcal{L}_{\text{GRPO}}(\theta) = -\frac{1}{\sum_{i=1}^{G} |o_i|} \sum_{i=1}^{G}\sum_{t=1}^{|o_i|} \Big[ \min\bigl(w_{i,t}\hat{A}_{i,t}, \mathrm{clip}(w_{i,t},1-\epsilon,1+\epsilon)\hat{A}_{i,t}\bigr) -\beta \mathcal{D}_{\mathrm{KL}}(\pi_\theta \| \pi_{\mathrm{ref}}) \Big],$$3 to $$\mathcal{L}_{\text{GRPO}}(\theta) = -\frac{1}{\sum_{i=1}^{G} |o_i|} \sum_{i=1}^{G}\sum_{t=1}^{|o_i|} \Big[ \min\bigl(w_{i,t}\hat{A}_{i,t}, \mathrm{clip}(w_{i,t},1-\epsilon,1+\epsilon)\hat{A}_{i,t}\bigr) -\beta \mathcal{D}_{\mathrm{KL}}(\pi_\theta \| \pi_{\mathrm{ref}}) \Big],$$4, the composite reasoning average from $$\mathcal{L}_{\text{GRPO}}(\theta) = -\frac{1}{\sum_{i=1}^{G} |o_i|} \sum_{i=1}^{G}\sum_{t=1}^{|o_i|} \Big[ \min\bigl(w_{i,t}\hat{A}_{i,t}, \mathrm{clip}(w_{i,t},1-\epsilon,1+\epsilon)\hat{A}_{i,t}\bigr) -\beta \mathcal{D}_{\mathrm{KL}}(\pi_\theta \| \pi_{\mathrm{ref}}) \Big],$$5 to $$\mathcal{L}_{\text{GRPO}}(\theta) = -\frac{1}{\sum_{i=1}^{G} |o_i|} \sum_{i=1}^{G}\sum_{t=1}^{|o_i|} \Big[ \min\bigl(w_{i,t}\hat{A}_{i,t}, \mathrm{clip}(w_{i,t},1-\epsilon,1+\epsilon)\hat{A}_{i,t}\bigr) -\beta \mathcal{D}_{\mathrm{KL}}(\pi_\theta \| \pi_{\mathrm{ref}}) \Big],$$6, and RewardBench from $$\mathcal{L}_{\text{GRPO}}(\theta) = -\frac{1}{\sum_{i=1}^{G} |o_i|} \sum_{i=1}^{G}\sum_{t=1}^{|o_i|} \Big[ \min\bigl(w_{i,t}\hat{A}_{i,t}, \mathrm{clip}(w_{i,t},1-\epsilon,1+\epsilon)\hat{A}_{i,t}\bigr) -\beta \mathcal{D}_{\mathrm{KL}}(\pi_\theta \| \pi_{\mathrm{ref}}) \Big],$$7 to $$\mathcal{L}_{\text{GRPO}}(\theta) = -\frac{1}{\sum_{i=1}^{G} |o_i|} \sum_{i=1}^{G}\sum_{t=1}^{|o_i|} \Big[ \min\bigl(w_{i,t}\hat{A}_{i,t}, \mathrm{clip}(w_{i,t},1-\epsilon,1+\epsilon)\hat{A}_{i,t}\bigr) -\beta \mathcal{D}_{\mathrm{KL}}(\pi_\theta \| \pi_{\mathrm{ref}}) \Big],$$8 relative to the stated baseline it replaces (Lu et al., 23 Jul 2025).

7. Application domains and empirical record

GRPO and its variants have been deployed across an unusually broad range of tasks. In speech recognition, GRPO with rule-based rewards produced up to $$\mathcal{L}_{\text{GRPO}}(\theta) = -\frac{1}{\sum_{i=1}^{G} |o_i|} \sum_{i=1}^{G}\sum_{t=1}^{|o_i|} \Big[ \min\bigl(w_{i,t}\hat{A}_{i,t}, \mathrm{clip}(w_{i,t},1-\epsilon,1+\epsilon)\hat{A}_{i,t}\bigr) -\beta \mathcal{D}_{\mathrm{KL}}(\pi_\theta \| \pi_{\mathrm{ref}}) \Big],$$9 relative reduction in word error rate, together with reduction in hallucinations, increased robustness on out-of-domain datasets, and effectiveness in domain adaptation (Shivakumar et al., 2 Sep 2025). In multi-objective RL, MO-GRPO was evaluated on multi-armed bandits, simulated control in Mo-Gymnasium, WMT machine translation for En-Ja and En-Zh, and instruction following; the reported findings were stable learning, avoidance of reward hacking, and the highest human-judged translation quality among the compared GRPO-style methods in the translation setting (Ichihara et al., 26 Sep 2025).

Instruction following and tool use form another major deployment area. MDP-GRPO was evaluated on FollowBench, IFEval, and a curated multi-constraint dataset, where it outperformed standard GRPO in strict constraint satisfaction while maintaining general capabilities (Salmani-Zarchi et al., 4 Jun 2026). RC-GRPO targeted BFCLv4 multi-turn tool calling, where sparse trajectory-level rewards and low within-group variation make standard SFT-plus-GRPO brittle; the reward-conditioned approach was reported to improve performance consistently over baselines (Zhong et al., 3 Feb 2026). In personalized alignment, P-GRPO was evaluated on content recommendation, synthesis, and review generation, and the paper reports higher rewards and faster convergence without loss of general capability on MMLU (Wang et al., 17 Feb 2026).

Reasoning remains the most visible arena for GRPO research. The method has been used in mathematical reasoning, code generation, and verifiable-reward post-training more broadly, and a large fraction of later GRPO variants are expressly designed to improve reasoning credit assignment or exploration (Mroueh, 9 Mar 2025, Guo et al., 18 Aug 2025). PM4GRPO, GRPO-VPS, EP-GRPO, GTPO, and GRPO-S all report gains on mathematical benchmarks such as MATH500, OlympiadBench, Minerva, AIME24, and AIME25, though they do so through markedly different mechanisms: teacher conformance, segment-wise answer-belief probing, entropy-progress alignment, and token-level entropy weighting, respectively (Park et al., 29 Oct 2025, Wang et al., 22 Apr 2026, Yu et al., 6 May 2026, Tan et al., 6 Aug 2025).

The scope of GRPO has also extended beyond text-only language modeling. Pro-GRPO reports improvements for diffusion-based and flow-based text-to-image generation on rewards such as HPS-v2.1, CLIP, and PickScore, while BiasGRPO adapts the method to subjective social-bias mitigation across BOLD, RealToxicityPrompts, BBQ, and TruthfulQA (Ge et al., 17 Dec 2025, Reddy et al., 3 Jun 2026). This suggests that GRPO is best understood not as a single fixed recipe, but as a family of group-normalized policy optimization methods whose practical behavior depends crucially on how rewards are scaled, decomposed, guided, and sampled.