---
title: Grouped Regularized Policy Optimization (GRPO)
url: https://www.emergentmind.com/topics/grouped-regularized-policy-optimization-grpo
type: topic
---

# Grouped Regularized Policy Optimization (GRPO)

Grouped Regularized Policy Optimization (GRPO) is a reinforcement learning (RL) algorithm for critic-free, group-based policy optimization, originally introduced to advance reasoning capabilities in large language models (LLMs) trained with verifiable or binary rewards. The central idea is to normalize advantages within a group of parallel rollouts for each prompt, stabilizing updates without requiring a learned value function. GRPO forms a flexible foundation for subsequent algorithmic innovations—including Focal GRPO (F-GRPO)—and extensions to other domains such as speech recognition, multi-agent control, and neural combinatorial optimization, due to its baseline-free, variance-reducing design [2602.06717].

## 1. Core GRPO Algorithm: Objective and Advantage Normalization

GRPO operates by generating, for each input $x$, a group of $N$ trajectories $\{o_i\}_{i=1}^N$ using the current or previous policy $\pi_\theta$. Each trajectory is assigned a verifiable (often binary) reward:
\[
R_i = R_c \cdot 1[o_i\;\text{correct}] + R_w \cdot 1[o_i\;\text{incorrect}]
\]
with $R_c > R_w$ (e.g., $R_c=1$, $R_w=0$ or $-1$). For this group, the mean $\bar R$ and standard deviation $\sigma_R$ of the rewards are computed:
\[
\bar R = \frac{1}{N}\sum_{j=1}^N R_j,\qquad
\sigma_R = \sqrt{\frac{1}{N}\sum_{j=1}^N (R_j - \bar R)^2}
\]
The group-relative advantage for each trajectory is defined as:
\[
\widehat A_i^{GRPO} = \frac{R_i - \bar R}{\sigma_R + \epsilon}
\]
where $\epsilon$ ensures numerical stability. The token-level policy update uses PPO-style likelihood ratios $r_{i,t}(\theta)$ (between new and old policy at each token), with a clipped surrogate objective:
\[
L_{GRPO}(\theta) = \mathbb{E}_x \left[ \frac{1}{N}\sum_{i=1}^N \frac{1}{T_i}\sum_{t=1}^{T_i} \min\left(r_{i,t} \widehat A_i,\, \mathrm{clip}(r_{i,t}, 1-\epsilon_{low}, 1+\epsilon_{high})\,\widehat A_i\right) \right]
\]
Here, all per-prompt statistics are localized within each group, allowing scale-invariance and variance suppression without a value network or explicit baseline [2602.06717].

## 2. Theoretical Analysis: Finite-Group Bias and Tail-Miss Probability

Although large group sizes $N$ approximate population statistics and minimize bias, they are computationally infeasible in practice. Finite $N$ introduces a characteristic bias in policy learning: rare but correct modes are often unsampled and thus ignored or even downweighted by the normalization.

The "tail-miss" probability $P_{miss}(N)$ quantifies the chance that, for a prompt $x$, an update occurs (group contains both correct and incorrect trajectories) but none of the correct rollouts is from the rare, desired subspace. This is expressed as:
\[
P_{miss}(N) = (1-\tau)^N - (\mu_{pos} - \tau)^N - (1-\mu_{pos})^N
\]
where $\mu_{pos}$ is the current probability of generating a correct trajectory and $\tau$ the mass on "rare-correct" solutions. $P_{miss}(N)$ is non-monotonic in group size: it vanishes for very small $N$ (no updates), also for very large $N$ (coverage complete), but peaks at modest sizes where active updates bias learning toward common solutions while missing the rare.

Another consequence is the shrinking of "unsampled-correct mass": the probability mass on correct solutions not appearing in any sampled trajectory. Even as total correct mass can grow, drift induced by the group baseline can systematically reduce unsampled-correct mass, impeding exploration of rare-but-desirable solutions [2602.06717].

## 3. F-GRPO: Focal Difficulty-Aware Scaling for Diversity Recovery

Motivated by the bias identified above, Focal Group Relative Policy Optimization (F-GRPO) introduces a simple, prompt-specific scaling coefficient inspired by the Focal loss for classification:

1. Estimate per-prompt empirical success rate:
   \[
   \widehat\mu_{pos}(x) = \frac{\bar R(x) - R_w}{R_c - R_w} = \frac{X}{N}
   \]
   where $X$ is the number of correct trajectories in the group.

2. Apply a Focal scaling with exponent $\gamma\geq 0$:
   \[
   g(x) = [1 - \widehat \mu_{pos}(x)]^\gamma
   \]
   which downweights the gradient update for prompts with many successes ("easy" prompts), thus emphasizing harder cases and rare modes.

3. The group-relative advantage is rescaled:
   \[
   \widehat A_i^{F\!-\!GRPO} = g(x)\,\widehat A_i^{GRPO}
   \]
   and the surrogate loss becomes:
   \[
   L_{F\!-\!GRPO}(\theta) = \mathbb{E}_x \left[g(x)\, \frac{1}{N}\sum_{i,t} L_{i,t}^{clip}\right]
   \]
   As $\gamma \rightarrow 0$, this recovers vanilla GRPO; as $\gamma > 0$, "obvious" (high-success) prompts are suppressed, counteracting group-drift bias [2602.06717].

## 4. Algorithmic Workflow and Pseudocode

F-GRPO differs from GRPO only in per-prompt computation of the empirical success rate and scaling of group advantages. The main steps per training iteration are as follows:

1. Draw a batch of prompts $\{x_b\}_{b=1}^B$.
2. For each prompt $x$, sample $N$ rollouts $\{o_i\}_{i=1}^N$ under $\pi_\theta$.
3. Compute rewards $R_i$, and the empirical success rate $\widehat\mu_{pos}$.
4. Set the focal scaling $g(x) = [1 - \widehat \mu_{pos}(x)]^\gamma$.
5. Compute group mean $\bar R$, standard deviation $\sigma_R$, and (scaled) group-relative advantages for each trajectory.
6. Compute token-level surrogate losses (as in PPO), using the scaled advantage.
7. Aggregate gradients and perform parameter update.

Pseudocode excerpt [2602.06717]:

```python
for each prompt x in batch:
    draw N rollouts {o_i ~ π_θ(·|x)}
    compute rewards R_i ∈ {R_c,R_w}
    μ̂_pos = (sum R_i / N - R_w)/(R_c - R_w)
    g = (1 - μ̂_pos)^γ
    compute group mean \bar R, std σ_R
    for i in 1..N:
        Â_i = (R_i - \bar R)/(σ_R+ε)
        Â_i ← g · Â_i
# compute PPO-style gradients using Â_i and perform update
```

## 5. Empirical Performance and Group Size Trade-Offs

Applied to the Qwen2.5-7B LLM, F-GRPO (with group size $N=8$ and $\gamma=0.5$) achieves substantial accuracy improvements in both in-domain and out-of-domain mathematical reasoning tasks:

- Baseline GRPO: pass@256 = 64.1, pass@1 ≈ 37.3
- F-GRPO (same $N$): pass@256 = 70.3, pass@1 = 38.6

F-GRPO at group size $N=8$ matches or slightly exceeds GRPO with $N=32$ (which achieves pass@256 ≈ 70.1), yielding a $4\times$ reduction in rollout cost for similar diversity and success metrics. On out-of-domain tasks, F-GRPO improves pass@256 from 55.9 to 63.3. The method similarly benefits DAPO and CISPO policy optimization variants.

Group size effects:
- $N=2$: most groups lack both success and failure, increasing diversity but reducing pass@1.
- $N=8$: updates boost pass@1 but at the expense of pass@256 (diversity).
- $N=32$: larger groups recover diversity, but at higher compute cost.
- F-GRPO with $N=8$ achieves the diversity and success of larger groups at no extra cost [2602.06717].

## 6. Broader Significance and Applicability

Theoretical and empirical results show that GRPO's group normalization, while robust and scalable, introduces a non-monotonic, group-size-dependent bias toward common solutions, especially when rare-correct modes are under-sampled. F-GRPO supplies a minimal, focal-inspired variant that actively corrects this effect by adaptively weighting updates according to observed group success rates. This technique is agnostic to the underlying group-relative RL algorithm and can be directly applied to DAPO, CISPO, and other group-normalized RLVR methods.

By stabilizing and diversifying policy updates without incurring additional sampling or computational cost, F-GRPO enables practical deployment of group-relative RL schemes in domains where rare modes are critical (reasoning, code generation, safety-sensitive settings), and large group sizes are not computationally viable [2602.06717].

## 7. Limitations and Directions for Future Research

F-GRPO's focal scaling necessarily depends upon accurate online estimation of group success rates, which may be impacted by reward sparsity or early training dynamics. Although the method alleviates group-drift and loss of rare-correct mass, further advances may require integrating explicit rare-mode tracking, enhanced rollout-generation strategies, or importance correction for under-sampled trajectories.

Possible research directions include analytical study of scaling behavior across RLVR tasks, adaptive scheduling of the focal exponent $\gamma$ based on training signals, and application to settings with reward corruption or significant noise.

---

**References:**
- F-GRPO and all above results: [2602.06717]

Source: https://www.emergentmind.com/topics/grouped-regularized-policy-optimization-grpo