---
title: 'GEPO: Group Expectation Policy Optimization'
url: https://www.emergentmind.com/topics/group-expectation-policy-optimization-gepo
type: topic
---

# GEPO: Group Expectation Policy Optimization

Group Expectation Policy Optimization (GEPO) is an off-policy correction method for asynchronous policy-gradient training in geographically distributed, high-latency environments. In the HeteroRL architecture, rollout actors sample trajectories under a stale policy $\pi_{\theta_k}$ while a learner updates a more recent policy $\pi_{\theta_{k+\tau}}$; as network delay $\tau$ increases, the resulting policy mismatch raises the KL divergence and can cause standard importance sampling to fail through high-variance weights. GEPO addresses this by moving from token-level to full-sample importance ratios and replacing each sample’s proposal probability in the denominator with a group-wise expected proposal probability, together with a defensive smoothing term. On MATH-500 with Qwen3-1.7B, the reported implementation maintains superior stability over methods such as GRPO and shows less than $3\%$ performance degradation under $1800$-second delays [2508.17850].

## 1. Distributed RL setting and motivation

GEPO was introduced for heterogeneous distributed reinforcement learning in large language models, where rollout sampling and parameter learning are decoupled. In HeteroRL, rollout actors generate trajectories under the behavior policy $q(y|x)=\pi_{\theta_k}(y|x)$, but the learner performs updates using the newer policy $p(y|x)=\pi_{\theta_{k+\tau}}(y|x)$. The paper identifies latency-induced KL divergence between these policies as the central cause of instability: when network delay grows, the conventional importance ratio becomes increasingly variable, and importance sampling can fail [2508.17850].

The standard clipped sequence-level importance weight is

$$
w_{\mathrm{std}}(y|x)=\operatorname{clip}\!\left(\frac{p(y|x)}{q(y|x)},\,1-\epsilon,\,1+\epsilon\right).
$$

In the formulation used for GEPO, this failure mode is tied not merely to off-policy training in the abstract, but specifically to asynchronous rollout collection under stale sampler parameters. The method therefore targets a narrower problem than generic PPO-style stabilization: it is designed for settings in which the behavior policy is delayed, geographically distributed, and potentially far from the learner’s current policy.

## 2. Group-expectation importance weighting

The core construction in GEPO is the Group Expectation Importance Weight (GEIW). For each input $x$, the sampler generates a mini-group of $K$ responses $\{y_1,\dots,y_K\}\sim q(\cdot|x)$. Their proposal probabilities under the stale policy are defined as

$$
q_i = q(y_i|x)=\prod_{t=1}^{T} q(y_{i,t}\mid x,y_{i,<t})^{1/T}.
$$

These values are normalized within the group as

$$
\hat q_i = \frac{q_i}{\sum_{j=1}^{K} q_j},
$$

and the expected proposal probability under $q$ is then estimated by

$$
\hat{\mathbb E}_q[q(y|x)] = \sum_{i=1}^{K} q_i \hat q_i
= \frac{\sum_{i=1}^{K} q_i^2}{\sum_{i=1}^{K} q_i}.
$$

GEPO replaces the per-sample denominator $q(y_i|x)$ with this group-wise expectation, producing the unclipped weight

$$
w_{\mathrm{GEIW}}(y_i|x)=\frac{p(y_i|x)}{\hat{\mathbb E}_q[q(y|x)]}.
$$

To guard against bias when the group estimate is poor, the paper introduces a defensive-sampling variant,

$$
w_{\mathrm{final}}(y|x)=
\frac{p(y|x)}
{\epsilon\cdot \operatorname{detach}[p(y|x)] + (1-\epsilon)\cdot \hat{\mathbb E}_q[q(y|x)]},
$$

where $\epsilon\in(0,1)$ is a smoothing coefficient and $\operatorname{detach}[\cdot]$ prevents gradients from flowing through $p(y|x)$. The paper states that this smoothing coefficient is set proportional to $\operatorname{Var}(q_i)$ in experiments [2508.17850].

This construction differs materially from standard importance sampling. Rather than correcting each trajectory with its own proposal probability, GEPO uses a shared denominator estimated from a local response group. A plausible implication is that variance control is transferred from trajectory-specific correction to group-level estimation.

## 3. Objective and algorithmic structure

GEPO is embedded in a CPPO-KL-style update. With advantage $A(x,y)$, the learner optimizes

$$
L(\theta)=
- \mathbb{E}_{(x,y)\sim \mathrm{rollouts}}
\left[
\min\!\left(
w_{\mathrm{final}}(y|x)\,A(x,y),\,
\operatorname{clip}(w_{\mathrm{final}}(y|x),1-\epsilon,1+\epsilon)\,A(x,y)
\right)
+ \beta\,\mathrm{KL}(\pi_{\theta_k}\|\pi_{\theta_{k+\tau}})
\right].
$$

The paper’s pseudocode specifies the following loop. For each learner iteration, a batch of prompts $\{x_1,\dots,x_M\}$ is collected. For each prompt $x_j$, the stale sampler policy $\pi_{\theta_k}$ generates $K$ trajectories $\{y_{j,1},\dots,y_{j,K}\}$. The learner then computes $q_{j,i}=\pi_{\theta_k}(y_{j,i}|x_j)^{1/|y|}$ and $p_{j,i}=\pi_{\theta_{k+\tau}}(y_{j,i}|x_j)^{1/|y|}$, forms the group denominator

$$
D_j=\frac{\sum_{i=1}^{K} q_{j,i}^2}{\sum_{i=1}^{K} q_{j,i}},
$$

sets a prompt-specific smoothing coefficient $\epsilon_j\propto \operatorname{Var}(\{q_{j,i}\})$, and computes

$$
w_{j,i}=
\frac{p_{j,i}}
{\epsilon_j\cdot \operatorname{detach}(p_{j,i})+(1-\epsilon_j)\cdot D_j}.
$$

Advantages are estimated by group-centered rewards,

$$
A_{j,i}=r(x_j,y_{j,i})-\operatorname{mean}_i(r(x_j,y_{j,i})),
$$

and the policy is updated with the clipped surrogate plus the KL penalty. After the update, the new parameters are broadcast back to samplers, again with network delay [2508.17850].

The published complexity discussion attributes the extra cost primarily to $O(K)$ additional forward-probability evaluations per prompt and two small reductions, $\sum q_i$ and $\sum q_i^2$. Compared with token-level importance sampling, GEPO avoids per-token clipping and gradients; compared with vanilla PPO or GRPO, the main overhead is the $K$-fold sequence probability computation, which the paper characterizes as minor when $K\ll$ batch size and sequence length [2508.17850].

## 4. Theoretical variance reduction

The main theoretical claim is an exponential variance reduction theorem. The paper states that there exists a constant $C$ such that

$$
\operatorname{Var}_q\!\left[\frac{p(y|x)}{q(y|x)}\right]
-
\operatorname{Var}_q\!\left[\frac{p(y|x)}{\hat{\mathbb E}_q[q(y|x)]}\right]
\ge
\exp\!\big(D_{\mathrm{KL}}(p\|q)\big)-C.
$$

In particular, if

$$
D_{\mathrm{KL}}(p\|q)>\ln C,
$$

then the variance of the standard weight is larger than the variance of the group-expectation weight [2508.17850].

The proof sketch in the paper proceeds through auxiliary quantities including $I_1=\sum p_i^2/q_i$, the relation $D_{\chi^2}(p\|q)=I_1-1$, and a bound from Pinsker’s or Csiszár’s inequality yielding $I_1\ge \exp(D_{\mathrm{KL}})$. It then defines $A=(\sum q_i^2)^2$, $B=(\sum p_i q_i)^2$, and $I_2=\sum p_i^2 q_i$, and derives a lower bound on the variance difference of the form $\Delta\ge \exp(D_{\mathrm{KL}})-(n^2+1)$, from which one can set $C=n^2+1$ [2508.17850].

The same section also records the method’s limitations. GEPO assumes that group size $K$ is sufficient to approximate $\mathbb E_q[q(y|x)]$; in very small-$K$ regimes the variance reduction is smaller. The estimator introduces a small bias because the denominator no longer exactly normalizes. The smoothing coefficient must be tuned. In low-KL, low-delay settings, GEPO’s variance can sometimes slightly exceed standard importance sampling, although the reported overall stability gains still dominate. Proposed extensions include adaptive group sizes, alternative estimators of $\mathbb E_q[q]$, and dynamic $\epsilon$ schedules [2508.17850].

## 5. Experimental behavior under delay

The reported experiments use MATH-500 with Pass@1 on Qwen3-1.7B. Delays are simulated up to $1800$ seconds, corresponding to approximately $64$ update steps of staleness. Baselines are GRPO, described as token-level importance sampling, and GSPO, described as sequence-level importance sampling; all methods use CPPO-KL regularization [2508.17850].

| Method | Delay | Best / Last |
|---|---:|---:|
| GRPO | 0 steps | 77.1 / 41.2 |
| GRPO | 8 steps | 73.3 / 44.6 |
| GRPO | 64 steps | 70.2 / 27.3 |
| GSPO | 0 steps | 78.2 / 31.6 |
| GSPO | 8 steps | 74.5 / 60.4 |
| GSPO | 64 steps | 71.8 / 58.7 |
| GEPO | 0 steps | 78.5 / 77.3 |
| GEPO | 8 steps | 75.8 / 75.7 |
| GEPO | 64 steps | 75.3 / 74.4 |

Under the most severe delay regime, GEPO’s final accuracy of approximately $74.4$ is reported as only $3\%$ below the zero-delay synchronous baseline of $77.3$. The accompanying figures are described as showing low variance of $w$, stable gradient norms, and no collapse in training reward for GEPO, while GRPO and GSPO suffer catastrophic divergence under high delay [2508.17850].

Within the scope of these experiments, GEPO’s empirical contribution is therefore not simply higher peak performance, but the preservation of final performance as rollout staleness grows. This suggests that the method is primarily a stability mechanism for asynchronous large-scale RL rather than a generic reward-optimization heuristic.

## 6. Related methods and nomenclature

The term “group” appears in several nearby LLM-RL methods, but the aggregated quantity differs across papers. In Group-Relative Policy Optimization (GRPO), as summarized in “BiasGRPO: Stabilizing Bias Mitigation in High-Variance Reward Landscapes via Group-Relative Policy Optimization” [2606.04807], the group statistic is a reward baseline: for each prompt, $G$ completions are sampled, scalar rewards $r_i$ are scored, the group mean $\bar r=(1/G)\sum_i r_i$ and often the group standard deviation $\sigma$ are computed, and the advantage becomes either $(r_i-\bar r)/\sigma$ or $r_i-\bar r$. GRPO thus removes the learned critic from PPO and replaces it with a group-relative baseline. On BOLD, RealToxicityPrompts, BBQ, and TruthfulQA, the reported table shows GRPO outperforming DPO and PPO on the listed bias metrics while preserving TruthfulQA accuracy [2606.04807].

A separate nomenclature issue arises from “On-Policy Optimization with Group Equivalent Preference for Multi-Programming Language Understanding” [2505.12723], where the acronym GEPO denotes **Group Equivalent Preference Optimization**, not Group Expectation Policy Optimization. That method is an off-policy preference-optimization loss defined over groups of equivalent and inequivalent compiler IRs. Its objective combines a group-wise Bradley–Terry term, which raises the average log-probability ratio of winner IRs above that of loser IRs, with a variance penalty over the winner group to enforce similar scores among equivalent IRs. It is integrated into the OORL framework alongside on-policy RL for code translation, with reported settings $\beta=1.0$, $\lambda=1.0$, winner-group size $n=4$, loser-group size $m=4$, and weight $w_{\mathrm{gepo}}=0.01$ [2505.12723].

Accordingly, “GEPO” is not a univocal acronym in the 2025–2026 LLM-RL literature. Group Expectation Policy Optimization aggregates proposal probabilities for off-policy correction under stale-policy mismatch [2508.17850]; GRPO aggregates rewards to form a baseline [2606.04807]; Group Equivalent Preference Optimization aggregates preference signals over IR groups in code translation [2505.12723]. The expansion and training context therefore determine which method is being referenced.

Source: https://www.emergentmind.com/topics/group-expectation-policy-optimization-gepo