GEPO: Group Expectation Policy Optimization

• GEPO is an off-policy correction method that replaces token-level importance sampling with group-wise expected proposal probabilities to reduce variance in asynchronous RL settings.
• The approach aggregates probabilities over mini-groups and incorporates a defensive smoothing term to stabilize updates under high network delays.
• Experiments on MATH-500 with Qwen3-1.7B demonstrate that GEPO maintains less than 3% performance degradation compared to synchronous baselines in severe delay conditions.

Group Expectation Policy Optimization (GEPO) is an off-policy correction method for asynchronous policy-gradient training in geographically distributed, high-latency environments. In the HeteroRL architecture, rollout actors sample trajectories under a stale policy $\pi_{\theta_k}$ while a learner updates a more recent policy $\pi_{\theta_{k+\tau}}$; as network delay $\tau$ increases, the resulting policy mismatch raises the KL divergence and can cause standard importance sampling to fail through high-variance weights. GEPO addresses this by moving from token-level to full-sample importance ratios and replacing each sample’s proposal probability in the denominator with a group-wise expected proposal probability, together with a defensive smoothing term. On MATH-500 with Qwen3-1.7B, the reported implementation maintains superior stability over methods such as GRPO and shows less than $3\%$ performance degradation under $1800$-second delays (Zhang et al., 25 Aug 2025).

1. Distributed RL setting and motivation

GEPO was introduced for heterogeneous distributed reinforcement learning in LLMs, where rollout sampling and parameter learning are decoupled. In HeteroRL, rollout actors generate trajectories under the behavior policy $q(y|x)=\pi_{\theta_k}(y|x)$, but the learner performs updates using the newer policy $p(y|x)=\pi_{\theta_{k+\tau}}(y|x)$. The paper identifies latency-induced KL divergence between these policies as the central cause of instability: when network delay grows, the conventional importance ratio becomes increasingly variable, and importance sampling can fail (Zhang et al., 25 Aug 2025).

The standard clipped sequence-level importance weight is

$$w_{\mathrm{std}}(y|x)=\operatorname{clip}\!\left(\frac{p(y|x)}{q(y|x)},\,1-\epsilon,\,1+\epsilon\right).$$

In the formulation used for GEPO, this failure mode is tied not merely to off-policy training in the abstract, but specifically to asynchronous rollout collection under stale sampler parameters. The method therefore targets a narrower problem than generic PPO-style stabilization: it is designed for settings in which the behavior policy is delayed, geographically distributed, and potentially far from the learner’s current policy.

2. Group-expectation importance weighting

The core construction in GEPO is the Group Expectation Importance Weight (GEIW). For each input $x$, the sampler generates a mini-group of $K$ responses $\pi_{\theta_{k+\tau}}$0. Their proposal probabilities under the stale policy are defined as

$\pi_{\theta_{k+\tau}}$1

These values are normalized within the group as

$\pi_{\theta_{k+\tau}}$2

and the expected proposal probability under $\pi_{\theta_{k+\tau}}$3 is then estimated by

$\pi_{\theta_{k+\tau}}$4

GEPO replaces the per-sample denominator $\pi_{\theta_{k+\tau}}$5 with this group-wise expectation, producing the unclipped weight

$\pi_{\theta_{k+\tau}}$6

To guard against bias when the group estimate is poor, the paper introduces a defensive-sampling variant,

$\pi_{\theta_{k+\tau}}$7

where $\pi_{\theta_{k+\tau}}$8 is a smoothing coefficient and $\pi_{\theta_{k+\tau}}$9 prevents gradients from flowing through $\tau$0. The paper states that this smoothing coefficient is set proportional to $\tau$1 in experiments (Zhang et al., 25 Aug 2025).

This construction differs materially from standard importance sampling. Rather than correcting each trajectory with its own proposal probability, GEPO uses a shared denominator estimated from a local response group. A plausible implication is that variance control is transferred from trajectory-specific correction to group-level estimation.

3. Objective and algorithmic structure

GEPO is embedded in a CPPO-KL-style update. With advantage $\tau$2, the learner optimizes

$\tau$3

The paper’s pseudocode specifies the following loop. For each learner iteration, a batch of prompts $\tau$4 is collected. For each prompt $\tau$5, the stale sampler policy $\tau$6 generates $\tau$7 trajectories $\tau$8. The learner then computes $\tau$9 and $3\%$0, forms the group denominator

$3\%$1

sets a prompt-specific smoothing coefficient $3\%$2, and computes

$3\%$3

Advantages are estimated by group-centered rewards,

$3\%$4

and the policy is updated with the clipped surrogate plus the KL penalty. After the update, the new parameters are broadcast back to samplers, again with network delay (Zhang et al., 25 Aug 2025).

The published complexity discussion attributes the extra cost primarily to $3\%$5 additional forward-probability evaluations per prompt and two small reductions, $3\%$6 and $3\%$7. Compared with token-level importance sampling, GEPO avoids per-token clipping and gradients; compared with vanilla PPO or GRPO, the main overhead is the $3\%$8-fold sequence probability computation, which the paper characterizes as minor when $3\%$9 batch size and sequence length (Zhang et al., 25 Aug 2025).

4. Theoretical variance reduction

The main theoretical claim is an exponential variance reduction theorem. The paper states that there exists a constant $1800$0 such that

$1800$1

In particular, if

$1800$2

then the variance of the standard weight is larger than the variance of the group-expectation weight (Zhang et al., 25 Aug 2025).

The proof sketch in the paper proceeds through auxiliary quantities including $1800$3, the relation $1800$4, and a bound from Pinsker’s or Csiszár’s inequality yielding $1800$5. It then defines $1800$6, $1800$7, and $1800$8, and derives a lower bound on the variance difference of the form $1800$9, from which one can set $q(y|x)=\pi_{\theta_k}(y|x)$0 (Zhang et al., 25 Aug 2025).

The same section also records the method’s limitations. GEPO assumes that group size $q(y|x)=\pi_{\theta_k}(y|x)$1 is sufficient to approximate $q(y|x)=\pi_{\theta_k}(y|x)$2; in very small-$q(y|x)=\pi_{\theta_k}(y|x)$3 regimes the variance reduction is smaller. The estimator introduces a small bias because the denominator no longer exactly normalizes. The smoothing coefficient must be tuned. In low-KL, low-delay settings, GEPO’s variance can sometimes slightly exceed standard importance sampling, although the reported overall stability gains still dominate. Proposed extensions include adaptive group sizes, alternative estimators of $q(y|x)=\pi_{\theta_k}(y|x)$4, and dynamic $q(y|x)=\pi_{\theta_k}(y|x)$5 schedules (Zhang et al., 25 Aug 2025).

5. Experimental behavior under delay

The reported experiments use MATH-500 with Pass@1 on Qwen3-1.7B. Delays are simulated up to $q(y|x)=\pi_{\theta_k}(y|x)$6 seconds, corresponding to approximately $q(y|x)=\pi_{\theta_k}(y|x)$7 update steps of staleness. Baselines are GRPO, described as token-level importance sampling, and GSPO, described as sequence-level importance sampling; all methods use CPPO-KL regularization (Zhang et al., 25 Aug 2025).

Method	Delay	Best / Last
GRPO	0 steps	77.1 / 41.2
GRPO	8 steps	73.3 / 44.6
GRPO	64 steps	70.2 / 27.3
GSPO	0 steps	78.2 / 31.6
GSPO	8 steps	74.5 / 60.4
GSPO	64 steps	71.8 / 58.7
GEPO	0 steps	78.5 / 77.3
GEPO	8 steps	75.8 / 75.7
GEPO	64 steps	75.3 / 74.4

Under the most severe delay regime, GEPO’s final accuracy of approximately $q(y|x)=\pi_{\theta_k}(y|x)$8 is reported as only $q(y|x)=\pi_{\theta_k}(y|x)$9 below the zero-delay synchronous baseline of $p(y|x)=\pi_{\theta_{k+\tau}}(y|x)$0. The accompanying figures are described as showing low variance of $p(y|x)=\pi_{\theta_{k+\tau}}(y|x)$1, stable gradient norms, and no collapse in training reward for GEPO, while GRPO and GSPO suffer catastrophic divergence under high delay (Zhang et al., 25 Aug 2025).

Within the scope of these experiments, GEPO’s empirical contribution is therefore not simply higher peak performance, but the preservation of final performance as rollout staleness grows. This suggests that the method is primarily a stability mechanism for asynchronous large-scale RL rather than a generic reward-optimization heuristic.

6. Related methods and nomenclature

The term “group” appears in several nearby LLM-RL methods, but the aggregated quantity differs across papers. In Group-Relative Policy Optimization (GRPO), as summarized in “BiasGRPO: Stabilizing Bias Mitigation in High-Variance Reward Landscapes via Group-Relative Policy Optimization” (Reddy et al., 3 Jun 2026), the group statistic is a reward baseline: for each prompt, $p(y|x)=\pi_{\theta_{k+\tau}}(y|x)$2 completions are sampled, scalar rewards $p(y|x)=\pi_{\theta_{k+\tau}}(y|x)$3 are scored, the group mean $p(y|x)=\pi_{\theta_{k+\tau}}(y|x)$4 and often the group standard deviation $p(y|x)=\pi_{\theta_{k+\tau}}(y|x)$5 are computed, and the advantage becomes either $p(y|x)=\pi_{\theta_{k+\tau}}(y|x)$6 or $p(y|x)=\pi_{\theta_{k+\tau}}(y|x)$7. GRPO thus removes the learned critic from PPO and replaces it with a group-relative baseline. On BOLD, RealToxicityPrompts, BBQ, and TruthfulQA, the reported table shows GRPO outperforming DPO and PPO on the listed bias metrics while preserving TruthfulQA accuracy (Reddy et al., 3 Jun 2026).

A separate nomenclature issue arises from “On-Policy Optimization with Group Equivalent Preference for Multi-Programming Language Understanding” (Wu et al., 19 May 2025), where the acronym GEPO denotes Group Equivalent Preference Optimization, not Group Expectation Policy Optimization. That method is an off-policy preference-optimization loss defined over groups of equivalent and inequivalent compiler IRs. Its objective combines a group-wise Bradley–Terry term, which raises the average log-probability ratio of winner IRs above that of loser IRs, with a variance penalty over the winner group to enforce similar scores among equivalent IRs. It is integrated into the OORL framework alongside on-policy RL for code translation, with reported settings $p(y|x)=\pi_{\theta_{k+\tau}}(y|x)$8, $p(y|x)=\pi_{\theta_{k+\tau}}(y|x)$9, winner-group size $$w_{\mathrm{std}}(y|x)=\operatorname{clip}\!\left(\frac{p(y|x)}{q(y|x)},\,1-\epsilon,\,1+\epsilon\right).$$0, loser-group size $$w_{\mathrm{std}}(y|x)=\operatorname{clip}\!\left(\frac{p(y|x)}{q(y|x)},\,1-\epsilon,\,1+\epsilon\right).$$1, and weight $$w_{\mathrm{std}}(y|x)=\operatorname{clip}\!\left(\frac{p(y|x)}{q(y|x)},\,1-\epsilon,\,1+\epsilon\right).$$2 (Wu et al., 19 May 2025).

Accordingly, “GEPO” is not a univocal acronym in the 2025–2026 LLM-RL literature. Group Expectation Policy Optimization aggregates proposal probabilities for off-policy correction under stale-policy mismatch (Zhang et al., 25 Aug 2025); GRPO aggregates rewards to form a baseline (Reddy et al., 3 Jun 2026); Group Equivalent Preference Optimization aggregates preference signals over IR groups in code translation (Wu et al., 19 May 2025). The expansion and training context therefore determine which method is being referenced.