Empirical Bayes Policy Optimization (EBPO)

• EBPO is a reinforcement learning framework that employs a global shrinkage estimator to stabilize policy updates and reduce high variance in group-based rewards.
• It combines local group statistics with a global prior using efficient running estimators, yielding lower mean squared error and non-vanishing gradients.
• EBPO enhances exploration and performance, particularly under small-sample and curriculum learning regimes, as demonstrated on LLM reasoning benchmarks.

Empirical Bayes Policy Optimization (EBPO) is a reinforcement learning framework designed to address stability and efficiency limitations in Group-Relative Policy Optimization (GRPO). While GRPO is widely used to optimize LLMs in the Reinforcement Learning with Verifiable Rewards (RLVR) paradigm, it faces challenges with high-variance estimators and vanishing gradients, particularly under constrained computational budgets and saturated regimes. EBPO introduces a statistically principled shrinkage method that regularizes local group-based advantage estimators using a global prior, resulting in provably lower estimator variance, non-vanishing gradients, and more robust policy updates (Han et al., 5 Feb 2026).

1. RLVR and the Limitations of GRPO

RLVR tasks require a policy $\pi_\theta$ to generate continuations in response to a given prompt $q$, where each continuation $o_i$ receives a “verifiable” reward $r_i$ (binary or scalar). In GRPO, to forgo a separate value network, advantages are computed by normalizing rewards within a group:

• $\mu_\text{group} = \frac{1}{G} \sum_{i=1}^G r_i$
• $$\sigma^2_\text{group} = \frac{1}{G-1} \sum (r_i-\mu_\text{group})^2$$
• $$A_i = \frac{r_i-\mu_\text{group}}{\sigma_\text{group}+\epsilon}$$.

These advantages are used in a clipped-PPO (Proximal Policy Optimization) surrogate objective. The primary limitations of GRPO are:

• High variance for small group sizes ($G$): Local means $\mu_\text{group}$ are noisy, leading to erratic gradients and rapid entropy loss.
• Vanishing gradients under saturation: When all rewards are identical (all $r_i=0$ or $r_i=1$), gradients for that sample vanish, impeding learning progress (Han et al., 5 Feb 2026).

2. Mathematical Formulation of Empirical Bayes Policy Optimization

EBPO enhances baseline estimation by introducing a shrinkage estimator:

1. Modeling Assumptions: Each prompt $q$ has a latent success probability $\theta_q$, drawn from a global prior $\theta_q \sim \mathcal{N}(\mu_\text{glob}, \tau^2)$.
2. Group-based reward: The sample mean $\mu_\text{group}$ is an unbiased estimator for $\theta_q$ with variance $\sigma^2/G$.
3. Shrinkage Estimator:
   • Shrinkage factor: $S_q = \frac{\sigma^2/G}{\sigma^2/G + \tau^2}$
   • Baseline: $$V_q^\text{EB} = (1-S_q)\mu_\text{group} + S_q \mu_\text{glob}$$
   • Raw advantage: $A_i^\text{raw} = r_i - V_q^\text{EB}$
   • Final advantage: batch-normalize $A_i^\text{raw}$ to $$A_i = \frac{A_i^\text{raw} - \mu_A}{\sigma_A + \epsilon}$$.
4. Online Estimation: Running statistics for $\mu_\text{glob}$, $\sigma^2$, and $\tau^2$ are updated using Welford's algorithm, requiring only $O(MG)$ computation per batch without storing full histories (Han et al., 5 Feb 2026).

3. Algorithmic Framework and Implementation

The EBPO procedure per iteration is as follows:

• Sample $M$ prompts.
• For each, generate $G$ continuations and record rewards.
• Compute local group statistics and update global reward statistics via Welford's algorithm.
• For each prompt, calculate shrinkage $S_q$, EBPO baseline $V_q^\text{EB}$, and raw advantages.
• Batch-normalize advantages and update the policy through a clipped surrogate objective.

Complexity: The methodology introduces negligible computational overhead, limited to scalar shrinkage calculations and Welford updates. No hyperparameters are added beyond those in standard PPO (e.g., $\epsilon$, learning rate). EBPO is modular and reuses PPO machinery (Han et al., 5 Feb 2026).

4. Theoretical Analysis

EBPO’s formulation yields several provable guarantees in the Gaussian approximation:

1. Non-Vanishing Gradients: For fully failed groups ($r_i=0$), GRPO advantage $A_i=0$ but EBPO yields $A_i^\text{raw} = -S_q \mu_\text{glob} < 0$, enabling penalization and updates even in saturated regimes.
2. Estimator Variance Reduction: EBPO’s baseline has strictly lower mean squared error (MSE) relative to GRPO, minimizing $MSE(V^\text{EBPO}, \theta_q)$ due to optimal linear shrinkage.
3. Entropy Preservation: The expected per-step entropy reduction $\Delta H(\pi)$ is lower for EBPO than GRPO, as shrinkage suppresses erratic updates, thereby preserving exploration.
4. Clustered Sampling Benefits: When tasks are sampled by topic or difficulty clusters, the prior tracks cluster means, further reducing estimator MSE compared to random shuffling. This suggests that curriculum strategies enhance prior accuracy and training stability (Han et al., 5 Feb 2026).

5. Empirical Performance and Stability

5.1 Benchmarks and Evaluation Protocol

Empirical results are reported on standardized mathematical reasoning datasets (AIME 2024/25, AMC23, MATH-500, OlympiadBench) using models such as Qwen3-8B, LLaMA3-8B, and Qwen3-14B. Competitors include GRPO, DAPO, Dr-GRPO, and EntropyMech, with Pass@1 as the main metric. Curriculum learning strategies are incorporated:

• EBPO-topic: Clustering prompts by semantic domain.
• EBPO-diff: Difficulty-based ordering (easy to hard).

5.2 Accuracy and Sample Efficiency

Method	MATH-500	AIME-24	AIME-25	AMC23	Olympiad	Avg.
EBPO-topic	76.80	56.04	47.92	86.25	54.93	64.39
GRPO	65.60	50.21	42.29	89.53	45.99	58.72
Dr-GRPO	67.68	51.04	32.71	85.00	44.91	56.67
DAPO	58.39	45.63	32.71	82.81	43.62	52.63
EntropyMech	53.88	37.92	30.42	79.99	43.69	49.18

EBPO-topic consistently leads, outperforming GRPO by approximately 5.7 percentage points on average and yielding top results across most benchmarks for Qwen3-8B. This indicates robust generalization and improved sample usage, especially under topic clustering and small group conditions.

5.3 Policy Stability and Gradient Dynamics

Empirical studies demonstrate:

• Persistent, non-vanishing policy gradients ($\|\nabla_\theta J\|$) under EBPO, addressing silent gradient issues common in late-stage GRPO training.
• Lower per-step KL-divergence between policy iterates, controlling abrupt shifts.
• Slower entropy decay, supporting maintained policy exploration.

When the group size $G$ varies, EBPO sustains higher sample efficiency, leading GRPO by 11.3 points in average Pass@1 for $G=8$.

5.4 Synergy with Curriculum Learning

Difficulty-ordered curricula (EBPO-diff) further stabilize prior estimates and improve performance, outstripping GRPO by 4–6 points on elite benchmarks (AIME) under $G=4$. A plausible implication is that strategic task ordering amplifies the effect of global shrinkage, particularly on high-difficulty samples.

6. Comparative Summary and Future Directions

EBPO addresses core weaknesses in GRPO by transitioning from a pure local (within-group) baseline to a hybrid shrinkage estimator that leverages both group-level evidence and a dynamically maintained global prior. Theoretical results establish lower estimator error, persistent gradients, and improved exploration. These benefits are realized empirically across reasoning-focused LLM benchmarks, with pronounced gains under small-sample and curriculum-based regimes (Han et al., 5 Feb 2026). Future investigation may assess EBPO’s generality beyond LLMs, explore alternative prior structures, or develop adaptive curricula to further boost estimator reliability and policy robustness.