Exploration-Enhanced Policy Optimization (EEPO)

• EEPO is a family of reinforcement learning methods that integrates explicit exploration mechanisms with policy optimization to mitigate premature convergence.
• It employs techniques such as intrinsic rewards, uncertainty bonuses, population diversity, and trajectory-level diversification to enhance exploration.
• The RLVR variant uses a sample-then-forget rollout procedure to preserve entropy and improve performance in sparse-reward and long-horizon environments.

Searching arXiv for the cited EEPO-related papers to ground the article in current arXiv records. Exploration-Enhanced Policy Optimization (EEPO) denotes a line of reinforcement learning methods that augment policy optimization with explicit mechanisms for broader or more directed exploration. In the usage reflected across recent work, the term can refer both to a general design pattern—policy optimization coupled to intrinsic rewards, uncertainty bonuses, population diversity, model-based discrepancies, entropy control, or trajectory-level diversification—and to a specific RLVR framework for LLMs that uses a “sample-then-forget” procedure to disrupt self-reinforcing rollout modes (Pan et al., 2018, Chen et al., 7 Oct 2025). Across these formulations, the central objective is to mitigate premature convergence, entropy collapse, local optima, or sparse-reward stagnation by altering either the reward signal, the advantage estimate, the rollout distribution, or the policy update itself.

1. Conceptual scope and problem setting

EEPO arises from a common diagnosis: standard policy optimization methods are often exploitation-dominant. In actor-critic control, insufficient exploration of the spatial space can yield suboptimal policies, as observed for Twin Delayed Deep Deterministic policy gradient (TD3) on 7-DOF robotic arm grasping (Hsieh et al., 2024). In on-policy optimization, PPO is described as suffering from limited exploration, premature convergence, or sensitivity to Gaussian exploration schedules (Zhang et al., 2020, Czworkowski et al., 21 Jan 2026). In RL for LLMs, GRPO is characterized as exploitation driven, with entropy decreasing monotonically and exploration fading (Li et al., 19 Jan 2026). In RLVR, repeated sampling and rewarding of dominant modes produces a self-reinforcing loop that erodes exploratory capacity (Chen et al., 7 Oct 2025).

Within this literature, EEPO methods do not replace policy optimization; they modify it so that the training signal better values uncertainty, novelty, diversity, or entropy. This suggests a unifying interpretation in which exploration is treated not as an external heuristic layered on top of policy learning, but as a first-class component of the optimization objective or of rollout generation itself. A plausible implication is that EEPO is best viewed as a family of policy-optimization augmentations rather than a single algorithmic template.

Representative instantiations span several regimes. POME adds a model-based/model-free target discrepancy to PPO as an exploration value (Pan et al., 2018). IEM-PPO adds an uncertainty-estimation intrinsic reward to PPO (Zhang et al., 2020). MPPO maintains a population of policies and selects among them using a combined performance-entropy rule (Pan et al., 2019). OPPO and related optimistic methods insert bonus terms directly into policy improvement with regret guarantees (Cai et al., 2019, Luo et al., 2021). In RL for LLMs, DCPO, HAEPO, EPO, AEPO, and the specific EEPO framework all alter entropy control, trajectory weighting, or rollout diversification to preserve exploration under long-horizon or sparse-reward conditions (Li et al., 19 Jan 2026, Trivedi et al., 26 Aug 2025, Xu et al., 26 Sep 2025, Liu et al., 7 Aug 2025, Chen et al., 7 Oct 2025).

2. Mechanistic families of exploration enhancement

The literature organizes naturally into several mechanism classes. The following grouping is an overview of named methods in the cited works.

Mechanism family	Representative method	Core exploration signal
Intrinsic novelty reward	TD3 + EECL	Novel-state reward from KDTree Euclidean distance
Uncertainty-based reward or advantage	IEM-PPO, EPPO	Uncertainty bonus or UCB advantage
Model-based discrepancy bonus	POME	$|Q^*_{b,t} - Q^*_{f,t}|$
Population or evolutionary diversity	MPPO, EPO, POEM	Policy entropy, mutation, or population diversity
Entropy or distribution regulation	DCPO, EPO, HAEPO	Distribution-level entropy control or trajectory entropy
Rollout diversification	EEPO, AEPO	Sample-then-forget or multi-answer generation

One family uses intrinsic rewards for novelty. EECL stores previously explored states in a buffer, organizes them with a KDTree, detects novelty through Euclidean distance, and adds a decaying exploration reward to the environment reward before TD3 updates (Hsieh et al., 2024). IEM-PPO similarly reshapes PPO with an uncertainty-based intrinsic reward, where the mixed reward is $r^+(s, a) = r(s, a) + c_1 \hat{N}(s)$ (Zhang et al., 2020).

A second family derives exploration from uncertainty in value estimation or dynamics. POME computes both model-free and model-based targets,

$$Q^*_{f,t} = r_t + \gamma V_\phi(s_{t+1}), \qquad Q^*_{b,t} = \hat r(s_t, a_t) + \gamma V_\phi(\hat T(s_t, a_t)),$$

and defines the exploration value as

$\epsilon_t = | Q^*_{b,t} - Q^*_{f,t} |,$

which is added to the TD-style advantage used in PPO (Pan et al., 2018). EPPO uses an evidential critic whose predictive variance induces a UCB-style advantage,

$$\hat{A}_t^{\mathrm{UCB}} = \mathbb{E}[\hat{A}_t^{\text{GAE}}] + \kappa \sqrt{\mathrm{Var}[\hat{A}_t^{\text{GAE}}]},$$

so that high-uncertainty regions are preferentially explored in non-stationary control (Akgül et al., 3 Mar 2025).

A third family introduces diversity structurally. MPPO maintains a buffer of $K$ policies and scores each by

$$f_k(\pi_t) = (1 - \alpha)\hat{J}_k(\pi_t) + \alpha \hat{\mathcal{H}}_k(\pi_t),$$

thereby selecting policies that jointly balance performance and entropy (Pan et al., 2019). EPO hybridizes neuroevolution with PPO-style exploitation, using crossover, adaptive mutation, and local fine-tuning (Mustafaoglu et al., 17 Apr 2025). POEM triggers policy-parameter mutation when KL divergence to a moving-average policy falls below a threshold, signaling stagnation (Czworkowski et al., 21 Jan 2026).

A fourth family operates at the distribution or trajectory level. HAEPO compresses each trajectory into a cumulative logarithmic likelihood and normalizes trajectories with a Plackett-Luce softmax, then adds entropy regularization and a soft KL penalty (Trivedi et al., 26 Aug 2025). DCPO regularizes the policy at the distribution level rather than through sample-centric heuristics, using a higher-temperature target distribution and double importance sampling (Li et al., 19 Jan 2026). EPO for multi-turn LLM agents regulates trajectory-aware entropy, adds an entropy smoothing regularizer, and modulates the effect through adaptive phase-based weighting (Xu et al., 26 Sep 2025).

3. Canonical mathematical formulations

Despite their diversity, EEPO methods repeatedly modify one of four mathematical objects: the reward, the advantage, the policy update, or the rollout distribution.

In reward shaping approaches, exploration enters as an additive intrinsic term. EECL detects a novel state when

$\|s - s'\|_2 \geq \epsilon,$

and assigns an exponentially decaying exploration reward

$r_e = r_{\text{max}} \times \gamma^n,$

which is then stored in the replay buffer as part of $(s, a, r + r_e, s')$ (Hsieh et al., 2024). EIPO formulates the use of intrinsic rewards as a constrained optimization problem rather than a fixed scalar mixture: $r^+(s, a) = r(s, a) + c_1 \hat{N}(s)$0 with a Lagrangian multiplier $r^+(s, a) = r(s, a) + c_1 \hat{N}(s)$1 that automatically increases or suppresses the weight on extrinsic reward as needed (Chen et al., 2022).

In advantage shaping, the exploration signal perturbs the policy gradient target more directly. POME defines

$r^+(s, a) = r(s, a) + c_1 \hat{N}(s)$2

and uses the resulting modified advantage inside the PPO surrogate loss (Pan et al., 2018). EPPO modifies the advantage through uncertainty propagation from an evidential value model (Akgül et al., 3 Mar 2025).

In optimistic policy improvement, exploration appears as a bonus in $r^+(s, a) = r(s, a) + c_1 \hat{N}(s)$3 or loss estimates. OPPO uses an optimistic $r^+(s, a) = r(s, a) + c_1 \hat{N}(s)$4 estimate,

$r^+(s, a) = r(s, a) + c_1 \hat{N}(s)$5

and updates the policy by exponential tilting,

$r^+(s, a) = r(s, a) + c_1 \hat{N}(s)$6

yielding a provably efficient optimistic version of PPO (Cai et al., 2019). The related dilated-bonus framework defines a recursive bonus

$r^+(s, a) = r(s, a) + c_1 \hat{N}(s)$7

which is subtracted from estimated losses in the policy update to induce global exploration (Luo et al., 2021).

In trajectory-level or rollout-level methods, exploration is enforced by controlling which trajectories are sampled or how entire histories are weighted. HAEPO sets

$r^+(s, a) = r(s, a) + c_1 \hat{N}(s)$8

then optimizes a reward-weighted, entropy-regularized, KL-regularized trajectory objective (Trivedi et al., 26 Aug 2025). The specific EEPO framework for RLVR instead alters the rollout process itself through adaptive unlearning of already sampled responses (Chen et al., 7 Oct 2025).

4. The specific RLVR framework titled EEPO

The paper explicitly titled “Exploration-Enhanced Policy Optimization via Sample-Then-Forget” defines EEPO as a two-stage rollout procedure for RL with verifiable rewards for LLMs (Chen et al., 7 Oct 2025). Its starting point is the claim that conventional adjustments such as raising temperature, adding entropy bonuses, or increasing rollout count do not sufficiently shift probability mass away from dominant modes. The framework therefore targets rollout generation rather than only the update objective.

The procedure is as follows. In Stage 1, a rollout model $r^+(s, a) = r(s, a) + c_1 \hat{N}(s)$9 samples half of the trajectories. A lightweight unlearning step is then applied to temporarily suppress these sampled responses in the rollout model. In Stage 2, the remaining trajectories are sampled from the modified rollout model, which is now discouraged from regenerating the same responses (Chen et al., 7 Oct 2025). The main policy is then updated with the usual GRPO objective using all collected trajectories.

Unlearning is gated by entropy. It is triggered only when a moving average of rollout entropy falls below a threshold: $$Q^*_{f,t} = r_t + \gamma V_\phi(s_{t+1}), \qquad Q^*_{b,t} = \hat r(s_t, a_t) + \gamma V_\phi(\hat T(s_t, a_t)),$$0 The paper introduces a complementary unlearning loss,

$$Q^*_{f,t} = r_t + \gamma V_\phi(s_{t+1}), \qquad Q^*_{b,t} = \hat r(s_t, a_t) + \gamma V_\phi(\hat T(s_t, a_t)),$$1

implemented with clipped probabilities for numerical stability, so that high-probability sampled tokens are penalized more heavily (Chen et al., 7 Oct 2025). The batch unlearning objective is

$$Q^*_{f,t} = r_t + \gamma V_\phi(s_{t+1}), \qquad Q^*_{b,t} = \hat r(s_t, a_t) + \gamma V_\phi(\hat T(s_t, a_t)),$$2

A defining feature is that the unlearning step is temporary and confined to the rollout model. The rollout model is reset to the current policy parameters at the start of each training iteration, so the forgetting effect does not accumulate across iterations (Chen et al., 7 Oct 2025). This design is intended to break the self-reinforcing loop in which dominant responses are repeatedly sampled and rewarded.

Empirically, the method is reported to outperform GRPO across five reasoning benchmarks, with average relative gains of 24.3% on Qwen2.5-3B, 33.0% on Llama3.2-3B-Instruct, and 10.4% on Qwen3-8B-Base (Chen et al., 7 Oct 2025). The paper further states that EEPO maintains higher rollout entropy, that Stage 2 rollouts have higher entropy than Stage 1 rollouts, and that the method introduces negligible overhead relative to GRPO (Chen et al., 7 Oct 2025). These results position the named EEPO framework as a rollout-diversification method rather than a conventional entropy-bonus method.

5. Theoretical interpretations and guarantees

Several EEPO variants are motivated not only empirically but also through explicit regret or convergence analyses. OPPO proves $$Q^*_{f,t} = r_t + \gamma V_\phi(s_{t+1}), \qquad Q^*_{b,t} = \hat r(s_t, a_t) + \gamma V_\phi(\hat T(s_t, a_t)),$$3 regret in episodic MDPs with linear function approximation, unknown transition, and adversarial reward with full-information feedback, and is described as the first provably efficient policy optimization algorithm that explores (Cai et al., 2019). The dilated-bonus framework improves policy optimization in adversarial MDPs to $$Q^*_{f,t} = r_t + \gamma V_\phi(s_{t+1}), \qquad Q^*_{b,t} = \hat r(s_t, a_t) + \gamma V_\phi(\hat T(s_t, a_t)),$$4 regret in the tabular case and to $$Q^*_{f,t} = r_t + \gamma V_\phi(s_{t+1}), \qquad Q^*_{b,t} = \hat r(s_t, a_t) + \gamma V_\phi(\hat T(s_t, a_t)),$$5 in a linear MDP setting without a simulator, while removing the need for an exploratory policy in other linear settings (Luo et al., 2021).

EE-Politex provides a different theoretical route. It modifies POLITEX so that a single sufficiently exploring policy $$Q^*_{f,t} = r_t + \gamma V_\phi(s_{t+1}), \qquad Q^*_{b,t} = \hat r(s_t, a_t) + \gamma V_\phi(\hat T(s_t, a_t)),$$6 is available beforehand, replacing the earlier requirement that all policies sufficiently explore the environment. The resulting regret bound is

$$Q^*_{f,t} = r_t + \gamma V_\phi(s_{t+1}), \qquad Q^*_{b,t} = \hat r(s_t, a_t) + \gamma V_\phi(\hat T(s_t, a_t)),$$7

which the paper emphasizes as the first result showing how to control regret in the presence of function approximation errors on problems where exploration is nontrivial (1908.10479). This suggests a modular interpretation of EEPO in which exploration and control can be partially decoupled.

ERSAC provides another formalization by endowing the agent with an epistemic-risk-seeking utility function,

$$Q^*_{f,t} = r_t + \gamma V_\phi(s_{t+1}), \qquad Q^*_{b,t} = \hat r(s_t, a_t) + \gamma V_\phi(\hat T(s_t, a_t)),$$8

and optimizing a zero-sum game between the policy and the risk-seeking parameter $$Q^*_{f,t} = r_t + \gamma V_\phi(s_{t+1}), \qquad Q^*_{b,t} = \hat r(s_t, a_t) + \gamma V_\phi(\hat T(s_t, a_t)),$$9 (O'Donoghue, 2023). The paper states that the saddle-point solution minimizes an upper bound on regret and yields efficient exploration even under function approximation.

For more recent LLM-oriented methods, the dominant theoretical theme is entropy control rather than classical regret. DCPO argues that entropy collapse is governed by the distribution itself rather than individual samples, and therefore regularizes the policy at the distribution level (Li et al., 19 Jan 2026). EPO for multi-turn sparse-reward LLM agents states that its entropy smoothing regularizer guarantees monotonically decreasing entropy variance while maintaining convergence (Xu et al., 26 Sep 2025). HAEPO states that listwise normalization and centralized score-function gradients provide low-variance, unbiased updates (Trivedi et al., 26 Aug 2025). These analyses indicate a shift in EEPO theory from optimism and regret bounds toward entropy stability and rollout-distribution control in sequence models.

6. Applications, empirical patterns, and recurrent controversies

EEPO methods have been applied across robotic control, Atari, MuJoCo, GUI grounding, sparse-reward exploration, and RL for LLMs. In robotics, TD3 with EECL on the robosuite panda lift task is reported to achieve consistently higher returns, faster convergence, and more robust performance across multiple random seeds than baseline TD3 (Hsieh et al., 2024). In continuous control, IEM-PPO is reported to outperform PPO and ICM-PPO on HalfCheetah-v2, Swimmer-v2, Hopper-v2, and Walker2d-v2, with better sample efficiency, cumulative reward, stability, and robustness, albeit with longer training time (Zhang et al., 2020). In non-stationary control, EPPO outperforms PPO and PFO on slippery and paralysis settings by preserving critic plasticity and directing exploration through uncertainty (Akgül et al., 3 Mar 2025).

In game-like or sparse-reward benchmarks, POME outperforms PPO on 33 out of 49 Atari 2600 games (Pan et al., 2018). EPO improves policy quality and sample efficiency relative to standard policy-gradient and evolutionary baselines on Atari Pong and Breakout, particularly where sparse rewards make global exploration valuable (Mustafaoglu et al., 17 Apr 2025). MPPO significantly outperforms state-of-the-art exploration methods in sample efficiency and final performance on MuJoCo tasks and their sparsified variants (Pan et al., 2019). The uncertainty-guided tree-search framework that explicitly decouples exploration from policy optimization explores an order of magnitude more efficiently than standard intrinsic-motivation baselines on hard Atari benchmarks, then distills discovered trajectories into deployable policies (Mhammedi et al., 23 Mar 2026).

In multimodal or language-agent settings, AEPO improves GUI grounding by using multi-answer generation and an efficiency-based Adaptive Exploration Reward,

$\epsilon_t = | Q^*_{b,t} - Q^*_{f,t} |,$0

and reports relative improvements of up to 9.0% against naive RLVR baselines on GUI grounding benchmarks (Liu et al., 7 Aug 2025). LLM-Explorer, though not a policy-optimization objective in the narrow sense, acts as a plug-in exploration module compatible with DQN, DDPG, TD3, and variants, and reports average performance improvement up to 37.27% on Atari and MuJoCo benchmarks (Hao et al., 21 May 2025). HAEPO, DCPO, EPO, and EEPO address long-horizon or sparse-reward LLM training by operating on trajectory likelihoods, entropy regularization, or rollout diversification (Trivedi et al., 26 Aug 2025, Li et al., 19 Jan 2026, Xu et al., 26 Sep 2025, Chen et al., 7 Oct 2025).

A recurring controversy concerns whether exploration should be induced by sample-level bonuses, entropy bonuses, population diversity, or direct control of the rollout or policy distribution. DCPO explicitly criticizes sample-centric fixes for depending on the “luck” of informative samples and argues for a distribution-centric perspective (Li et al., 19 Jan 2026). The EEPO sample-then-forget method similarly argues that simply increasing stochasticity does not escape dominant behavioral modes (Chen et al., 7 Oct 2025). Conversely, methods based on intrinsic rewards or optimism retain the view that carefully designed bonuses remain sufficient when uncertainty or novelty is well estimated (Pan et al., 2018, Zhang et al., 2020, Cai et al., 2019). Another recurring issue is computational overhead: IEM-PPO requires longer training time, AEPO and LLM-Explorer introduce additional inference or prompting costs, whereas EEPO reports negligible overhead and MPPO reports less than 5% overhead with $\epsilon_t = | Q^*_{b,t} - Q^*_{f,t} |,$1 (Zhang et al., 2020, Liu et al., 7 Aug 2025, Hao et al., 21 May 2025, Pan et al., 2019).

7. Relation to adjacent paradigms and open directions

EEPO overlaps with, but is not identical to, curiosity-driven RL, maximum-entropy RL, optimism in the face of uncertainty, evolutionary RL, and decoupled search. Curiosity and intrinsic-reward methods such as ICM-PPO and EECL use reward shaping to value novelty (Hsieh et al., 2024, Zhang et al., 2020). Maximum-entropy exploration is pursued explicitly in EVE, which optimizes the entropy of the steady-state visitation distribution without explicit rollouts and uses dominant eigenvectors of a tilted transition matrix (Adamczyk et al., 12 Mar 2026). Optimistic policy optimization appears in OPPO and in dilated-bonus methods (Cai et al., 2019, Luo et al., 2021). Evolutionary mechanisms appear in EPO and POEM (Mustafaoglu et al., 17 Apr 2025, Czworkowski et al., 21 Jan 2026). Decoupled exploration is taken furthest by uncertainty-guided tree search, which bypasses RL during exploration and postpones policy optimization to a later distillation phase (Mhammedi et al., 23 Mar 2026).

The literature also raises a persistent design question: whether exploration and exploitation should be jointly optimized within one policy update or separated into different phases, models, or distributions. EE-Politex and the tree-search framework both provide arguments for modularity (1908.10479, Mhammedi et al., 23 Mar 2026). In contrast, DCPO, HAEPO, EPPO, and EPO pursue integrated objectives that keep exploration inside the policy-optimization loop through entropy or distributional regularization (Li et al., 19 Jan 2026, Trivedi et al., 26 Aug 2025, Akgül et al., 3 Mar 2025, Xu et al., 26 Sep 2025). A plausible implication is that future EEPO work will continue to differentiate between exploration as a property of data collection and exploration as a property of the learned policy distribution.

Across these strands, EEPO can be understood as the systematic effort to make policy optimization explore on purpose rather than by incidental stochasticity. The concrete implementation varies—novel-state rewards, discrepancy bonuses, uncertainty-aware critics, population buffers, evolutionary mutations, entropy smoothing, distribution-level regularization, multi-answer generation, or sample-then-forget rollouts—but the common target remains the same: broader, more informative coverage of the decision space without forfeiting stable policy improvement (Hsieh et al., 2024, Pan et al., 2018, Chen et al., 7 Oct 2025).