--- title: Exploration-Enhanced Policy Optimization (EEPO) url: https://www.emergentmind.com/topics/exploration-enhanced-policy-optimization-eepo type: topic --- # Exploration-Enhanced Policy Optimization (EEPO) Searching arXiv for the cited EEPO-related papers to ground the article in current arXiv records. Exploration-Enhanced Policy Optimization (EEPO) denotes a line of reinforcement learning methods that augment policy optimization with explicit mechanisms for broader or more directed exploration. In the usage reflected across recent work, the term can refer both to a general design pattern—policy optimization coupled to intrinsic rewards, uncertainty bonuses, population diversity, model-based discrepancies, entropy control, or trajectory-level diversification—and to a specific RLVR framework for large language models that uses a “sample-then-forget” procedure to disrupt self-reinforcing rollout modes [1811.07350], [2510.05837]. Across these formulations, the central objective is to mitigate premature convergence, entropy collapse, local optima, or sparse-reward stagnation by altering either the reward signal, the advantage estimate, the rollout distribution, or the policy update itself. ## 1. Conceptual scope and problem setting EEPO arises from a common diagnosis: standard policy optimization methods are often exploitation-dominant. In actor-critic control, insufficient exploration of the spatial space can yield suboptimal policies, as observed for Twin Delayed Deep Deterministic policy gradient (TD3) on 7-DOF robotic arm grasping [2408.14009]. In on-policy optimization, PPO is described as suffering from limited exploration, premature convergence, or sensitivity to Gaussian exploration schedules [2011.05525], [2601.14705]. In RL for LLMs, GRPO is characterized as exploitation driven, with entropy decreasing monotonically and exploration fading [2601.12730]. In RLVR, repeated sampling and rewarding of dominant modes produces a self-reinforcing loop that erodes exploratory capacity [2510.05837]. Within this literature, EEPO methods do not replace policy optimization; they modify it so that the training signal better values uncertainty, novelty, diversity, or entropy. This suggests a unifying interpretation in which exploration is treated not as an external heuristic layered on top of policy learning, but as a first-class component of the optimization objective or of rollout generation itself. A plausible implication is that EEPO is best viewed as a family of policy-optimization augmentations rather than a single algorithmic template. Representative instantiations span several regimes. POME adds a model-based/model-free target discrepancy to PPO as an exploration value [1811.07350]. IEM-PPO adds an uncertainty-estimation intrinsic reward to PPO [2011.05525]. MPPO maintains a population of policies and selects among them using a combined performance-entropy rule [1911.04207]. OPPO and related optimistic methods insert bonus terms directly into policy improvement with regret guarantees [1912.05830], [2107.08346]. In RL for LLMs, DCPO, HAEPO, EPO, AEPO, and the specific EEPO framework all alter entropy control, trajectory weighting, or rollout diversification to preserve exploration under long-horizon or sparse-reward conditions [2601.12730], [2508.18884], [2509.22576], [2508.05731], [2510.05837]. ## 2. Mechanistic families of exploration enhancement The literature organizes naturally into several mechanism classes. The following grouping is a synthesis of named methods in the cited works. | Mechanism family | Representative method | Core exploration signal | |---|---|---| | Intrinsic novelty reward | TD3 + EECL | Novel-state reward from KDTree Euclidean distance | | Uncertainty-based reward or advantage | IEM-PPO, EPPO | Uncertainty bonus or UCB advantage | | Model-based discrepancy bonus | POME | $|Q^*_{b,t} - Q^*_{f,t}|$ | | Population or evolutionary diversity | MPPO, EPO, POEM | Policy entropy, mutation, or population diversity | | Entropy or distribution regulation | DCPO, EPO, HAEPO | Distribution-level entropy control or trajectory entropy | | Rollout diversification | EEPO, AEPO | Sample-then-forget or multi-answer generation | One family uses intrinsic rewards for novelty. EECL stores previously explored states in a buffer, organizes them with a KDTree, detects novelty through Euclidean distance, and adds a decaying exploration reward to the environment reward before TD3 updates [2408.14009]. IEM-PPO similarly reshapes PPO with an uncertainty-based intrinsic reward, where the mixed reward is $r^+(s, a) = r(s, a) + c_1 \hat{N}(s)$ [2011.05525]. A second family derives exploration from uncertainty in value estimation or dynamics. POME computes both model-free and model-based targets, \[ Q^*_{f,t} = r_t + \gamma V_\phi(s_{t+1}), \qquad Q^*_{b,t} = \hat r(s_t, a_t) + \gamma V_\phi(\hat T(s_t, a_t)), \] and defines the exploration value as \[ \epsilon_t = | Q^*_{b,t} - Q^*_{f,t} |, \] which is added to the TD-style advantage used in PPO [1811.07350]. EPPO uses an evidential critic whose predictive variance induces a UCB-style advantage, \[ \hat{A}_t^{\mathrm{UCB}} = \mathbb{E}[\hat{A}_t^{\text{GAE}}] + \kappa \sqrt{\mathrm{Var}[\hat{A}_t^{\text{GAE}}]}, \] so that high-uncertainty regions are preferentially explored in non-stationary control [2503.01468]. A third family introduces diversity structurally. MPPO maintains a buffer of $K$ policies and scores each by \[ f_k(\pi_t) = (1 - \alpha)\hat{J}_k(\pi_t) + \alpha \hat{\mathcal{H}}_k(\pi_t), \] thereby selecting policies that jointly balance performance and entropy [1911.04207]. EPO hybridizes neuroevolution with PPO-style exploitation, using crossover, adaptive mutation, and local fine-tuning [2504.12568]. POEM triggers policy-parameter mutation when KL divergence to a moving-average policy falls below a threshold, signaling stagnation [2601.14705]. A fourth family operates at the distribution or trajectory level. HAEPO compresses each trajectory into a cumulative logarithmic likelihood and normalizes trajectories with a Plackett-Luce softmax, then adds entropy regularization and a soft KL penalty [2508.18884]. DCPO regularizes the policy at the distribution level rather than through sample-centric heuristics, using a higher-temperature target distribution and double importance sampling [2601.12730]. EPO for multi-turn LLM agents regulates trajectory-aware entropy, adds an entropy smoothing regularizer, and modulates the effect through adaptive phase-based weighting [2509.22576]. ## 3. Canonical mathematical formulations Despite their diversity, EEPO methods repeatedly modify one of four mathematical objects: the reward, the advantage, the policy update, or the rollout distribution. In reward shaping approaches, exploration enters as an additive intrinsic term. EECL detects a novel state when \[ \|s - s'\|_2 \geq \epsilon, \] and assigns an exponentially decaying exploration reward \[ r_e = r_{\text{max}} \times \gamma^n, \] which is then stored in the replay buffer as part of $(s, a, r + r_e, s')$ [2408.14009]. EIPO formulates the use of intrinsic rewards as a constrained optimization problem rather than a fixed scalar mixture: \[ \max_{\pi_{E+I} \in \Pi} J_{E+I}(\pi_{E+I}) \quad \text{subject to} \quad J_E(\pi_{E+I}) - \max_{\pi_E \in \Pi} J_E(\pi_E) = 0, \] with a Lagrangian multiplier $\alpha$ that automatically increases or suppresses the weight on extrinsic reward as needed [2211.07627]. In advantage shaping, the exploration signal perturbs the policy gradient target more directly. POME defines \[ \text{POME}_t = \delta_t + \alpha\,\text{clip}(\epsilon_t - \bar\epsilon, -|\delta_t|, |\delta_t|), \] and uses the resulting modified advantage inside the PPO surrogate loss [1811.07350]. EPPO modifies the advantage through uncertainty propagation from an evidential value model [2503.01468]. In optimistic policy improvement, exploration appears as a bonus in $Q$ or loss estimates. OPPO uses an optimistic $Q$ estimate, \[ Q_h^k(x, a) = \min\left\{ r_h^k(x, a) + \phi_h^k(x, a)^\top w_h^k + \Gamma_h^k(x, a),\ H-h+1 \right\}^+, \] and updates the policy by exponential tilting, \[ \pi_h^k(\cdot \mid x) \propto \pi_h^{k-1}(\cdot \mid x)\, \exp\left(\alpha Q_h^{k-1}(x,\cdot)\right), \] yielding a provably efficient optimistic version of PPO [1912.05830]. The related dilated-bonus framework defines a recursive bonus \[ B_t(x, a) = b_t(x, a) + \left(1 + \frac{1}{H}\right)\mathbb{E}[B_t(x', a')], \] which is subtracted from estimated losses in the policy update to induce global exploration [2107.08346]. In trajectory-level or rollout-level methods, exploration is enforced by controlling which trajectories are sampled or how entire histories are weighted. HAEPO sets \[ L_k = \sum_{t=1}^{T_k} \log \pi_\theta(a_t^{(k)} | s_t^{(k)}), \qquad w_k = \frac{\exp(L_k)}{\sum_{j=1}^M \exp(L_j)}, \] then optimizes a reward-weighted, entropy-regularized, KL-regularized trajectory objective [2508.18884]. The specific EEPO framework for RLVR instead alters the rollout process itself through adaptive unlearning of already sampled responses [2510.05837]. ## 4. The specific RLVR framework titled EEPO The paper explicitly titled “Exploration-Enhanced Policy Optimization via Sample-Then-Forget” defines EEPO as a two-stage rollout procedure for RL with verifiable rewards for LLMs [2510.05837]. Its starting point is the claim that conventional adjustments such as raising temperature, adding entropy bonuses, or increasing rollout count do not sufficiently shift probability mass away from dominant modes. The framework therefore targets rollout generation rather than only the update objective. The procedure is as follows. In Stage 1, a rollout model $\pi_{\text{rollout}}$ samples half of the trajectories. A lightweight unlearning step is then applied to temporarily suppress these sampled responses in the rollout model. In Stage 2, the remaining trajectories are sampled from the modified rollout model, which is now discouraged from regenerating the same responses [2510.05837]. The main policy is then updated with the usual GRPO objective using all collected trajectories. Unlearning is gated by entropy. It is triggered only when a moving average of rollout entropy falls below a threshold: \[ \mathbb{I}_t = \mathbb{I}\left[\overline{\mathcal{H}_t^{(m)}} < \alpha\right]. \] The paper introduces a complementary unlearning loss, \[ \mathcal{L}_{\text{Comp}} = \log \left(1 - \pi_{\text{rollout}}(o_{k,t} \mid q, o_{k,