---
title: Perception-Exploration Policy Optimization
url: https://www.emergentmind.com/topics/perception-exploration-policy-optimization-pepo
type: topic
---

# Perception-Exploration Policy Optimization

Searching arXiv for recent papers on “Perception-Exploration Policy Optimization” and closely related formulations to ground the article in current literature.
arXiv search query: "Perception-Exploration Policy Optimization multimodal chain-of-thought PEPO"
Perception-Exploration Policy Optimization (PEPO) denotes an emerging family of policy optimization formulations in which the policy is trained to improve perception quality, information acquisition, or environment understanding rather than task reward alone. In current arXiv usage, the label covers several technically distinct settings: active perception in hidden Markov models with controllable emissions, embodied exploration for collecting informative perception data, token-level reinforcement learning for multimodal chain-of-thought in large vision-language models, and diverse parallel exploration for LLM-based agents [2409.16439][2303.10936][2603.22847][2604.24320]. Across these settings, the common structural pattern is that the policy controls what evidence is acquired—sensor queries, viewpoints, reasoning tokens, or parallel environment branches—and optimization is driven by perception-derived criteria such as conditional entropy, semantic disagreement, token visual grounding, uncertainty, or diversity of state transitions.

## 1. Scope, terminology, and naming ambiguity

The term is not yet standardized to a single algorithmic template. In "Rethinking Token-Level Policy Optimization for Multimodal Chain-of-Thought" [2603.22847], PEPO is an explicit token-level RL framework for large vision-language models. In "Active Perception with Initial-State Uncertainty: A Policy Gradient Method" [2409.16439], the method is presented as a concrete instance of a broader PEPO perspective: uncontrolled dynamics, controllable emissions, and an information-theoretic objective on a latent variable. In "Learning to Explore Informative Trajectories and Samples for Embodied Perception" [2303.10936], the authors do not use the term, but the method is described as an archetype of PEPO in practice. "DPEPO: Diverse Parallel Exploration Policy Optimization for LLM-based Agents" [2604.24320] makes the family interpretation explicit and specializes it to diverse parallel exploration.

A recurrent source of confusion is acronym overload. The identifier **PEPO** is also used in "Provably avoiding over-optimization in Direct Preference Optimization without knowing the data distribution" [2602.06239], where it stands for **Pessimistic Ensemble based Preference Optimization**, not Perception-Exploration Policy Optimization. That preference-learning method is offline, DPO-like, and centered on pessimistic ensemble aggregation rather than perception-driven exploration. The overlap is terminological rather than conceptual.

A second important distinction is methodological. In PEPO-style formulations, the optimized signal is not necessarily an additive task reward. In the active-perception formulation, the objective is the conditional entropy of the initial state given the full observation-action history, and Proposition 1 shows that this quantity cannot be represented as a sum of per-time-step belief rewards [2409.16439]. In the multimodal LVLM setting, the problem is not sparse exploration in environment state space but fine-grained credit assignment over reasoning tokens, where uniform sequence-level advantages obscure which tokens are visually grounded or exploratory [2603.22847]. This suggests that PEPO is better understood as a design principle for coupling evidence acquisition and policy optimization than as a single canonical loss.

## 2. Information-theoretic active perception as a canonical formulation

A mathematically explicit PEPO formulation appears in the HMM setting of controllable emissions. The system is modeled as  
\[
\mathcal{M}=\langle S,O,A,P,E,S_0\rangle,
\]
where \(S\) is the hidden state space, \(O\) the observation alphabet, \(A\) a perception or sensor-query action set, \(P\) an uncontrolled transition kernel, and \(E(o\mid s,a)\) a controllable emission function. The perception agent chooses \(A_t\) based on past observations but does not affect system dynamics. The inference target is the initial hidden state \(S_0\), and the planning horizon \(T\) is finite [2409.16439].

The objective is the Shannon conditional entropy
\[
H(S_0\mid Y_{0:T};\pi)
=
-\sum_{s_0\in\mathcal{S}_0}\sum_{y_{0:T}}
P^\pi(s_0,y_{0:T})\log P^\pi(s_0\mid y_{0:T}),
\]
where \(Y_{0:T}\) bundles observations and perception actions. Minimizing this objective is equivalent to maximizing mutual information \(I(S_0;Y_{0:T})\), since \(H(S_0)\) is fixed by the prior \(\mu_0\). The formulation therefore casts active perception as direct optimization of information leakage about a latent variable rather than as state control or reward maximization in the usual POMDP sense [2409.16439].

The main technical contribution is a policy-gradient identity that separates inference from exploration. Using observable operators
\[
W_{o\mid a}[i,j]=T_{i,j}\cdot O^a_{o,j},
\]
the method computes sequence probabilities by matrix products and derives
\[
\nabla_\theta \log P_\theta(y\mid s_0)
=
\nabla_\theta \log P_\theta(y)
=
\sum_{t=0}^{T}\nabla_\theta \log \pi_\theta(a_t\mid o_{0:t-1}).
\]
A further result shows that for any trajectory \(y\) with \(P_\theta(y)>0\),
\[
\nabla_\theta \log P_\theta(s_0\mid y)=0,
\]
so posterior computation for a fixed trajectory is independent of policy parameters in the derivative sense. The final gradient becomes
\[
\nabla_\theta H(S_0\mid Y;\theta)
=
\mathbb{E}_{y\sim M_\theta}
\left[
H(S_0\mid Y=y;\theta)\,\nabla_\theta \log P_\theta(y)
\right],
\]
which is REINFORCE-like, but with posterior entropy as a trajectory-level signal. The paper gives Lipschitz continuity and Lipschitz smoothness results under bounded first and second derivatives of \(\log \pi_\theta\), yielding convergence to stationary points under standard gradient-based assumptions [2409.16439].

This formulation supplies one of the clearest PEPO abstractions: exploration changes the distribution of evidence, while inference over a given evidence sequence remains a fixed model-based computation. A plausible implication is that PEPO objectives are naturally expressed at the trajectory level whenever the target latent variable is not the current state but a hidden type, intent, parameter, or initial mode.

## 3. Embodied perception and informative trajectory collection

In embodied perception, PEPO takes the form of learning navigation or exploration behavior that improves the training data presented to a perception model. "Learning to Explore Informative Trajectories and Samples for Embodied Perception" [2303.10936] considers an RGB-D agent in indoor 3D scenes with actions \(\{\text{move\_forward}, \text{turn\_left}, \text{turn\_right}\}\), a Mask R-CNN with FPN and ResNet-50 backbone pre-trained on COCO, and a 3D semantic distribution map \(M_t\in\mathbb{R}^{K\times L_M\times W_M\times H_M}\) whose channels store obstacle flags, explored flags, and semantic class distributions.

The exploration policy is trained with PPO, but its reward is entirely perception-derived. The map is updated by exponential moving average fusion
\[
M_t=
\begin{cases}
M_{t-1}, & t=1,\\
\lambda M_{t-1}+(1-\lambda)m_t, & t>1,
\end{cases}
\]
with \(\lambda=0.3\), so multi-view predictions form a soft pseudo ground truth. Two intrinsic rewards are then defined. The semantic distribution disagreement reward uses KL divergence between the current voxel predictions \(m_t\) and the previous map \(M_{t-1}\),
\[
r_d = KL(m_t,M_{t-1}),
\]
and the semantic distribution uncertainty reward is based on the second-largest class probability \(u_i=\mathrm{Second}_{\max}(P_i)\), thresholded as
\[
r_u=
\begin{cases}
1,& u>\delta,\\
0,& u<\delta,
\end{cases}
\]
with default \(\delta=0.1\). The exploration objective is
\[
\max_\pi J(\pi)=
\mathbb{E}_{\tau\sim\pi}
\left[
\sum_t \gamma^t (r_d(t)+r_u(t))
\right].
\]

The method couples exploration to downstream supervised perception optimization through hard sample selection. Images are retained only when at least one object has uncertainty above threshold; entropy-based uncertainty
\[
H(P_i)=-\sum_c P_i(c)\log P_i(c)
\]
is also evaluated and gives slightly better results in ablation. The resulting loop alternates between policy improvement and perception-model fine-tuning, which the paper explicitly describes as progressively training the exploration policy for \(n=1,2,3\) rounds [2303.10936].

Although the paper does not use the PEPO name, it is structurally aligned with it: the exploration policy is optimized using perception outputs, disagreement with a spatial world model, and uncertainty-driven sample valuation. This suggests a broader PEPO interpretation in which the policy need not optimize inference directly, as in the HMM case, but can instead optimize the data distribution on which perception learning is performed.

## 4. Token-level PEPO in multimodal chain-of-thought

In multimodal RL from verifiable rewards, PEPO is formulated as token-level redistribution of a sequence-level advantage according to two signals: perceptual grounding and exploratory uncertainty. The baseline setting is GRPO or DAPO, in which a sequence-level advantage
\[
A^{(i)}=
\frac{R^{(i)}-\mathrm{mean}_j R^{(j)}}{\mathrm{std}_j R^{(j)}}
\]
is shared uniformly across all tokens in a sampled response. "Rethinking Token-Level Policy Optimization for Multimodal Chain-of-Thought" [2603.22847] argues that this coarse granularity is particularly damaging for large vision-language models because some tokens are visually critical, some are exploratory branch points, and many are low-impact fillers.

The perceptual signal is token visual similarity,
\[
\mathrm{VS}_t
=
\frac{1}{LN}\sum_{l=1}^{L}\sum_{n=1}^{N}
\cos(h_{l,t},v_{l,n}),
\]
where \(h_{l,t}\) is the hidden state of response token \(t\) at layer \(l\) and \(v_{l,n}\) is the hidden state of vision token \(n\). The exploratory signal is token entropy
\[
H_t=-\sum_{x\in V} p_\theta(x\mid s_t)\log p_\theta(x\mid s_t).
\]
After per-response min-max normalization to \([0,1]\), the method forms a centered joint score
\[
g_t=\widetilde{\mathrm{VS}}_t+\widetilde{H}_t-\mathrm{mean}_u(\widetilde{\mathrm{VS}}_u+\widetilde{H}_u),
\]
then computes token weights
\[
w_t
=
T_i\cdot
\mathrm{Softmax}_t
\Big(
(1+\mathrm{atanh}(g_t))\cdot \widetilde{\mathrm{VS}}_t
\Big),
\]
with \(\sum_t w_t=T_i\), so average token weight is 1. Sequence-level advantages are then redistributed as
\[
A_t^{(i)}
=
\big[(1-\lambda_k)+\lambda_k w_t^{(i)}\big]A^{(i)},
\qquad
\lambda_k=\min(1,k/K_{\max}).
\]

Three design points are central. First, perception remains dominant because high entropy cannot produce large weight when visual similarity is near zero. Second, the schedule interpolates from ordinary sequence-level RL to fully token-weighted RL, which improves stability. Third, the total advantage mass per sequence is preserved, so PEPO changes credit allocation without changing the global scale of policy gradients [2603.22847].

A closely related line is "Not All Tokens See Equally: Perception-Grounded Policy Optimization for Large Vision-Language Models" [2604.01840], which replaces hidden-state similarity with **Token Visual Dependency**, defined as
\[
\mathcal{S}(s_t,I)
=
D_{\mathrm{KL}}
\big(
\pi_\theta(\cdot\mid s_t,I)
\;\|\;
\pi_\theta(\cdot\mid s_t)
\big),
\]
computed by comparing a visually conditioned forward pass with a text-only pass induced by attention masking. PGPO then applies threshold-gated, mass-conserving advantage reshaping to amplify visually dependent tokens and suppress visually irrelevant ones. The paper presents PGPO as a perception-grounded method rather than PEPO, but it explicitly frames PEPO as a natural extension that would add exploration mechanisms on top of perception-grounded token weighting [2604.01840].

## 5. Parallel exploration PEPO for LLM-based agents

In LLM agents acting in textual environments, PEPO has been specialized to multi-environment branching. DPEPO instantiates a parallel exploration paradigm in which the agent interacts with \(K\) environments simultaneously, all initialized from the same task state. At each step it selects a subset \(\mathcal{E}'_t\subseteq\mathcal{E}\), executes a parallel action set
\[
A_t=\{(E_i,a_t^{(i)})\mid E_i\in\mathcal{E}'_t\},
\]
and observes the resulting parallel states
\[
S_t=\{(E_i,s_t^{(i)})\mid E_i\in\mathcal{E}'_t\}.
\]
The full trajectory is therefore a sequence of sets, \(\tau=\{(S_t,A_t)\}_{t=1}^{T}\), rather than a single linear rollout [2604.24320].

The optimization procedure has two stages: cold-start supervised fine-tuning for parallel reasoning and structured action formatting, followed by GRPO-style reinforcement learning with hierarchical rewards. The trajectory-level success reward is
\[
R_{\mathrm{traj}}(\tau)=
\begin{cases}
1,& \text{if } S_T\cap\mathcal{G}\neq\emptyset,\\
0,& \text{otherwise},
\end{cases}
\]
so the whole parallel rollout succeeds if any environment reaches a goal state. To encourage exploration breadth, DPEPO adds two step-level rewards: Diverse Action Reward, which penalizes repeated actions within an environment and duplicated actions across environments, and Diverse State Transition Reward, which penalizes repeated state-action transitions in depth and width. These are combined as
\[
R_{\mathrm{step}}(A_t)
=
\frac{1}{2}\big(
R_{\mathrm{action}}(A_t)+R_{\mathrm{transition}}(A_t)
\big).
\]

Step-level diversity is then modulated by group-relative trajectory success through
\[
\Phi_{\mathrm{step}}(A_{i,t})
=
\begin{cases}
R_{\mathrm{step}}(A_{i,t}),& \Phi_{\mathrm{traj}}(\tau_i)>0,\\
2-R_{\mathrm{step}}(A_{i,t}),& \text{otherwise},
\end{cases}
\qquad
\Phi(A_{i,t})=\Phi_{\mathrm{step}}(A_{i,t})\cdot \Phi_{\mathrm{traj}}(\tau_i).
\]
The policy update uses the same PPO-like clipped objective as GRPO but with these hierarchical advantages. The paper interprets this directly as PEPO: richer parallel observations improve the LLM’s internal environmental understanding, while diversity rewards optimize how the agent explores alternative branches [2604.24320].

This variant differs from multimodal token-level PEPO in locus of control. In the LVLM setting, the policy redistributes learning pressure over tokens in a fixed response. In DPEPO, the policy actively chooses multiple external interaction branches whose observations are shared in-context across parallel trajectories. A plausible implication is that PEPO spans both **credit assignment over perceptual computation** and **action selection for gathering broader environmental evidence**.

## 6. Empirical profile, misconceptions, and open directions

Empirically, PEPO-style methods have been reported to improve both information acquisition and downstream task performance. In the HMM active-perception setting, the entropy-minimization policy gradient converges to a policy with conditional entropy approximately \(0.22\) bits; by time \(T=9\), posterior mass on the true type reaches approximately \(0.94\), \(0.96\), and \(1.00\) for the three robot types, compared with weaker identifiability under the best random policy [2409.16439]. In embodied perception, the EITS exploration policy reaches detection AP50 \(34.07\) at \(n=1\) and \(35.03\) at \(n=3\), compared with \(32.46\) for Semantic Curiosity, while real-robot fine-tuning with just 170 hard-labeled samples improves detection AP50 from \(79.1\%\) to \(97.3\%\) and segmentation AP50 from \(76.7\%\) to \(96.1\%\) [2303.10936]. In multimodal CoT RL, PEPO improves Qwen2.5-VL-3B geometry-and-logic average accuracy from \(32.64\) under GRPO to \(36.70\), and improves visual grounding average IoU@50 from \(82.60\) to \(83.46\); entropy-only baselines can collapse in visual grounding and few-shot classification [2603.22847]. In agentic text environments, DPEPO reports \(98.2\%\) average success on ALFWorld and \(61.4\%\) average on ScienceWorld, outperforming sequential baselines while remaining competitive in efficiency [2604.24320].

Several misconceptions are explicitly contradicted by these results. PEPO is not simply entropy regularization: in multimodal reasoning, entropy alone is described as language-centric and can be unstable, whereas combining it with a perception prior is the central design principle [2603.22847]. PEPO is not equivalent to standard belief-reward POMDP planning: the conditional entropy objective over the initial state cannot, in general, be written as a sum of per-step belief rewards [2409.16439]. Nor is PEPO restricted to external sensing actions; token-level PEPO treats reasoning tokens themselves as policy actions whose advantages should reflect perceptual grounding [2603.22847].

The open problems identified across the literature are comparatively consistent. The HMM formulation points to model uncertainty, full POMDP control where actions affect both transition and emission, and cost-aware perception as immediate extensions [2409.16439]. Embodied perception work identifies dependence on accurate depth and pose, static perception models during exploration, and the labeling burden of segmentation masks [2303.10936]. Token-level LVLM PEPO remains evaluated primarily on 2B–3B models and image-based tasks, with video, tool use, and larger architectures left open [2603.22847]. DPEPO depends on parallelizable environments and sufficiently large context windows, which may be straightforward in simulators but not in physical settings [2604.24320]. Taken together, these constraints suggest that PEPO is presently a methodological direction rather than a closed theory: its strongest current forms are domain-specific, but they share a stable core idea—policy optimization should target the acquisition and allocation of perceptually useful evidence, not merely the maximization of undifferentiated return.

Source: https://www.emergentmind.com/topics/perception-exploration-policy-optimization-pepo