Perception-Exploration Policy Optimization

• Perception-Exploration Policy Optimization (PEPO) is a family of approaches that train policies to acquire informative evidence through active perception and exploration rather than merely maximizing task rewards.
• PEPO methods encompass active perception in HMMs, embodied exploration, token-level optimization in multimodal chain-of-thoughts, and parallel exploration for LLM-based agents, each tailoring evidence acquisition to specific contexts.
• Empirical evaluations show that PEPO techniques improve model performance by optimizing metrics such as conditional entropy, visual grounding accuracy, and diversity of state transitions.

Searching arXiv for papers on “Perception-Exploration Policy Optimization” and closely related formulations to ground the article in current literature. arXiv search query: "Perception-Exploration Policy Optimization multimodal chain-of-thought PEPO" Perception-Exploration Policy Optimization (PEPO) denotes an emerging family of policy optimization formulations in which the policy is trained to improve perception quality, information acquisition, or environment understanding rather than task reward alone. In current arXiv usage, the label covers several technically distinct settings: active perception in hidden Markov models with controllable emissions, embodied exploration for collecting informative perception data, token-level reinforcement learning for multimodal chain-of-thought in large vision-LLMs, and diverse parallel exploration for LLM-based agents (Shi et al., 2024, Jing et al., 2023, Li et al., 24 Mar 2026, Zhang et al., 27 Apr 2026). Across these settings, the common structural pattern is that the policy controls what evidence is acquired—sensor queries, viewpoints, reasoning tokens, or parallel environment branches—and optimization is driven by perception-derived criteria such as conditional entropy, semantic disagreement, token visual grounding, uncertainty, or diversity of state transitions.

1. Scope, terminology, and naming ambiguity

The term is not yet standardized to a single algorithmic template. In "Rethinking Token-Level Policy Optimization for Multimodal Chain-of-Thought" (Li et al., 24 Mar 2026), PEPO is an explicit token-level RL framework for large vision-LLMs. In "Active Perception with Initial-State Uncertainty: A Policy Gradient Method" (Shi et al., 2024), the method is presented as a concrete instance of a broader PEPO perspective: uncontrolled dynamics, controllable emissions, and an information-theoretic objective on a latent variable. In "Learning to Explore Informative Trajectories and Samples for Embodied Perception" (Jing et al., 2023), the authors do not use the term, but the method is described as an archetype of PEPO in practice. "DPEPO: Diverse Parallel Exploration Policy Optimization for LLM-based Agents" (Zhang et al., 27 Apr 2026) makes the family interpretation explicit and specializes it to diverse parallel exploration.

A recurrent source of confusion is acronym overload. The identifier PEPO is also used in "Provably avoiding over-optimization in Direct Preference Optimization without knowing the data distribution" (Barla et al., 5 Feb 2026), where it stands for Pessimistic Ensemble based Preference Optimization, not Perception-Exploration Policy Optimization. That preference-learning method is offline, DPO-like, and centered on pessimistic ensemble aggregation rather than perception-driven exploration. The overlap is terminological rather than conceptual.

A second important distinction is methodological. In PEPO-style formulations, the optimized signal is not necessarily an additive task reward. In the active-perception formulation, the objective is the conditional entropy of the initial state given the full observation-action history, and Proposition 1 shows that this quantity cannot be represented as a sum of per-time-step belief rewards (Shi et al., 2024). In the multimodal LVLM setting, the problem is not sparse exploration in environment state space but fine-grained credit assignment over reasoning tokens, where uniform sequence-level advantages obscure which tokens are visually grounded or exploratory (Li et al., 24 Mar 2026). This suggests that PEPO is better understood as a design principle for coupling evidence acquisition and policy optimization than as a single canonical loss.

2. Information-theoretic active perception as a canonical formulation

A mathematically explicit PEPO formulation appears in the HMM setting of controllable emissions. The system is modeled as

$\mathcal{M}=\langle S,O,A,P,E,S_0\rangle,$

where $S$ is the hidden state space, $O$ the observation alphabet, $A$ a perception or sensor-query action set, $P$ an uncontrolled transition kernel, and $E(o\mid s,a)$ a controllable emission function. The perception agent chooses $A_t$ based on past observations but does not affect system dynamics. The inference target is the initial hidden state $S_0$, and the planning horizon $T$ is finite (Shi et al., 2024).

The objective is the Shannon conditional entropy

$$H(S_0\mid Y_{0:T};\pi) = -\sum_{s_0\in\mathcal{S}_0}\sum_{y_{0:T}} P^\pi(s_0,y_{0:T})\log P^\pi(s_0\mid y_{0:T}),$$

where $S$0 bundles observations and perception actions. Minimizing this objective is equivalent to maximizing mutual information $S$1, since $S$2 is fixed by the prior $S$3. The formulation therefore casts active perception as direct optimization of information leakage about a latent variable rather than as state control or reward maximization in the usual POMDP sense (Shi et al., 2024).

The main technical contribution is a policy-gradient identity that separates inference from exploration. Using observable operators

$S$4

the method computes sequence probabilities by matrix products and derives

$S$5

A further result shows that for any trajectory $S$6 with $S$7,

$S$8

so posterior computation for a fixed trajectory is independent of policy parameters in the derivative sense. The final gradient becomes

$S$9

which is REINFORCE-like, but with posterior entropy as a trajectory-level signal. The paper gives Lipschitz continuity and Lipschitz smoothness results under bounded first and second derivatives of $O$0, yielding convergence to stationary points under standard gradient-based assumptions (Shi et al., 2024).

This formulation supplies one of the clearest PEPO abstractions: exploration changes the distribution of evidence, while inference over a given evidence sequence remains a fixed model-based computation. A plausible implication is that PEPO objectives are naturally expressed at the trajectory level whenever the target latent variable is not the current state but a hidden type, intent, parameter, or initial mode.

3. Embodied perception and informative trajectory collection

In embodied perception, PEPO takes the form of learning navigation or exploration behavior that improves the training data presented to a perception model. "Learning to Explore Informative Trajectories and Samples for Embodied Perception" (Jing et al., 2023) considers an RGB-D agent in indoor 3D scenes with actions $O$1, a Mask R-CNN with FPN and ResNet-50 backbone pre-trained on COCO, and a 3D semantic distribution map $O$2 whose channels store obstacle flags, explored flags, and semantic class distributions.

The exploration policy is trained with PPO, but its reward is entirely perception-derived. The map is updated by exponential moving average fusion

$O$3

with $O$4, so multi-view predictions form a soft pseudo ground truth. Two intrinsic rewards are then defined. The semantic distribution disagreement reward uses KL divergence between the current voxel predictions $O$5 and the previous map $O$6,

$O$7

and the semantic distribution uncertainty reward is based on the second-largest class probability $O$8, thresholded as

$O$9

with default $A$0. The exploration objective is

$A$1

The method couples exploration to downstream supervised perception optimization through hard sample selection. Images are retained only when at least one object has uncertainty above threshold; entropy-based uncertainty

$A$2

is also evaluated and gives slightly better results in ablation. The resulting loop alternates between policy improvement and perception-model fine-tuning, which the paper explicitly describes as progressively training the exploration policy for $A$3 rounds (Jing et al., 2023).

Although the paper does not use the PEPO name, it is structurally aligned with it: the exploration policy is optimized using perception outputs, disagreement with a spatial world model, and uncertainty-driven sample valuation. This suggests a broader PEPO interpretation in which the policy need not optimize inference directly, as in the HMM case, but can instead optimize the data distribution on which perception learning is performed.

4. Token-level PEPO in multimodal chain-of-thought

In multimodal RL from verifiable rewards, PEPO is formulated as token-level redistribution of a sequence-level advantage according to two signals: perceptual grounding and exploratory uncertainty. The baseline setting is GRPO or DAPO, in which a sequence-level advantage

$A$4

is shared uniformly across all tokens in a sampled response. "Rethinking Token-Level Policy Optimization for Multimodal Chain-of-Thought" (Li et al., 24 Mar 2026) argues that this coarse granularity is particularly damaging for large vision-LLMs because some tokens are visually critical, some are exploratory branch points, and many are low-impact fillers.

The perceptual signal is token visual similarity,

$A$5

where $A$6 is the hidden state of response token $A$7 at layer $A$8 and $A$9 is the hidden state of vision token $P$0. The exploratory signal is token entropy

$P$1

After per-response min-max normalization to $P$2, the method forms a centered joint score

$P$3

then computes token weights

$P$4

with $P$5, so average token weight is 1. Sequence-level advantages are then redistributed as

$P$6

Three design points are central. First, perception remains dominant because high entropy cannot produce large weight when visual similarity is near zero. Second, the schedule interpolates from ordinary sequence-level RL to fully token-weighted RL, which improves stability. Third, the total advantage mass per sequence is preserved, so PEPO changes credit allocation without changing the global scale of policy gradients (Li et al., 24 Mar 2026).

A closely related line is "Not All Tokens See Equally: Perception-Grounded Policy Optimization for Large Vision-LLMs" (Ye et al., 2 Apr 2026), which replaces hidden-state similarity with Token Visual Dependency, defined as

$P$7

computed by comparing a visually conditioned forward pass with a text-only pass induced by attention masking. PGPO then applies threshold-gated, mass-conserving advantage reshaping to amplify visually dependent tokens and suppress visually irrelevant ones. The paper presents PGPO as a perception-grounded method rather than PEPO, but it explicitly frames PEPO as a natural extension that would add exploration mechanisms on top of perception-grounded token weighting (Ye et al., 2 Apr 2026).

5. Parallel exploration PEPO for LLM-based agents

In LLM agents acting in textual environments, PEPO has been specialized to multi-environment branching. DPEPO instantiates a parallel exploration paradigm in which the agent interacts with $P$8 environments simultaneously, all initialized from the same task state. At each step it selects a subset $P$9, executes a parallel action set

$E(o\mid s,a)$0

and observes the resulting parallel states

$E(o\mid s,a)$1

The full trajectory is therefore a sequence of sets, $E(o\mid s,a)$2, rather than a single linear rollout (Zhang et al., 27 Apr 2026).

The optimization procedure has two stages: cold-start supervised fine-tuning for parallel reasoning and structured action formatting, followed by GRPO-style reinforcement learning with hierarchical rewards. The trajectory-level success reward is

$E(o\mid s,a)$3

so the whole parallel rollout succeeds if any environment reaches a goal state. To encourage exploration breadth, DPEPO adds two step-level rewards: Diverse Action Reward, which penalizes repeated actions within an environment and duplicated actions across environments, and Diverse State Transition Reward, which penalizes repeated state-action transitions in depth and width. These are combined as

$E(o\mid s,a)$4

Step-level diversity is then modulated by group-relative trajectory success through

$E(o\mid s,a)$5

The policy update uses the same PPO-like clipped objective as GRPO but with these hierarchical advantages. The paper interprets this directly as PEPO: richer parallel observations improve the LLM’s internal environmental understanding, while diversity rewards optimize how the agent explores alternative branches (Zhang et al., 27 Apr 2026).

This variant differs from multimodal token-level PEPO in locus of control. In the LVLM setting, the policy redistributes learning pressure over tokens in a fixed response. In DPEPO, the policy actively chooses multiple external interaction branches whose observations are shared in-context across parallel trajectories. A plausible implication is that PEPO spans both credit assignment over perceptual computation and action selection for gathering broader environmental evidence.

6. Empirical profile, misconceptions, and open directions

Empirically, PEPO-style methods have been reported to improve both information acquisition and downstream task performance. In the HMM active-perception setting, the entropy-minimization policy gradient converges to a policy with conditional entropy approximately $E(o\mid s,a)$6 bits; by time $E(o\mid s,a)$7, posterior mass on the true type reaches approximately $E(o\mid s,a)$8, $E(o\mid s,a)$9, and $A_t$0 for the three robot types, compared with weaker identifiability under the best random policy (Shi et al., 2024). In embodied perception, the EITS exploration policy reaches detection AP50 $A_t$1 at $A_t$2 and $A_t$3 at $A_t$4, compared with $A_t$5 for Semantic Curiosity, while real-robot fine-tuning with just 170 hard-labeled samples improves detection AP50 from $A_t$6 to $A_t$7 and segmentation AP50 from $A_t$8 to $A_t$9 (Jing et al., 2023). In multimodal CoT RL, PEPO improves Qwen2.5-VL-3B geometry-and-logic average accuracy from $S_0$0 under GRPO to $S_0$1, and improves visual grounding average IoU@50 from $S_0$2 to $S_0$3; entropy-only baselines can collapse in visual grounding and few-shot classification (Li et al., 24 Mar 2026). In agentic text environments, DPEPO reports $S_0$4 average success on ALFWorld and $S_0$5 average on ScienceWorld, outperforming sequential baselines while remaining competitive in efficiency (Zhang et al., 27 Apr 2026).

Several misconceptions are explicitly contradicted by these results. PEPO is not simply entropy regularization: in multimodal reasoning, entropy alone is described as language-centric and can be unstable, whereas combining it with a perception prior is the central design principle (Li et al., 24 Mar 2026). PEPO is not equivalent to standard belief-reward POMDP planning: the conditional entropy objective over the initial state cannot, in general, be written as a sum of per-step belief rewards (Shi et al., 2024). Nor is PEPO restricted to external sensing actions; token-level PEPO treats reasoning tokens themselves as policy actions whose advantages should reflect perceptual grounding (Li et al., 24 Mar 2026).

The open problems identified across the literature are comparatively consistent. The HMM formulation points to model uncertainty, full POMDP control where actions affect both transition and emission, and cost-aware perception as immediate extensions (Shi et al., 2024). Embodied perception work identifies dependence on accurate depth and pose, static perception models during exploration, and the labeling burden of segmentation masks (Jing et al., 2023). Token-level LVLM PEPO remains evaluated primarily on 2B–3B models and image-based tasks, with video, tool use, and larger architectures left open (Li et al., 24 Mar 2026). DPEPO depends on parallelizable environments and sufficiently large context windows, which may be straightforward in simulators but not in physical settings (Zhang et al., 27 Apr 2026). Taken together, these constraints suggest that PEPO is presently a methodological direction rather than a closed theory: its strongest current forms are domain-specific, but they share a stable core idea—policy optimization should target the acquisition and allocation of perceptually useful evidence, not merely the maximization of undifferentiated return.