Interaction-Perceptive Agentic Policy Optimization

• IPA is a framework that integrates structured perception, theory-of-mind inference, reflective expertise, and chunk-based credit assignment to enhance agentic LLM optimization.
• It employs modular components and LLM-driven reflection to dynamically update policies, reducing gradient variance and improving long-horizon task stability.
• Empirical evaluations in imperfect-information games and agentic crafting tasks demonstrate IPA’s superior sample efficiency and strategic adaptability.

Interaction-Perceptive Agentic Policy Optimization (IPA) encompasses a class of frameworks and algorithms for optimizing agentic LLMs in interactive, multi-turn environments. IPA extends traditional RL-style and prompt-based policies by integrating high-level perception, theory of mind (ToM), reflective correction, and, in algorithmic variants, chunk-level credit assignment for stability in long-horizon tasks. IPA has been instantiated in both human-aligned cognitive architectures (e.g., PolicyEvol-Agent for imperfect-information games) and high-throughput open-agentic learning ecosystems (e.g., the ROME model in ALE), demonstrating stable and human-like adaptive learning in interactive domains (Yu et al., 20 Apr 2025, Wang et al., 31 Dec 2025).

1. Architectural Principles and System Modules

IPA systems are characterized by modular cognitive components, tightly coupled by policy feedback loops that allow for continual policy evolution:

• Perception (Observation Interpretation): Environmental state is mapped via LLMs and domain-specific rules into structured, human-interpretable prompts serving as the perceptual substrate for downstream cognition. In PolicyEvol-Agent, raw state vectors (such as cards and chip stacks) are rendered as textual descriptions (Obs), enabling subsequent reasoning and plan generation (Yu et al., 20 Apr 2025).
• Theory of Mind (ToM): ToM-driven inference is incorporated in belief generation and policy updates. The LLM, prompted appropriately, derives both the opponent’s intent (BeliefEnv) and the agent’s own inferred behavioral pattern (BeliefSelf), conditioning on historical trajectory and the current policy (Yu et al., 20 Apr 2025).
• Reflective Expertise (Policy Evolution): Post-episode, the agent reflects on empirical action distributions, quantifies divergence from its prior policy, and prompts the LLM (with ToM-prompts) to revise its policy in light of both its own and the opponent’s behavior patterns (Yu et al., 20 Apr 2025).
• Policy Evolution (Dynamic Guideline Adjustment): The outcome of reflective expertise is a normalization step yielding a new policy, which becomes the baseline for subsequent rounds. The process iterates over entire gameplay or agentic lifecycles.

These principles are instantiated in frameworks such as the PolicyEvol-Agent and the ALE/ROME ecosystem, with variants in how perception, reflection, and policy update are realized (Yu et al., 20 Apr 2025, Wang et al., 31 Dec 2025).

2. Formal Mathematical Formulations

IPA adopts both distribution-level and gradient-based optimization objectives tailored to multi-turn, interactive agent settings:

• Observation Mapping:

$$z_t^i = f_p\bigl(o_t^i\bigr) = LLM_\theta\bigl(\text{Prompt},\,\{x_1,\dots,x_J\}\bigr)$$

generates Obs for step $t$ as an autoregressive LLM process over low-level observables.

• Belief Inference (ToM):

$$\varphi_t^{Env} = g_{ToM}^{Env}(z_t^i, H_t, P_{old}),\quad \varphi_t^{Self} = g_{ToM}^{Self}(z_t^i, H_t, \varphi_t^{Env}, P_{old})$$

where $H_t$ is action-observation history.

• Reflective Policy Update:

$$P_{detect}(a,c) = \frac{\#\{(a,c) \text{ in History}\}}{\#\{c \text{ in History}\}}$$

and, after reflection,

$$P_{new}(a|c) \approx \frac{P(a,c\,|\,\text{Reflection})}{P(c\,|\,\text{History})}$$

with $P(a,c\,|\,\text{Reflection})$ inferred via ToM-enabled LLM queries.

• Objective (Augmented Return):

$$\pi^{(k+1)} = \arg\max_{\pi}\mathbb{E}_{\tau\sim\pi} \left[ \sum_{t=0}^T r_t + \alpha\,\mathcal{D}_{ToM}\bigl(\pi,\varphi_t\bigr) \right ]$$

where $\mathcal{D}_{ToM}$ measures policy-ToM divergence.

• Policy Evolution Dynamics: When cast as evolutionary updates (non-gradient, but analogous),

$\psi_{k+1} = \psi_k \oplus \Delta_{Reflect}$

where $\psi$ denotes parameters (prompts, policy counts), updated by the LLM-derived difference between $P_{old}$ and $P_{new}$.

Under chunk-based RL (as in the ALE/ROME IPA algorithm), credit is assigned at the level of “interaction chunks”:

• Chunk-Discounted Return:

$G_k = \gamma^{K-k} R_{\mathrm{final}}$

• Chunk IS Ratio:

$$\rho(c) = \left( \prod_{t\in c} \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{old}}(a_t|s_t)} \right )^{1/|c|}$$

• Combined Gradient Estimate:

$$\nabla_\theta J_{\mathrm{IPA}}(\pi_\theta) = \sum_{c\in\mathcal{T}^+} \mu_{old}(c)\,G_c\, \sum_{t\in c} m_c \nabla_\theta\log\pi_\theta(a_t|s_t) + \sum_{c\in\mathcal{T}^-} \mu_{old}(c)[\rho(c)]_0^1\,G_c\,\sum_{t\in c} m_c\nabla_\theta\log\pi_\theta(a_t|s_t)$$

with masking $m_c$ and importance clipping.

3. Algorithmic Workflow and Implementation

A canonical IPA loop, whether reflective (PolicyEvol-Agent) or chunk-based (ALE/ROME), follows sequential perceptual, cognitive, and evolutionary stages:

1. Initialization: Uniform or expert-bootstrapped policy $P_{old}$ or model weights $\theta$.
2. Perception: Extraction of state $o_t$ and LLM-based conversion to Obs $z_t$.
3. Belief Generation (ToM): Sequential inference of environmental and self-patterns via ToM-oriented prompts.
4. Plan Recommendation: LLM proposes action plans, typically with win-rate or success estimates.
5. Action Execution: Agent selects the optimal plan, updates state.
6. Reflection and Policy Evolution: After each trajectory, perform reflective expertise update; in chunk-based IPA, segment a trajectory, compute chunk-level returns and IS ratios, apply masked/weighted gradient updates.
7. Policy Update: Normalize or update policy parameters according to LLM guidance or gradient steps; synchronize with historical baseline.
8. Iteration: The process continues over multiple episodes, enabling adaptive, continual policy evolution.

In reflective frameworks, the process is non-gradient and driven by LLM text outputs, whereas in chunk-based IPA, formal gradient updates are performed over grouped token sequences corresponding to atomic environment actions (Yu et al., 20 Apr 2025, Wang et al., 31 Dec 2025).

4. Empirical Evaluation and Benchmarking

IPA-based systems have been evaluated in both simulated gaming environments and open-ended trajectory-based RL scenarios.

Environment	Baselines	Main Results (vs. Baseline)
Leduc Hold’em	NFSP, DQN, DMC, CFR, Suspicion-Agent	+123 (NFSP), +21 (DQN), +12 (DMC), +41 (Susp.-Agent)
Agentic crafting (ROME)	PPO, token-level RL	+15 absolute validation, 80% train success vs. 30% baseline, 5× reduction in gradient variance

Ablation studies show that plan recommendation and belief generation are critical in PolicyEvol-Agent—omitting these degrades performance most severely. Evolution phase analyses reveal progression from conservative, adaptively aggressive, to mastery-like policy behaviors. Behavioral breakdowns (action-position patterns) demonstrate learned bluffing and risk management.

In chunk-based agentic learning, chunk-level RL resolves instability, high variance, and low sample efficiency typical of token-level RL for long-horizon, multi-turn scenarios; parallelized chunk resampling is essential, as its removal reduces final test success by 30% (Yu et al., 20 Apr 2025, Wang et al., 31 Dec 2025).

5. Chunk-Level Interaction in Agentic RL

The core insight of chunk-level IPA is formalizing trajectories as sequences of interaction “chunks”—variable-length token subsequences terminating in atomic environment-altering actions (e.g., tool calls, shell commands). Assignment of reward and credit at this semantically coherent granularity addresses three major challenges in token-level RL for open-agent LLMs:

• Credit Assignment: Aligns update signals with actual environment-affecting actions, not interleaved “reasoning” tokens.
• Variance Reduction: Dramatically lowers the variance of gradient estimates and IS weights; masking and clipping further stabilize updates.
• Horizon Consistency: Chunk-based discounting compensates for the otherwise extreme vanishing of $\gamma^T$ in token-wise updates, concentrating learning on success-proximal decisions (Wang et al., 31 Dec 2025).

IPA further incorporates trajectory resampling, imitation learning on prefilled expert chunks, and data composition protocols (deterministic test-curated RL tasks, multi-filtered trajectory pipelines).

6. Analysis of Strengths, Limitations, and Future Directions

IPA, by integrating perception, ToM inference, reflectivity, and chunked RL, achieves adaptive and robust interactive policy learning:

• Strengths:
  • Human-interpretable cognition chaining via structured perception and reflection.
  • Multifaceted belief states over both opponents and self, facilitating nuanced strategic adaptation.
  • Empirically superior sample efficiency, rapid learning, and win-rate dominance on standard imperfect-information and agentic crafting tasks.
  • Systematic ablation and phase analyses revealing the criticality of plan-level reasoning and evolutionary correction (Yu et al., 20 Apr 2025, Wang et al., 31 Dec 2025).
• Limitations:
  • Current instantiations mostly address two-agent, confrontational or single-threaded problems; multi-agent, real-time, cooperative/competitive settings remain open.
  • The underlying LLM’s architecture and scale crucially affect reasoning accuracy and computational tractability.
  • Reflective and chunk-based updates operate in distinct regimes; unification via differentiable, gradient-based prompt-tuning is suggested as future work.
• Future Directions:
  • Extension to multi-agent, mixed-motive interaction domains.
  • Exploring various LLM designs for cost-quality trade-offs.
  • Developing differentiable IPA frameworks for direct gradient-based fine-tuning at the chunk/prompt level, bridging self-reflective LLM reasoning with classical RL gradients (Yu et al., 20 Apr 2025).

7. Comparative Insights and Significance

IPA marks a methodological advance over prior RL and prompt-based approaches for agentic LLMs by combining cognitive modularity, reflective adaptation, and, in ALE/ROME, robust chunk-based credit assignment. The net impact includes a 5× reduction in gradient variance and a 2× acceleration to reach 50% success in long-horizon tasks, alongside empirical mastery in nontrivial imperfect-information games (Wang et al., 31 Dec 2025). IPA thus operationalizes agentic learning in open worlds, aligning LLM actions with both empirical evidence and theory-of-mind-driven beliefs, facilitating continual, data-efficient strategic improvement of agentic LLMs.