Agent Explorative Policy Optimization for Multimodal Agentic Reasoning

Abstract: Vision-LLMs with extended reasoning succeed on complex problems, but many real-world problems require external tools that internal reasoning alone often cannot resolve. Agentic reasoning therefore interleaves two behaviors with a structural asymmetry: thinking (the self-contained default) and tool use (a high-variance auxiliary acting). We refer to this asymmetry as the Thinking-Acting Gap. Under standard RL recipes like GRPO, the gap manifests as two diagnostic symptoms during training: tool use is attempted on only ~30% of rollouts, and when attempted, the tool-using rollouts within a group are all-wrong on ~40% of questions, suppressing the learning signal at the tool calls that needed it. We propose AXPO (Agent eXplorative Policy Optimization): for each all-wrong tool-using subgroup, AXPO fixes the thinking prefix and resamples the tool call and its continuation, paired with uncertainty-based prefix selection. Across nine multimodal benchmarks and three scales of Qwen3-VL-Thinking, SFT+AXPO outperforms SFT+GRPO at average (+1.8pp Pass@1 and +1.8pp Pass@4 at 8B on average) and 8B with SFT+AXPO surpasses the 32B Base on Pass@4 with 4 times fewer parameters.

• The paper introduces AXPO, a reinforcement learning algorithm that targets the Thinking-Acting Gap by resampling at tool-call boundaries.
• AXPO fixes the reasoning prefix and selectively resamples only the tool call and its continuation, leading to significant improvements in Pass@1 and Pass@4 metrics.
• Empirical results demonstrate that AXPO attains superior performance with fewer parameters, confirming enhanced tool-use efficiency and robust agentic behavior.

Agent Explorative Policy Optimization for Multimodal Agentic Reasoning

Motivation: The Thinking-Acting Gap in Agentic Multimodal RL

This work investigates agentic reasoning in vision-LLMs (VLMs) that interleave multi-step internal reasoning with external tool use, such as Python execution, web search, and image manipulation. A structural challenge emerges in training such agentic models: there is a sharp asymmetry between the well-internalized “thinking” behaviors (token prediction within the model manifold) and the fragile, high-variance “acting” behaviors required for successful tool interaction. The authors define this as the Thinking-Acting Gap, which is empirically characterized by two distinct pathologies under standard RL (GRPO): (i) tool use appears in only ~30% of rollouts, and (ii) entire subgroups of tool-using rollouts are frequently all-wrong (~40% of cases), leading to the collapse of the learning signal at tool-call positions.

The AXPO Algorithm: Targeted Exploration at Tool Calls

The paper introduces Agent eXplorative Policy Optimization (AXPO), a reinforcement learning algorithm that directly addresses the Thinking-Acting Gap by concentrating exploration at the tool call transition. Rather than spending rollout budget on the entire trajectory, AXPO fixes the “thinking” prefix up to the first tool call for all-wrong tool-using groups and resamples only the tool call and its continuation. This strategic resampling is triggered only in cases where all tool-using rollouts fail, and is guided by an uncertainty-driven prefix ranking, measured by low confidence or high entropy at the tool call.

Figure 1: AXPO focuses exploration by anchoring the (failed) thinking prefix and resampling the tool call and its continuation, creating direct positive learning signal on tool-use.

The key theoretical insight is that resampling at the tool-call boundary provably dominates from-scratch rollout sampling (for tool-using coverage) by removing the waste inherent when only a minority of trajectories invoke a tool. Thus, a fixed rollout budget through AXPO always yields higher coverage on correct tool-using rollouts.

Empirical Characterization of the Gap

A detailed empirical breakdown shows the severity and persistence of the Thinking-Acting Gap under existing RL methods:

• Under-attempted tool use: Only 20–35% of rollouts contain tool use, with no significant change during RL (see below).
• All-wrong tool-using subgroups: Among groups where tools are attempted, the proportion where all tool-using rollouts fail is ~40%—vs. ~25% for non-tool subgroups.
• Tool call variance: Resampling tool calls from a fixed reasoning prefix demonstrates substantial semantic diversity, indicating that the tool-call is a genuine divergence point for agentic exploration.

Figure 2: Distribution over tool-use attempt counts per question—tool use is persistently a minority behavior under GRPO.

These observations demonstrate that standard group-relative RL dynamics lead to persistent under-exploration of the key region (tool-call), suppressing gradient signal at the precise locus where agentic models struggle most.

Main Results: AXPO Narrows the Gap and Outperforms Strong Baselines

Across nine multimodal benchmarks (spanning mathematical reasoning, fine-grained perception, and multi-hop search) and three model scales (2B/4B/8B), AXPO consistently improves both first-attempt (Pass@1) and multi-attempt (Pass@4) performance. At 8B scale, AXPO not only surpasses standard SFT+GRPO baselines, but also outperforms a 32B base model’s Pass@4 score while using 4× fewer parameters. Gains concentrate on perception and search-intensive tasks, where tool use is essential.

Figure 3: AXPO narrows the agentic gap to a 4$\times$ larger baseline, outperforming both SFT+GRPO and scaling the 8B model above the 32B base on Pass@4.

Moreover, the authors show that the improvement is not simply due to higher compute budget—an extra 25% resampling budget via AXPO yields significantly larger gains than a 100% rollout increase to the baseline, confirming that AXPO’s benefit is specifically about focused exploration at tool calls.

Training Dynamics: Resampling Restores Learning Signal

During training, standard SFT+GRPO leaves the tool-use rate flat and the all-wrong subgroup rate high. In contrast, AXPO both increases use of tool calls and steadily reduces the all-wrong rate through targeted recovery—empirically validating the hypothesized mechanism.

Figure 4: AXPO reverses both diagnostic symptoms of the Thinking-Acting Gap (increasing tool-use rate and decreasing all-wrong subgroup rate), with measurable per-tool recovery.

At policy convergence, only AXPO expands both the tool-attempt rate and conditional Pass@1—while alternatives only trade one against the other or plateau.

Figure 5: Only AXPO expands both the tool-attempt rate and the quality of conditional Pass@1, indicating true improvement in agentic behavior.

On matched-tool evaluation (restricting to questions where both baseline and AXPO use the same tool), AXPO also improves conditional accuracy, demonstrating that the gains reflect actual improvement in tool-use quality, not just higher frequency.

Algorithmic Components and Ablations

AXPO combines three critical elements:

• Selective resampling: Branch only at all-wrong tool-using subgroups.
• Uncertainty-based prefix selection: Allocate resamples to low-confidence prefixes.
• Per-prefix advantage and recovery reward: Structure credit assignment such that the thinking prefix is rewarded whenever any resampled continuation is correct, avoiding gradient conflict.

Ablations confirm that each of these choices is necessary for full gains—removing any component (e.g., fixing the prefix, per-prefix credit) degrades both Pass@1 and Pass@4.

Implications and Theoretical Perspective

The implications are multifaceted:

• RL for agentic VLMs: AXPO exposes a canonical failure mode in RL for agentic models—learning signal collapse in high-variance action regimes—and introduces a method for systematically correcting it.
• Sample efficiency tradeoffs: By targeting stochasticity at the critical tool-call boundary, AXPO enables higher effective sample efficiency versus uniform rollout scaling or reward shaping.
• Generalization to new tools: AXPO’s induced tool-use policy exhibits some generalization to unseen tool schemas at inference, outperforming standard RL in first-attempt accuracy when a novel image-search tool is enabled.
• Future scaling: As tool variety and environment complexity in VLM-derived agents increase, targeted credit restoration and exploration techniques (in the spirit of AXPO) are likely to be necessary for robust scaling in agentic RL.

From a theoretical standpoint, the formal analysis leverages coverage arguments that generalize standard group-wise rollout theory to agentic settings, introducing prefix-anchored resampling as a means to asymptotically dominate uniform sampling for recovery of rare-but-essential behaviors.

Conclusion

This work provides a systematic characterization of the structural weaknesses in agentic RL for multimodal large models, identifies the Thinking-Acting Gap as a bottleneck for tool-integrated reasoning, and demonstrates that collapsing rollout variance at the tool-call boundary through explicit resampling (AXPO) yields strong and consistent gains. The policy optimization principles outlined here are general and are expected to influence future pipelines for highly capable, robustly acting language-vision agents that coordinate complex internal and external computation.

Figure 6: Model confidence and policy entropy at tool-call positions are strongly correlated, supporting the use of confidence as a proxy for uncertainty ranking in AXPO.

Reference:

"Agent Explorative Policy Optimization for Multimodal Agentic Reasoning" (2605.28774)