Visually-Guided Policy Optimization for Multimodal Reasoning

Abstract: Reinforcement learning with verifiable rewards (RLVR) has significantly advanced the reasoning ability of vision-LLMs (VLMs). However, the inherent text-dominated nature of VLMs often leads to insufficient visual faithfulness, characterized by sparse attention activation to visual tokens. More importantly, our empirical analysis reveals that temporal visual forgetting along reasoning steps exacerbates this deficiency. To bridge this gap, we propose Visually-Guided Policy Optimization (VGPO), a novel framework to reinforce visual focus during policy optimization. Specifically, VGPO initially introduces a Visual Attention Compensation mechanism that leverages visual similarity to localize and amplify visual cues, while progressively elevating visual expectations in later steps to counteract visual forgetting. Building on this mechanism, we implement a dual-grained advantage re-weighting strategy: the intra-trajectory level highlights tokens exhibiting relatively high visual activation, while the inter-trajectory level prioritizes trajectories demonstrating superior visual accumulation. Extensive experiments demonstrate that VGPO achieves better visual activation and superior performance in mathematical multimodal reasoning and visual-dependent tasks.

• The paper introduces VGPO, a novel RL framework that enhances visual focus for multimodal reasoning by compensating for temporal visual forgetting.
• It leverages a continuous Visual Focus Score and a linear Visual Attention Compensation mechanism to maintain accuracy during multi-step generative processes.
• VGPO achieves significant gains in performance, training stability, and data efficiency across benchmarks like MathVista and LogicVista.

Visually-Guided Policy Optimization for Multimodal Reasoning: An Expert Analysis

Motivation and Problem Characterization

Vision-LLMs (VLMs) have advanced multimodal reasoning capabilities but consistently display insufficient visual faithfulness, predominantly attending to linguistic signals over visual input. Empirical analyses reveal that not only do VLMs allocate sparse attention to image tokens, but this visual focus degrades temporally as reasoning progresses—a phenomenon the authors denote as "temporal visual forgetting." This results in brittle multimodal reasoning, where models hallucinate or lose fidelity to visual facts during multi-step generative processes (Figure 1).

Figure 1: Quantitative dissection of attention allocation reveals a pronounced bias towards textual tokens and substantial visual focus decay during extended reasoning, with correct answers correlating to sustained late-stage visual engagement.

Related Work and Limitations

Existing methods for addressing weak visual grounding typically leverage external interventions: auxiliary tokens prompting visual re-evaluation [yang2025lookback], additional models verifying visual cues mid-reasoning [tian2025VAPO], or KL-divergence-based perturbation detection to identify visually grounded spans [huang2025spotlight]. While these approaches can enhance visual activation, they incur non-trivial training, inference, or infrastructure overhead, and their reliance on extrinsics complicates scalability and reproducibility. The prevailing challenge thus centers on amplifying and sustaining visual focus using only intrinsic signals.

VGPO: Methodological Advances

The Visually-Guided Policy Optimization (VGPO) framework addresses these limitations through an internally consistent approach that impels robust visual grounding in multimodal RL agents. Key methodological contributions are as follows:

1. Visual Focus Score: VGPO capitalizes on the latent state manifold, operationalizing the cosine similarity between generated token embeddings and an image-token–derived “visual prototype” as a continuous visual focus score. This design permits accurate localization of visually-grounded tokens without auxiliary supervision.
2. Visual Attention Compensation (VAC): To actively counteract temporal visual forgetting, this mechanism dynamically scales the visual focus score upwards for later reasoning steps, effectively re-incentivizing visual tokens as the generation unfolds. The compensation schedule is linear, yielding maximal empirical benefits over step-function or exponential alternatives.
3. Dual-Grained Advantage Re-weighting: VAC is operationalized in RL via multiplicative modulation of the base trajectory advantage. At the intra-trajectory level, tokens with above-average visual focus (relative to their path) receive positive incentive, while intra-sample normalization attenuates text-dominated spans. At the inter-trajectory level, rollouts with globally superior visual focus are preferentially upweighted, promoting holistic visual engagement.

   Figure 2: VGPO architecture: Hidden-state similarity (left), attention compensation (center), and dual-scale advantage re-weighting (right) coalesce into a coherent training signal for visually-faithful policy optimization.

Experimental Results

The empirical evaluation encompasses a suite of multimodal reasoning and vision-dependent benchmarks (MathVista, MathVerse, MMMU-Pro, LogicVista, etc.), using the Qwen2.5-VL backbone at 3B, 7B, and 32B scales. Multiple baseline frameworks (GRPO, DAPO, PAPO, VPPO, NoisyRollout) are compared.

Key claims, substantiated by the results, include:

• Substantial performance gains: VGPO on Qwen2.5-VL-7B achieves 33.2% and 30.0% relative improvement on general mathematical and vision-dependent tasks, respectively, over the base model, and consistently matches or surpasses much larger models (e.g., Qwen2.5-VL-72B).
• Enhanced training stability and generalization: VGPO exhibits monotonic improvement in both training reward and validation accuracy, outperforming DAPO and GRPO baselines across model scales and dataset sizes (Figures 4, 7, 8).
• Data and parameter efficiency: VGPO retains its edge even in restricted data regimes, and the relative gains become more pronounced as the visual encoder capacity degrades, highlighting robustness.
• Stronger visual attention: A marked increase in visual attention ratio across benchmarks confirms sustained mitigation of visual forgetting (Figure 3).

Figure 4: VGPO yields more stable and consistently superior learning curves in training reward and validation metrics relative to leading RL baselines.

Figure 3: Distributional shift analysis demonstrates a global upregulation of vision attention ratios post-VGPO intervention across diverse benchmarks.

Ablation and Sensitivity Analyses

Extensive ablation validates the complementary dynamics of intra- and inter-trajectory re-weighting; omitting either harms or destabilizes improvement. Notably, the late-stage linear visual compensation schedule dominates alternatives, as early/naïve compensation can distract text parsing or promote over-attention to non-visual cue tokens.

Sensitivity experiments identify $\beta=0.3$, $\kappa=0.2$, and $\gamma=0.5$ as optimal, and training dynamics are robust within reasonable hyperparameter intervals, tightly coupling performance peaks with late/early visual activation equilibrium (Figure 5).

Figure 5: Late/early ratio and performance are maximized under a carefully calibrated linear compensation schedule.

Comparison With Other Advantage Shaping Methods

Contrasted against entropy-based and KL-based advantage modulation, VGPO's vision-guided shaping delivers higher accuracy, especially for vision-centric reasoning. Unlike prior methods, VGPO avoids reliance on multi-pass forward computation or external models, maintaining training efficiency.

Implications, Limitations, and Outlook

VGPO’s intrinsic, RL-based amplification of visual focus directly targets known bottlenecks in VLMs: text dominance and rapid visual forgetting. Its independence from external scaffolding renders it scalable, efficient, and adaptable to a broad set of deployed architectures. However, the upper bound of efficacy is set by the latent representational fidelity of the visual encoder; models incapable of initially encoding discriminative visual details cannot be retroactively “re-focused” by policy re-weighting alone.

Future development directions should prioritize:

• Context-adaptive visual compensation schedules, determining, in a content-aware fashion, when visual focus is critical versus when pure logical/textual reasoning predominates.
• Joint optimization of encoder representations and policy re-weighting, possibly in a meta-RL framework.
• Systematic integration with contrastive or generative self-supervised signals to construct unified “visual faithfulness” objectives.

Conclusion

The authors establish several critical empirical findings about VLMs’ temporal visual focus decay and proactively address them with a fully intrinsic, RL-driven framework. VGPO sets a reproducible baseline for research in visually faithful, long-horizon multimodal reasoning, achieving both SOTA performance and interpretability. Its approach suggests that future advances will increasingly meld fine-grained introspection with scalable policy optimization, pushing towards VLMs that are not only logically coherent but visually grounded throughout complex generative trajectories.

(2604.09349)