Spotlight on Token Perception for Multimodal Reinforcement Learning (2510.09285v1)

Abstract: While Reinforcement Learning with Verifiable Rewards (RLVR) has advanced the reasoning capabilities of Large Vision-LLMs (LVLMs), most existing methods in multimodal reasoning neglect the critical role of visual perception within the RLVR optimization process. In this paper, we undertake a pioneering exploration of multimodal RLVR through the novel perspective of token perception, which measures the visual dependency of each generated token. With a granular analysis of Chain-of-Thought (CoT) processes, we uncover two key insights: first, token perception in a rollout trajectory is sparsely distributed, where only a small fraction of tokens have high visual dependency for visually-grounded reasoning; second, different trajectories exhibit significant divergence in their overall visual dependency. Based on these observations, we propose Visually-Perceptive Policy Optimization (VPPO), a novel policy gradient algorithm that explicitly leverages token perception to refine the learning signal. Specifically, VPPO achieves this through a dual mechanism: it reweights a trajectory's advantage by its overall visual dependency, and focuses policy updates exclusively on perceptually pivotal tokens. On a comprehensive suite of eight perception and reasoning benchmarks, VPPO demonstrates substantial gains over leading open-source RL-tuned models, with its effectiveness consistently validated across 7B and 32B model scales. Our findings not only establish a new token-level perceptual perspective for analyzing multimodal RLVR but also present a novel and effective optimization strategy to significantly enhance the multimodal reasoning capabilities of LVLMs.

• The paper presents VPPO, a novel approach that leverages token-level visual dependency for targeted credit assignment in multimodal reinforcement learning.
• VPPO employs token-level gradient filtering and trajectory-level advantage shaping to reduce variance and enhance training stability.
• Empirical results demonstrate up to 19.2% improvement on 7B models and robust scalability to larger models, confirming its practical benefits.

Visually-Perceptive Policy Optimization: Token-Level Perception in Multimodal RL

Introduction

The paper "Spotlight on Token Perception for Multimodal Reinforcement Learning" (2510.09285) introduces Visually-Perceptive Policy Optimization (VPPO), a policy gradient algorithm for Large Vision-LLMs (LVLMs) that explicitly leverages token-level visual dependency to address the limitations of uniform reward broadcasting in multimodal RL with verifiable rewards (RLVR). The work is motivated by the observation that, in multimodal reasoning, only a sparse subset of tokens in a reasoning trajectory are genuinely visually grounded, and that not all correct trajectories are equally perception-driven. VPPO introduces a hierarchical signal modulation mechanism—trajectory-level advantage shaping and token-level gradient filtering—yielding substantial improvements in both accuracy and training stability across a suite of challenging multimodal reasoning benchmarks.

Motivation: The Role of Token Perception in Multimodal RL

Prevailing RLVR frameworks for LVLMs, such as GRPO and DAPO, apply a uniform, coarse reward signal to all tokens in a trajectory, regardless of their actual contribution to visually-grounded reasoning. This approach fails to distinguish between tokens that are pivotal for perception and those that are not, and it does not differentiate between trajectories that are genuinely perception-driven and those that reach correct answers via spurious shortcuts or linguistic priors. The result is a diluted learning signal, suboptimal convergence, and limited perceptual reasoning capabilities.

The authors empirically analyze the distribution of token-level visual dependency in Chain-of-Thought (CoT) reasoning and find that:

• Token-level visual dependency is highly sparse: Only a small fraction of tokens in a trajectory are strongly dependent on the visual input.
• Trajectory-level visual dependency is heterogeneous: Some trajectories are robustly perception-driven, while others are not, even if both yield correct answers.

These findings motivate a targeted optimization strategy that focuses the learning signal on perceptually pivotal tokens and robustly perception-driven trajectories.

VPPO: Methodology and Algorithmic Design

Quantifying Token Visual Dependency

Token visual dependency is defined as the KL divergence between the model's output distribution conditioned on the original image and a perturbed (masked) version:

$$\mathcal{S}(s_t, I) := D_{\mathrm{KL}}\left( \pi_\theta(\cdot | s_t, I) \parallel \pi_\theta(\cdot | s_t, I') \right)$$

where $s_t$ is the current state (prompt and previous tokens), $I$ is the original image, and $I'$ is a non-informative, perturbed image (e.g., via random patch blackening).

This metric robustly identifies tokens whose prediction is critically influenced by the visual context.

Figure 1: Overview of the VPPO framework, showing the computation of token-level visual dependency and its use in hierarchical signal modulation.

Hierarchical Signal Modulation

VPPO introduces two key mechanisms:

1. Token-level Gradient Filtering (TGF): For each trajectory, only the top-$k\%$ tokens with the highest visual dependency receive nonzero policy gradients. This focuses learning on the sparse set of visually-grounded reasoning steps, reducing gradient variance and combating signal dilution.
2. Trajectory-level Advantage Shaping (TAS): The advantage for each trajectory is reweighted by a normalized function of its mean token-level visual dependency. This amplifies updates for robustly perception-driven trajectories and dampens those for less visually-grounded ones.

The final VPPO objective is:

$$\mathcal{L}^{\mathrm{VPPO}}(\theta) = \mathbb{E} \left[ \frac{1}{G} \sum_{i=1}^{G} \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} m_{i,t} \cdot \min \left( r_{i,t}(\theta) \hat{A}'_i, \mathrm{clip}(r_{i,t}(\theta), 1-\varepsilon, 1+\varepsilon) \hat{A}'_i \right) \right]$$

where $m_{i,t}$ is the binary mask for pivotal tokens, and $\hat{A}'_i$ is the shaped advantage.

Theoretical Analysis

The authors provide a variance reduction analysis, showing that VPPO's estimator variance is reduced by a factor of $k \cdot \mathbb{E}[\alpha(\tau)^2]$ compared to the baseline, where $k$ is the sparsity ratio and $\alpha(\tau)$ is the trajectory shaping factor. This reduction is substantial in practice, leading to more stable and efficient training.

Implementation Details

• Dependency Calculation: KL divergence is computed between the output distributions for the original and masked images. Random patch blackening is used as the masking strategy, which aligns well with ViT-based architectures.
• Gradient Filtering Ratio: Empirically, $k=0.4$ (top 40% most visually-dependent tokens) yields optimal performance.
• Advantage Shaping Range: A dynamic upper bound with a conservative lower bound ($\beta_{\min}=0.9$) is used for normalization.
• Training Setup: Models are trained for two epochs on the ViRL39K dataset, with a rollout group size of 8, and a small entropy penalty (0.06) is applied to stabilize training.
• Computational Overhead: The method introduces a minor overhead (~10% increase in training time) due to the additional forward pass for dependency calculation.

Empirical Results

VPPO is evaluated on eight multimodal reasoning benchmarks, including MathVerse, DynaMath, Geo3k, MathVision, MMK12, We-Math, LogicVista, and MMMU-Pro. The method is tested on both 7B and 32B Qwen2.5-VL models.

• 7B Model: VPPO achieves a 19.2% average accuracy improvement over the baseline, outperforming all open-source RL-tuned models.
• 32B Model: VPPO yields a 7.6% average accuracy improvement, demonstrating robust scalability.
• Training Dynamics: VPPO exhibits faster convergence and greater training stability compared to baselines.

Figure 3: Training accuracy dynamics, showing faster and more stable convergence for VPPO compared to baselines.

Ablation and Qualitative Analysis

• Component Ablation: Both TGF and TAS individually improve performance, but their combination yields the best results, confirming the efficacy of the hierarchical design.
• Filtering Ratio Sensitivity: Performance peaks at $k=0.4$; too low or too high values degrade performance.
• Dependency Metric: KL divergence outperforms alternative heuristics (JSD, top-1 probability drop) for identifying visually-dependent tokens.
• Qualitative Examples: VPPO consistently focuses on the correct visually-grounded reasoning chain, while baselines often fail due to signal dilution.

Figure 4: The top 40% most visually-dependent tokens (highlighted in purple) form the core reasoning chain targeted by VPPO's gradient filtering.

Failure Modes and Regularization

Without regularization, baselines such as DAPO are prone to catastrophic policy collapse, degenerating into incoherent outputs due to the sparse, coarse reward signal. The entropy penalty is critical for stabilizing training, but VPPO's targeted signal modulation provides additional robustness.

Figure 2: Catastrophic policy collapse in the DAPO baseline when trained without regularization, illustrating the necessity of entropy regularization.

Implications and Future Directions

Practical Implications

• Fine-grained Credit Assignment: VPPO demonstrates that token-level perception metrics can be leveraged for more precise credit assignment in multimodal RL, leading to improved reasoning and perception capabilities.
• Plug-and-Play: The method is compatible with mainstream RLVR algorithms and can be integrated into existing LVLM training pipelines with minimal modification.
• Computational Efficiency: The modest overhead is justified by the substantial gains in accuracy and stability.

Theoretical Implications

• Variance Reduction: The hierarchical modulation of the learning signal provides a principled approach to variance reduction in policy gradient estimation for multimodal tasks.
• Perception-Reasoning Alignment: The work highlights the necessity of aligning the learning objective with perceptual grounding, rather than relying solely on outcome-based rewards.

Limitations and Future Work

• Scalability: While results are strong up to 32B models, efficacy on larger models (e.g., 72B+) remains to be validated.
• Generalization: The approach is demonstrated on reasoning-intensive tasks; its applicability to more subjective or creative multimodal tasks is an open question.
• Dependency Metric Optimization: Further research into more efficient or adaptive dependency metrics and masking strategies could further reduce computational overhead.

Conclusion

VPPO establishes a new paradigm for multimodal RL by explicitly modeling and leveraging token-level visual dependency. Its hierarchical signal modulation—combining trajectory-level advantage shaping and token-level gradient filtering—addresses the core bottleneck of uniform reward broadcasting, yielding state-of-the-art results on challenging multimodal reasoning benchmarks. The approach provides a robust, interpretable, and efficient framework for advancing the perceptual and reasoning capabilities of LVLMs, with broad implications for the design of future multimodal RL algorithms.