PRPO: Perception-Reinforced Policy Optimization via Token-Level Dynamic Advantage Reshaping

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has become an effective paradigm for improving the reasoning capability of Large Vision-LLMs (LVLMs). However, existing RLVR methods primarily rely on trajectory-level outcome rewards, which assign identical learning signals across all generated tokens. This coarse-grained credit assignment is fundamentally mismatched to multimodal reasoning, where only a sparse subset of tokens is causally grounded in visual evidence. Consequently, these pivotal perceptual tokens receive weak supervision and are often overwhelmed by language priors or reasoning-template tokens. To address this limitation, we propose Perception-Reinforced Policy Optimization (PRPO), a token-level reinforcement learning framework that explicitly identifies and reinforces pivotal perceptual tokens within long-horizon multimodal reasoning trajectories. PRPO introduces Robust Visual Dependency (RVD), a principled metric that identifies tokens whose predictions are both visually grounded and perturbation-stable, filtering out brittle or noisy visual tokens. Based on RVD, we further propose Perceptual Advantage Reshaping (PAR), a token-level credit assignment technique that amplifies perceptually informative tokens while preserving stable gradients for non-perceptual tokens. Extensive experiments on seven multimodal reasoning benchmarks demonstrate that PRPO consistently outperforms strong LVLM baselines across both 3B and 7B model scales, achieving average gains of 23.3% and 21.1%, respectively. PRPO achieves state-of-the-art performance with improved training efficiency and stronger cross-task generalization. Our findings highlight the importance of fine-grained credit assignment for scalable multimodal reinforcement learning.

• The paper introduces PRPO, a token-level reinforcement learning framework that dynamically reshapes advantages based on robust visual signals.
• It employs a dual-metric strategy combining visual dependency and inconsistency metrics to isolate perceptually pivotal tokens from noisy anchors.
• Empirical results on multimodal benchmarks show significant improvements in accuracy and training efficiency compared to existing RL baselines.

Perception-Reinforced Policy Optimization for Precise Token-Level Multimodal Credit Assignment

Problem Statement and Motivation

Standard reinforcement learning from verifiable rewards (RLVR) has substantially advanced multimodal reasoning in Large Vision-LLMs (LVLMs), particularly in mathematical and logical domains. Conventional RLVR frameworks, however, predominantly assign rewards at the sequence (trajectory) level, indiscriminately broadcasting identical optimization signals to all tokens, regardless of their functional or perceptual significance. This trajectory-level credit assignment is fundamentally mismatched to the multimodal reasoning setting: within such tasks, only a sparse, causally pivotal subset of tokens are truly grounded in visual perception. Uniformly applied rewards lead to gradient dilution, where critical perceptual signals are overwhelmed by high-volume linguistic priors, impeding the reinforcement of key multimodal capabilities.

Error analysis on benchmarks such as MathVerse demonstrates that LVLM failures predominantly arise from perception errors—grounding on incorrect visual regions, misreading local evidence, and extracting erroneous values—rather than from pure logical missteps (Figure 1). These findings delineate a core perceptual credit assignment problem: critical perceptual tokens are not effectively differentiated or reinforced, thus constraining multimodal reasoning development.

Figure 1: Error analysis of multimodal reasoning failures using Qwen2.5-VL-7B on the MathVerse benchmark; perceptual bottlenecks, not reasoning, dominate observed failures.

PRPO Framework: Robust Visual Credit Assignment

To resolve the perceptual credit bottleneck, the paper introduces Perception-Reinforced Policy Optimization (PRPO), a token-level RL framework designed for dynamic credit assignment in multimodal reasoning. PRPO prioritizes the identification and targeted reinforcement of perceptually pivotal tokens via two core mechanisms:

1. Robust Visual Dependency (RVD): RVD is a dual-metric, trajectory-level diagnostic that isolates visually grounded and perturbation-stable tokens. It computes visual dependency ($S_t$) as the KL divergence between the original and strongly perturbed (information-corrupted) visual states, measuring a token's semantic reliance on visual perception. Visual inconsistency ($S'_t$) gauges stability via KL divergence under weak, information-preserving perturbation, filtering out tokens that respond with high variance to small input jitter. The RVD metric (Equation 1) combines these to admit only those tokens that are both visually grounded and stable to noise.
2. Perceptual Advantage Reshaping (PAR): To avoid gradient collapse and suboptimal learning dynamics caused by the heavy-tailed RVD distribution, PAR applies a nonlinear, synaptic S-shaped gating curve to the token-level advantages. This transformation amplifies the credit for strong, robust perceptual tokens while ensuring a nonzero gradient for less-perceptual tokens, preserving global reasoning capabilities while focusing optimization on visually critical regions.

   Figure 2: Overview of PRPO framework, consisting of RVD for pivotal perceptual token identification and PAR for nonlinear token-level credit assignment.

Analysis of Perceptual Token Sparsity and Dual-Metric Decomposition

Fine-grained analysis on long-horizon multimodal trajectories reveals visual signal sparsity and the intricate coupling of visual dependency and instability:

• Visual Dependency–Inconsistency Coupling: Most generated tokens cluster in low-VD, low-VI regions (linguistic tokens), while a small subset exhibits high VD but also high VI (brittle anchors)—visual signals entwined with unstable, noise-sensitive predictions. The true pivotal perceptual tokens, in contrast, are characterized by high VD and low VI, representing causal, stable visual grounding.

Figure 3: Distribution of Visual Dependency (VD) and Visual Inconsistency (VI) on Qwen2.5-VL-7B.

• Implication for Credit Assignment: The dual-metric approach is necessary to disentangle robust anchors from brittle ones; sole reliance on dependency over-amplifies noise, while the combined metric precisely filters and routes reinforcement.

Empirical Results

Experiments on seven challenging multimodal reasoning benchmarks (Geo3k, MathVista, We-Math, MMK12, MathVerse, LogicVista, MMMU-Pro) demonstrate that PRPO achieves consistent and substantial improvements over both trajectory-level and prior token-focused RL baselines (GRPO, DAPO, PAPO, VPPO etc.) across both 3B and 7B parameter scales:

• Performance Gains: Average improvements reach +23.3% (3B) and +21.1% (7B) over the base Qwen2.5-VL series, with PRPO outperforming the strongest prior baseline VPPO by +1.9% and +1.7% respectively at these scales.
• Training Efficiency: PRPO converges faster and to higher final accuracy within identical computational budgets, attributable to the high signal-to-noise ratio of the dynamic, targeted credit assignment provided by RVD and PAR.
• Cross-task Generalization: Performance gains are consistent across tasks and are robust to ablations of the RVD and PAR modules—both are individually essential for maximal benefit.

  Figure 4: Comparison of generation results between PRPO and baseline using Qwen2.5-VL-7B, illustrating more concise and accurate reasoning with PRPO.

Interpretability and Lexical Insights

Lexical analysis of identified pivotal tokens and brittle anchors—those with high VD and high VI—demonstrates substantial overlap in mathematical vocabulary, numerals, and geometric identifiers. This overlap reinforces the risk of single-metric approaches and underscores the necessity of the dual-metric filter.

Figure 5: Word cloud of Brittle Anchors (High VD, High VI), highlighting the semantic indistinguishability between robust and brittle perceptual tokens.

Implications and Future Directions

The PRPO paradigm shifts multimodal RL optimization from coarse, trajectory-wide credit diffusion to high-precision, token-level dynamic reinforcement. This evolution is likely to become an essential ingredient for further advances in scalable, robust multimodal reasoning, particularly as models are increasingly deployed in domains requiring reliable perceptual grounding—such as scientific diagram understanding, robotics, and visual mathematical cognition.

The dual-metric filtering approach can be potentially extended to other domains where brittle high-variance signals evade easy identification, such as medical imaging or fine-grained object tracking. Extensions to continual learning regimes, or integration with advanced retrieval-augmented architectures, offer further research opportunities.

Conclusion

PRPO delivers an effective solution to the perceptual credit assignment problem in LVLMs, establishing state-of-the-art results across a broad swath of multimodal reasoning benchmarks. By introducing Robust Visual Dependency for precise pivotal token identification and Perceptual Advantage Reshaping for stable, nonlinear credit allocation, PRPO achieves significant, reliable gains in both reasoning accuracy and training efficiency. Its principles and design offer a robust blueprint for future RL algorithms targeting scalable multimodal intelligence.

---

Reference:

"PRPO: Perception-Reinforced Policy Optimization via Token-Level Dynamic Advantage Reshaping" (2606.08708)