Improving Vision-language Models with Perception-centric Process Reward Models

Abstract: Recent advancements in reinforcement learning with verifiable rewards (RLVR) have significantly improved the complex reasoning ability of vision-LLMs (VLMs). However, its outcome-level supervision is too coarse to diagnose and correct errors within the reasoning chain. To this end, we propose Perceval, a process reward model (PRM) that enables token-level error grounding, which can extract image-related claims from the response and compare them one by one with the visual evidence in the image, ultimately returning claims that contain perceptual errors. Perceval is trained with perception-intensive supervised training data. We then integrate Perceval into the RL training process to train the policy models. Specifically, compared to traditional GRPO, which applies sequence-level advantages, we apply token-level advantages by targeting penalties on hallucinated spans identified by Perceval, thus enabling fine-grained supervision signals. In addition to augmenting the training process, Perceval can also assist VLMs during the inference stage. Using Perceval, we can truncate the erroneous portions of the model's response, and then either have the model regenerate the response directly or induce the model to reflect on its previous output. This process can be repeated multiple times to achieve test-time scaling. Experiments show significant improvements on benchmarks from various domains across multiple reasoning VLMs trained with RL, highlighting the promise of perception-centric supervision as a general-purpose strategy. For test-time scaling, it also demonstrates consistent performance gains over other strategies, such as major voting. Our code and data will be publicly released at https://github.com/RUCAIBox/Perceval.

• The paper presents Perceval, a process reward model that provides token-level error grounding to overcome reward sparsity in VLM training.
• It employs a think-then-answer paradigm to isolate and penalize hallucinated tokens, enabling precise credit assignment and robust visual grounding.
• Experimental results show significant improvements, with enhanced visual search accuracy and generalization across diverse reasoning tasks.

Improving Visual Reasoning in Vision-LLMs via Perception-Centric Process Reward Models

Motivation and Problem Statement

The paper "Improving Vision-LLMs with Perception-centric Process Reward Models" (2604.24583) addresses the critical limitation of reward sparsity in reinforcement learning with verifiable rewards (RLVR) for vision-LLMs (VLMs). Conventional RLVR, particularly using Group Relative Policy Optimization (GRPO), provides only sequence-level supervision: a single scalar reward assigned after each output sequence, irrespective of correctness at intermediate reasoning steps. This constrains the model’s learning, since perceptual or logical errors cannot be attributed to specific steps or tokens, causing a substantial credit assignment bottleneck in visual reasoning tasks. Furthermore, VLMs routinely suffer from hallucinations—generating object or spatial relation claims unsupported by the actual image—which are not efficiently suppressed by outcome-level supervision.

To overcome these issues, the authors propose a process reward model (PRM)—Perceval—that offers token-level error grounding. Perceval isolates image-related claims within generated responses, verifies their alignment with visual evidence, and flags hallucinated spans. This supports fine-grained supervision during RL, enabling targeted penalties that are critical for improving both perceptual grounding and robustness to reward hacking.

Perceval: Design and Training

Perceval operates as a perception-centric external critic. Given a tuple $(v,q,o)$ consisting of image, query, and model response, Perceval decomposes the output into image-related claims and applies a verification schema. Adopting the "think-then-answer" paradigm, it produces a reasoning rationale (> ...) evaluating each claim against the input image, followed by a structured answer (<answer>...</answer>) reporting erroneous spans.

The training of Perceval utilizes a multi-stage pipeline:

• Query selection: Emphasizes perception-intensive benchmarks such as visual search and referring expression grounding.
• Response generation: Employs VLMs to produce candidate responses, including realistic hallucinations via imperfect perceptual alignment.
• Annotation and verification: Utilizes strong LLMs (e.g., Gemini-2.5-Pro) to annotate hallucinations with structured counter-evidence.
• Supervised fine-tuning: Adapts Perceval via SFT to reliably flag perceptual errors and return structured feedback.

This schema supports automatic and scalable supervision without expensive manual annotation.

Process-Level Supervision in RL

The integration with RLVR modifies the standard GRPO algorithm by replacing the sequence-level advantage $\hat{A}_i$ with a token-level advantage $\hat{A}'_{i,t}$:

$$\hat{A}'_{i,t} \coloneqq \hat{A}_i - \alpha \cdot m_{i,t} \cdot |\hat{A}_i|$$

where $m_{i,t}$ is a binary mask indicating hallucinated tokens and $\alpha$ is a penalty-strength hyperparameter. Correct tokens inherit the sequence-level advantage, while hallucinated tokens are penalized proportionately. This facilitates precise credit assignment and suppresses the emergence of reward hacking.

Figure 1: Process-Supervised GRPO uses Perceval to generate token-level penalty masks integrated into the RL objective for targeted suppression of hallucinations.

Test-Time Scaling with Perceval

Beyond training, Perceval enables iterative inference-time refinement. When erroneous spans are detected, the response is truncated before the hallucinated token, and regeneration is triggered from the verified prefix. An enhanced strategy introduces a reflective prompt encouraging the model to reconsider its rationale in subsequent generations. These test-time scaling approaches consistently yield superior factual grounding compared to majority voting—where multiple responses are generated and the plurality is chosen.

Experimental Evaluation

Comprehensive experiments evaluate models on diverse visual reasoning benchmarks, encompassing visual search, perception-intensive reasoning, mathematical and chart-based reasoning, and general VQA. The baseline comparisons include both SFT-trained VLMs and multiple RLVR-enhanced architectures.

• The proposed method exhibits strong numerical improvements: for Qwen2.5-VL-3B, visual search accuracy increases from 80.10 (GRPO) to 83.25 (Ours); for Qwen2.5-VL-7B, performance reaches 86.39 (Ours), competitive with tool-augmented state-of-the-art models.
• Percival-based training demonstrates notable generalization: improvements in foundational perception propagate to other domains, including math reasoning and chart analysis, even though process-level supervision was applied only to visual grounding tasks.
• Reward hacking is mitigated: Perceval’s token-level intervention prevents policy models from overfitting to scalar reward functions, as illustrated by the stabilization in hallucination rates during training.

  Figure 2: The proportion of responses with hallucinations, as identified by Perceval, stabilizes during training, signaling effective mitigation of reward hacking.

Ablation studies reveal that an optimal penalty strength $\alpha=0.1$ provides maximum suppression of hallucinations without degrading overall sequence quality.

Qualitative analysis further underscores the improvement: models trained with Perceval-supported RLVR proceed with stepwise perceptual localization, contrasting with baseline models that skip visual grounding and issue unsupported claims.

Figure 3: Example visual reasoning from GRPO and PRM-trained models; PRM-trained models show explicit grounding and stepwise reasoning.

Practical and Theoretical Implications

The Perceval approach establishes that perception-centric process reward models—capable of fine-grained token-level verification anchored in visual evidence—constitute a robust solution to reward sparsity and reward hacking in RLVR for VLMs. The method’s improved performance across perception-intensive and general reasoning tasks attests to the principle that strong perceptual grounding is foundational for multimodal reasoning.

Practically, this framework obviates the need for manual step-level annotation, scales efficiently via automated annotation pipelines, and demonstrates efficacy both as a training intervention and for test-time scaling.

From a theoretical perspective, this strategy validates the utility of process-level supervision: intermediate, factually verifiable claims deserve explicit supervision for optimal learning of visual reasoning chains. The findings further suggest that broader advances in multimodal RL (including reward modeling and test-time scaling) should prioritize stepwise, grounded feedback mechanisms.

Looking forward, integrating perception-centric PRMs with robust tool-augmented architectures and extending direct process supervision to other reasoning modalities (e.g., temporal, causal, spatial) is a promising avenue for advancing coherent multimodal reasoning in future AGI systems.

Conclusion

The paper provides a formal framework and empirical validation for enhancing vision-LLM reasoning using perception-centric process reward models. By enabling token-level error grounding and integrating fine-grained supervision into RL training and inference, the approach achieves substantial improvements in visual grounding, reduces hallucination rates, and fosters robust generalization to reasoning tasks beyond perception. The methodology circumvents reward sparsity and reward hacking, with significant implications for scalable multimodal model training and inference.