ReVPT: Reinforced Visual Perception Tools

• ReVPT is a reinforcement learning framework that enhances multimodal visual reasoning by orchestrating pretrained visual tools and a novel GRPO algorithm.
• It employs a unique Group-Relative Policy Optimization method to refine tool usage strategies and improve performance on perception-heavy tasks.
• Empirical results demonstrate significant performance gains over supervised methods on various visual question answering and reasoning benchmarks.

Reinforced Visual Perception with Tools (ReVPT) is a reinforcement learning-based framework for enhancing the visual reasoning capabilities of large multimodal LLMs via structured access to a suite of pretrained visual tools. ReVPT addresses limitations of prior supervised finetuning and text-based reinforcement learning approaches by directly optimizing tool usage strategies through a novel group-relative policy optimization (GRPO) algorithm, resulting in superior performance on a range of perception-heavy visual question answering and reasoning benchmarks (Zhou et al., 1 Sep 2025).

1. Problem Formulation

ReVPT frames visual reasoning as a finite-horizon Markov Decision Process (MDP) $(\mathcal{S}, \mathcal{A}, P, R, \gamma)$. The state at time $t$ is defined as $s_t = (Q, I, h_t)$, where $Q$ is the question, $I$ the raw image, and $h_t = [(a_1, o_1), \ldots, (a_{t-1}, o_{t-1})]$ encodes the history of tool calls and model observations. The agent can select among language actions or one of four visual tools: object detection, zoom in, edge detection, and depth estimation. Tool calls deterministically augment the history with structured outputs, while language actions produce standard generative tokens. The terminal reward is defined as $+1$ if the final answer is both correct and well-formatted, otherwise $-1$. The objective is to maximize the expected cumulative return $J(\theta) = \mathbb{E}_{Q,\tau} [\sum_t r_t]$ over rollouts $\tau$ drawn from the model policy $t$0.

2. Group-Relative Policy Optimization and Learning Objectives

ReVPT utilizes Group-Relative Policy Optimization (GRPO), a variant of Proximal Policy Optimization designed to enable better credit assignment and sample efficiency in scenarios with sparse and trajectory-level rewards. For each question prompt $t$1, $t$2 rollouts $t$3 are sampled; rewards $t$4 are normalized via the group mean $t$5 and standard deviation $t$6, forming advantages $t$7. The GRPO loss is: $t$8 where $t$9 is the importance ratio and $s_t = (Q, I, h_t)$0 the KL penalty coefficient. A supervised pretraining (SFT) phase precedes RL to initialize the policy on high-quality, tool-augmented demonstration data using negative log-likelihood loss. This staged approach addresses exploration difficulties and bootstrap bias.

3. Visual Tools and Integration

ReVPT’s action space includes calls to four specialized visual tools, each implemented using off-the-shelf vision models:

Tool	Input	Output
Object Detection	Image $s_t = (Q, I, h_t)$1, text query $s_t = (Q, I, h_t)$2	Annotated image, boxes
Zoom In	Image $s_t = (Q, I, h_t)$3, region $s_t = (Q, I, h_t)$4, factor	Cropped, magnified image
Edge Detection	Image $s_t = (Q, I, h_t)$5	Edge map $s_t = (Q, I, h_t)$6
Depth Estimation	Image $s_t = (Q, I, h_t)$7	Depth map $s_t = (Q, I, h_t)$8

Tool calls are serialized in output with a special <tool_call>…</tool_call> token sequence. The environment executes the specified tool and feeds the observation $s_t = (Q, I, h_t)$9 back to the policy. Backbone models include LLMDet (Fu et al., 31 Jan 2025) for detection, Scharr-based edge detection, and Depth-Anything-v2 for depth. Standard resizing and normalization are the only preprocessing steps prior to tool execution.

4. Architecture and Multimodal Fusion

ReVPT is instantiated on Qwen2.5-VL-3B and Qwen2.5-VL-7B checkpoints, comprising a frozen vision transformer (ViT-L), cross-attention Q-former, and 3B or 7B parameter Llama-style decoder. Visual features—2048-dimensional global tokens and 32 learned Q-tokens—are projected into the decoder’s embedding space and attended over cross-modally at each generation step. Fine-tuning is restricted to decoder layers, projection layers, and adapters after each cross-attention; ViT and Q-former weights remain fixed.

5. Training Regime and Datasets

Training proceeds in two phases: (1) SFT on synthetic tool-augmented rollouts (generated by GPT-4.1 and carefully filtered) and benchmark-style multiple-choice QA; (2) GRPO-based RL using batch RL on challenging “hard” questions from TACO and SAT datasets, filtered to exclude items already answerable by the base model. Core training hyperparameters include SFT for 2 epochs at LR=$Q$0, RL finetuning for 200 steps at LR=$Q$1, with group size $Q$2 and KL penalty $Q$3. Each prompt is allowed up to five tool calls in sequence.

6. Empirical Performance and Ablation Studies

ReVPT is evaluated over eight vision-centric benchmarks: CV-Bench, BLINK, BLINK-Hard, MMVP, MMStar, MMMU, MMBench, MathVista. Both ReVPT-3B and ReVPT-7B attain state-of-the-art performance on perception-heavy tasks, recording perception gains of +9.03% (3B) and +9.44% (7B) over baseline instruct models on CV-Bench, and +18.81% (3B) on BLINK-Hard. Pure supervised finetuning offers modest improvement but harms generalization, while text-only RL is consistently inferior in visual-heavy settings.

Ablations demonstrate that the combined inclusion of both synthesized tool-use and general supervised data achieves the best tradeoff between perception and generalization. Both object detection and depth estimation are shown to be indispensable; RL fine-tuning in combination with these tools amplifies benchmark gains. RL further recalibrates tool usage patterns, shifting preference from “zoom/edge” overused after SFT to a stable reliance on detection/depth for substantive perception accuracy improvements.

7. Discussion, Limitations, and Prospective Work

Comparison between RL and SFT reveals that GRPO-based RL overcomes the rigidity of supervised tool chains, supporting exploration of alternative execution paths and tool-use policies. Object detection and depth estimation are consistently the most impactful tools, while zoom and edge detection contribute secondary, context-specific utility. Smaller models (3B) benefit more from explicit tool access, though scaling benefits are retained at 7B. Noted limitations include susceptibility to misleading tool outputs and bias introduced by the cold-start demonstration phase. Future directions suggested include RL-driven discovery of new tools, broader tool repository integration, and merging RL with process–reward models to mitigate biases from synthetic demonstration data (Zhou et al., 1 Sep 2025).

ReVPT constitutes the reference implementation of RL-based tool orchestration for visual reasoning in large multimodal models, offering openly available code, models, and datasets as a foundation for subsequent work in the domain.