---
title: 'VisualProcessBench: Multimodal Reasoning Benchmark'
url: https://www.emergentmind.com/topics/visualprocessbench
type: topic
---

# VisualProcessBench: Multimodal Reasoning Benchmark

VisualProcessBench is a multimodal benchmark specifically designed to evaluate step-level reasoning correctness in visual language models and Process Reward Models (PRMs) for complex, multi-step vision-language reasoning tasks. It provides fine-grained, human-annotated labels on individual reasoning steps, enabling systematic measurement of PRM performance as intermediate “critics” in visual chain-of-thought (CoT) scenarios. It has become a crucial tool for advancing test-time scaling methods, PRM development, and rigorous error analysis in multimodal AI research [2503.10291][2506.09532].

## 1. Motivation and Benchmark Scope

The primary objective of VisualProcessBench is to measure the ability of PRMs and Multimodal Large Language Models (MLLMs) to accurately assess the correctness of each step in a visual CoT solution, rather than merely identifying the first erroneous step. This distinction is critical for robust performance in advanced test-time scaling (TTS) and Best-of-N (BoN) reranking workflows, where precise detection of all flawed reasoning steps supports downstream selection of optimal candidate solutions. Previous benchmarks only located the first error, underestimating detection difficulty when AI systems demonstrate self-reflection or produce plausible but faulty multi-step chains.

It addresses the need for a multimodal, human-annotated standard, exposing current models’ strengths and weaknesses in visuo-linguistic stepwise judgment and supporting systematic comparison across PRMs, ORMs, and alternative critic strategies [2503.10291].

## 2. Dataset Structure and Annotation Protocol

VisualProcessBench is constructed from five visual math-reasoning sources: MMMU (multidisciplinary), MathVision, MathVerse (vision-only split), DynaMath, and WeMath. Each sample comprises an image, a problem prompt, the ground-truth answer, and a full solution decomposed into “natural” steps (delimited by blank lines or explicit separators). The dataset is summarized as follows:

| Subset       | Problems | Annotated Steps | Solution Sources      |
|--------------|----------|-----------------|----------------------|
| MMMU         | 267      | –               | GPT-4o, Claude-3.5, QvQ-72B, InternVL2.5-78B |
| MathVision   | 712      | –               | As above             |
| MathVerse    | 1,026    | –               | As above             |
| DynaMath     | 570      | –               | As above             |
| WeMath       | 291      | –               | As above             |
| **Total**    | 2,866    | 26,950          | –                    |

Each step is annotated with one of three human-defined labels by expert annotators:
- “+”: Correct and contributes valid reasoning
- “−”: Incorrect due to logical or factual error
- “o”: Neutral—provides no reasoning or purely formatting (ignored in evaluation)

Thirteen annotators (≥ university degree) completed 39 person-days of labeling, with 10% spot-checking per batch and a policy allowing annotators to skip incomprehensible steps. Solutions were originally generated by current frontier MLLMs, increasing domain realism but limiting diversity to model-generated chains [2503.10291].

## 3. Evaluation Metrics and Protocols

VisualProcessBench enables both step-wise and end-to-end evaluations for critic models:

- **Single-Pass Step Judgment**: For each annotated step, the critic model produces a binary probability (“+” vs. “−”); “o” labels are excluded. The following definitions apply:
  - Precision and Recall per label:
    $$
    P = \frac{\mathrm{TP}}{\mathrm{TP} + \mathrm{FP}}, \quad R = \frac{\mathrm{TP}}{\mathrm{TP} + \mathrm{FN}}
    $$
  - F1 per label:
    $$
    F1 = \frac{2PR}{P+R}
    $$
  - Macro-F1:
    $$
    \mathrm{Macro\mbox{-}F1} = \frac{1}{2}(F1_{+} + F1_{-})
    $$
  - Micro-averaged F1 (as used by Athena-PRM):
    $$
    F1_{\text{micro}} = \frac{2 \cdot TP_{\text{all}}}{2 \cdot TP_{\text{all}} + FP_{\text{all}} + FN_{\text{all}}}
    $$

- **Best-of-N (BoN) Reranking**: The MLLM generates $N$ candidate solutions; the critic model scores and reranks based on mean step-level rewards, with the highest-scoring chain’s answer submitted for task evaluation. BoN performance is measured by task-level accuracy over the seven constituent benchmarks.

This dual evaluation paradigm supports direct comparison between PRMs, ORMs, self-consistency, and fallback baseline models [2503.10291][2506.09532].

## 4. Baseline Model Performance

VisualProcessBench provides detailed open-source and proprietary baseline results for single-pass step-level judgment (macro-F1) and BoN-8 accuracy. Selected results are summarized below:

| Model                     | Macro-F1 (%) |
|---------------------------|--------------|
| Random Guessing           | 50.0         |
| GPT-4o-Mini               | 57.9         |
| GPT-4o                    | 60.3         |
| Gemini-2.0-Flash          | 62.3         |
| VisualPRM-8B              | 62.0         |
| Athena-PRM (Qwen2.5-VL-7B)| 65.9         |
| Qwen2.5-VL-72B            | 60.5         |
| InternVL2.5-78B           | 52.6         |

Athena-PRM, using data-efficient consistency-based labeling and ORM initialization, establishes a new state-of-the-art macro-F1 (65.9%, +3.9 over VisualPRM-8B) and demonstrates marked gains especially in the MMMU and WeMath subsets (+15.6 and +12.0 F1, respectively) [2506.09532]. Ablation studies indicate incremental improvements from ORM initialization, filtered data, and negative up-sampling.

Open-source MLLMs exhibit high recall but poor precision on incorrect steps, often failing to localize errors precisely, while PRMs narrowed this gap. VisualPRM and Athena-PRM, utilizing advanced label filtering and negative mining, approach or surpass proprietary models such as Gemini-2.0-Flash in macro-F1.

## 5. Critical Properties and Limitations

VisualProcessBench is characterized by:
- **Multidomain coverage**: spanning multidisciplinary, mathematical, and logic reasoning over images.
- **Rigorous human annotation**: stepwise supervision from expert annotators ensures label quality and detection of nuanced logical flaws.
- **Granular step correctness**: evaluation focuses on exact identification of all erroneous reasoning steps, not solely the first failure.
- **Flexible error analysis**: metrics support macro- and micro-averaged F1 and ablation of specific error types/difficulties.

Limitations include:
- Annotation cost and dataset scale (≈2.9K questions, 27K steps).
- Solutions derived from a limited set of high-performing MLLMs, potentially restricting reasoning diversity.
- Neutral/formatting steps are ignored, and partial credit for intra-step errors is not awarded.
- The focus is exclusively on reasoning step correctness, not on answer span scoring or generative quality [2503.10291].

## 6. Integration Within the Multimodal Reasoning Ecosystem

VisualProcessBench catalyzed the development and evaluation of leading PRMs, including VisualPRM [2503.10291] and Athena-PRM [2506.09532]. It enables:
- Systematic benchmark-driven comparison between critic architectures.
- Support for advanced TTS and BoN reranking pipelines, contributing to higher final task accuracy (e.g., Athena-PRM yielded +10.2 points on WeMath in TTS).
- Fine-grained diagnostic studies, exposing current model failure modes (e.g., systematic overprediction of correctness by open models).
- Facilitating robust reward modeling, crucial for reinforcement learning from reasoning processes (as opposed to pure outcome reward).

Related resources such as VisualPRM400K (for PRM training) and alternate visual prompting benchmarks (e.g., VP-Bench for visual cue interpretation) address complementary facets of multimodal reasoning and grounding, but VisualProcessBench uniquely targets step-level visuo-linguistic CoT verification [2503.10291][2511.11438].

## 7. Access and Reproducibility

VisualProcessBench and associated PRMs/datasets are publicly available at https://internvl.github.io/blog/2025-03-13-VisualPRM/, with version 1.0 stable as of March 2025. This accessibility supports reproducible research and rapid iteration on visuo-linguistic reward models and evaluation protocols. Users are encouraged to cite Wang et al. (CVPR 2025) for benchmark usage [2503.10291].

Source: https://www.emergentmind.com/topics/visualprocessbench