VER-Bench: Fine-Grained Visual Evidence Benchmark

• VER-Bench is a benchmark that assesses if multimodal models can detect subtle visual clues, averaging just 0.25% of image area, to produce evidence-based answers.
• The design includes a multi-stage pipeline with automated processing, model-based filtering and cross-validation, followed by rigorous human expert review.
• Empirical results reveal that closed-source models excel in clue extraction and logical reasoning, exposing key limitations in smaller and open-source systems.

Searching arXiv for the specific VER-Bench paper and related naming overlap. VER-Bench is a benchmark for evaluating multimodal LLMs on reasoning that depends on fine-grained visual evidence rather than only on globally salient image content. It was introduced to assess whether models can identify subtle, localized clues and integrate them with world knowledge to produce evidence-based answers in real-world scenes (Qiang et al., 6 Aug 2025). In the benchmark, these clues occupy on average just 0.25% of the image area, and each question is paired with structured evidence consisting of visual clues and question-related reasoning derived from them (Qiang et al., 6 Aug 2025). A separate paper on embodied reinforcement learning also uses the label “VER-Bench” for benchmarking Variable Experience Rollout on navigation and mobile manipulation tasks; that usage refers to an RL benchmarking setting rather than the visual-evidence benchmark described here (Wijmans et al., 2022).

1. Definition and scope

VER-Bench evaluates two coupled capabilities in MLLMs: identifying fine-grained visual clues and integrating those clues with world knowledge for complex reasoning (Qiang et al., 6 Aug 2025). The benchmark is positioned against two common classes of prior evaluation: basic perception benchmarks, which focus on local details but lack deep reasoning, and mainstream reasoning benchmarks, which emphasize prominent image elements and may fail to assess subtle clues requiring intricate analysis (Qiang et al., 6 Aug 2025).

The central premise is that profound visual understanding and complex reasoning depend more on interpreting subtle, inconspicuous local details than on perceiving salient, macro-level objects (Qiang et al., 6 Aug 2025). The benchmark therefore emphasizes evidence that is semantically rich but spatially small. This suggests a shift from answer-only evaluation toward grounded assessment of how answers are visually supported.

The reasoning categories span Geospatial, Temporal, Situational, Intent, System State, and Symbolic reasoning, with each category designed to require both fine visual discrimination and higher-order contextual inference (Qiang et al., 6 Aug 2025). The result is a benchmark aimed at “genuine visual understanding and human-like analysis” through evidence extraction, integration, and reasoning (Qiang et al., 6 Aug 2025).

2. Benchmark design and contents

VER-Bench contains 342 images and 374 question-answer pairs (Qiang et al., 6 Aug 2025). Across these instances, it includes 614 unique visual clues annotated with approximately 977 bounding boxes, and the underlying scenes are described as real-world images with average resolution around $1500 \times 2000$ pixels (Qiang et al., 6 Aug 2025).

Each question is accompanied by structured evidence. Concretely, every QA instance contains the final answer and a list of evidence pairs consisting of a visual clue and an explanation or reasoning step precisely linked to image regions (Qiang et al., 6 Aug 2025). All visual clues are manually assigned bounding boxes for transparency and grounding (Qiang et al., 6 Aug 2025).

The six reasoning categories organize the dataset’s conceptual coverage:

Category	Evaluated ability
Geospatial Reasoning	Identify geographical location from subtle cues
Temporal Reasoning	Infer time or sequence from contextual clues
Situational Adaptation Reasoning	Assess context or activity from arrangements or behavior
Intent Inference	Deduce aims or plans of agents or objects
System State Understanding	Judge operational conditions or status
Symbolic Analysis	Interpret abstract symbols, codes, or non-linguistic imagery

This structure makes the benchmark neither a pure perception dataset nor a generic VQA collection. A plausible implication is that VER-Bench is intended to isolate a failure mode in which models can produce plausible answers without reliably grounding them in the small visual regions that actually determine the answer.

3. Annotation pipeline and evidence construction

The benchmark uses a multi-stage construction pipeline combining automated drafting, model-based filtering, model-based cross-validation, and human expert review (Qiang et al., 6 Aug 2025). In the first step, strong MLLMs generate scenario descriptions, reasoning-driven questions, answers, and initial evidence chains for each image (Qiang et al., 6 Aug 2025). In the second step, GPT-4o is used to validate the mutual relevance of image, scenario, and question, including robustness under partial masked images, and low-quality or ambiguous samples are filtered out (Qiang et al., 6 Aug 2025).

The third step uses multiple MLLMs, including Qwen2.5-VL-72B and Claude-3.5-Sonnet, to cross-validate answer, evidence, and reasoning consistency; instances with model consensus are retained (Qiang et al., 6 Aug 2025). The fourth step is manual verification by human experts, who check clue presence at box level, accuracy, and reasoning validity, correcting or discarding any hallucinated or missing clues (Qiang et al., 6 Aug 2025).

This construction protocol is central to the benchmark’s claims about groundedness. Because evidence pairs are explicitly tied to image regions and then manually verified, VER-Bench evaluates not only whether a model reaches an answer but whether it identifies the clue set and explanatory chain regarded as supporting that answer. This suggests that the dataset is designed to expose models that are verbally fluent yet weak at localized evidence extraction.

4. Evaluation protocol and metrics

VER-Bench prompts models to output a structured object of the form

$$\left\{A, \left[(c_1, r_1), \dots, (c_N, r_N)\right]\right\},$$

where $A$ is the final answer, $c_i$ is the $i$-th detected visual clue, and $r_i$ is the corresponding reasoning step connecting that clue to the answer (Qiang et al., 6 Aug 2025). The ground truth is represented analogously with a reference answer and a set of ground-truth clue-reasoning pairs (Qiang et al., 6 Aug 2025).

The benchmark uses four metrics, each scored in $[0,10]$ by GPT-4 (Qiang et al., 6 Aug 2025):

Metric	Definition
Answer Correctness (AC)	Semantic similarity between model and ground-truth answers
Clue Coverage (CC)	Matching degree between ground-truth and predicted clues
Reasoning Quality (RQ)	Semantic consistency between ground-truth and predicted reasoning
Evidence-Answer Relevance (ER)	Degree to which each evidence pair supports the answer

The paper defines Average Accuracy as

$$\text{Average Accuracy} = \frac{1}{K} \sum_{j=1}^{K} \left(\frac{AC_j + CC_j + RQ_j + ER_j}{4}\right) \times 100\%$$

for $K$ instances (Qiang et al., 6 Aug 2025).

A notable methodological choice is the use of holistic rather than one-to-one evaluation for Clue Coverage and Reasoning Quality: all model and ground-truth clues or reasoning are concatenated and compared at a holistic level (Qiang et al., 6 Aug 2025). The stated rationale is that reasoning steps in vision are often non-sequential and largely complementary (Qiang et al., 6 Aug 2025). This distinguishes VER-Bench from evaluations that depend on exact step alignment. A common misconception is that the benchmark is only measuring answer quality with an added rationale field; in fact, the metric design explicitly separates answer accuracy, clue identification, reasoning consistency, and evidence-answer coherence.

5. Empirical results and failure modes

VER-Bench evaluates 12 open-source models and 3 closed-source models, including Qwen2.5-VL variants, InternVL variants, MiniCPM variants, LLaVA-OneVision-72B, Gemini-2.5-Pro-Preview, GPT-4o, and Claude-3.7-Sonnet (Qiang et al., 6 Aug 2025). The reported results show substantial performance differences across model families.

Model	AC	CC	RQ	ER	Avg Acc.
Gemini-2.5-Pro-Preview	79.0	67.9	69.4	90.7	76.8
GPT-4o	62.8	46.7	54.5	93.3	64.4
Qwen2.5-VL-32B	63.6	49.6	45.6	84.4	60.8
Qwen2.5-VL-72B	61.8	38.2	46.6	92.0	59.6
InternVL3-78B	55.1	40.6	45.7	86.2	56.9
LLaVA-OneVision-72B	48.8	24.7	33.7	86.5	48.4

The benchmark reports that closed-source models substantially outperform open-sourced models, especially in identifying subtle clues and producing logical, stepwise reasoning (Qiang et al., 6 Aug 2025). Among open-source systems, Qwen2.5-VL-32B is identified as the best performer, slightly ahead of Qwen2.5-VL-72B and InternVL3-78B, with the summary attributing this to reinforcement learning and better instruction following (Qiang et al., 6 Aug 2025).

Several recurring limitations are highlighted. Most models struggle with Clue Coverage, indicating that they frequently miss small but critical regions (Qiang et al., 6 Aug 2025). Reasoning Quality is also a bottleneck, suggesting that even when clues are partially identified, models often fail to connect them into a logically consistent argument (Qiang et al., 6 Aug 2025). At the same time, Evidence-Answer Relevance is relatively high across models, which indicates that models can often produce internally supported arguments even when those arguments are incomplete or ultimately incorrect (Qiang et al., 6 Aug 2025). This pattern suggests that internal coherence should not be conflated with grounded correctness.

Another reported observation is that model size alone does not guarantee stronger performance: larger parameter counts do not consistently translate into better fine-grained reasoning (Qiang et al., 6 Aug 2025). The summary cites LLaVA-OneVision-72B as an example and suggests that cross-modal or video-transfer pretraining does not automatically resolve the benchmark’s localized-evidence demands (Qiang et al., 6 Aug 2025).

6. Interpretation, significance, and naming overlap

VER-Bench’s primary significance lies in making fine-grained visual evidence extraction an explicit evaluation target rather than an implicit prerequisite for answering questions correctly (Qiang et al., 6 Aug 2025). The benchmark therefore belongs to a broader movement toward grounded multimodal evaluation, but its distinctive emphasis is on subtle local clues, structured evidence, and evidence-based argument quality.

Its findings motivate several research directions already stated in the benchmark summary: advancing open-source MLLMs’ perception, evidence extraction, and step-wise grounded reasoning; improving instruction-following and logical consistency; and adopting metrics that go beyond answer correctness (Qiang et al., 6 Aug 2025). A plausible implication is that future model improvements on VER-Bench may require changes not only in reasoning modules or prompting, but also in visual tokenization, region sensitivity, and training procedures that explicitly reward clue discovery.

The term “VER-Bench” also has a separate usage in embodied reinforcement learning. In “VER: Scaling On-Policy RL Leads to the Emergence of Navigation in Embodied Rearrangement,” the label is used for a benchmarking framework comparing synchronous on-policy RL, asynchronous on-policy RL, and Variable Experience Rollout on Habitat navigation and rearrangement tasks (Wijmans et al., 2022). That framework reports throughput and sample-efficiency comparisons for VER against DD-PPO and SampleFactory, but it is conceptually unrelated to the fine-grained visual-evidence benchmark for MLLMs (Wijmans et al., 2022). The shared name is therefore a naming overlap rather than a common benchmark lineage.

VER-Bench is publicly released with dataset and additional materials at the repository specified by the paper (Qiang et al., 6 Aug 2025). Within the visual reasoning literature, it functions as a benchmark for testing whether models can ground conclusions in small, semantically decisive image evidence rather than merely generate plausible multimodal explanations.