Visual Reasoning Benchmarks

• Visual reasoning benchmarks are rigorously designed datasets that assess AI systems’ ability to extract, compare, and manipulate visual information across multi-modal tasks.
• They isolate core skills such as spatial planning, compositionality, and temporal ordering, driving advances in vision-language model architectures.
• Empirical findings reveal a persistent human-machine gap, with models struggling in tasks like temporal reasoning and multi-modal output despite increased model scaling.

Visual Reasoning Benchmarks

Visual reasoning benchmarks are rigorously designed datasets and evaluation protocols for assessing an AI system’s ability to extract, compare, and manipulate information from visual inputs, often in conjunction with language. These benchmarks isolate and quantify core cognitive skills such as comparison, abstraction, spatial planning, compositionality, and selective attention that are central to both human and artificial visual intelligence. Over the last decade, a diverse ecosystem of benchmarks has catalyzed advances in both vision-LLM (VLM) architectures and in the understanding of mechanistic failures and scaling limits.

1. Taxonomy and Evolution of Visual Reasoning Benchmarks

Visual reasoning benchmarks can be grouped by their targeted reasoning skill, input-output modality, level of abstraction, and real vs. synthetic provenance:

• Comparison Benchmarks: Test quantity, geometric, spatial, and temporal image comparison (e.g., CompareBench (Cai et al., 25 Sep 2025)).
• Perceptual Reasoning: Focus on visual illusions or adversarial images to stress perception over prior knowledge (e.g., BLINK-Twice (Ye et al., 10 Oct 2025)).
• Spatial and Planning Reasoning: Emphasize dynamic spatial manipulation, agent planning, or spatial alignment in 2D/3D (e.g., iVISPAR (Mayer et al., 5 Feb 2025)).
• Compositional and Abstract Reasoning: Feature multi-relation compositions, progression-style puzzles, and analogical reasoning (e.g., CVR (Zerroug et al., 2022), V-PROM (Teney et al., 2019), Raven/PGM derivatives (Mondal et al., 2023)).
• Mathematical and Chart-Based Reasoning: Integrate visual diagrams, charts, or math with textual reasoning; often probe vision–math fusion (e.g., ChartMuseum (Tang et al., 19 May 2025), VisAidMath (Ma et al., 30 Oct 2024)).
• Selective Attention & Argumentative Reasoning: Require isolating argument-relevant regions and linking them in reasoning chains (e.g., VisArgs (Chung et al., 27 Jun 2024)).
• Multi-modal Output and Editing: Mandate image-generation as part of reasoning (e.g., RBench-V (Guo et al., 22 May 2025), RISEBench (Zhao et al., 3 Apr 2025)).

This diversification reflects the field’s evolution from simple recognition and captioning (CLEVR, VQA) to structured, multi-step, and vision-centric tasks that quantify gaps in model (and often human) reasoning.

2. Representative Benchmarks: Scope and Task Design

Controlled Visual Comparison: CompareBench

CompareBench (Cai et al., 25 Sep 2025) isolates four core visual comparison skills: quantity (counting), temporal (chronological ordering), geometric (measuring lengths/areas), and spatial (depth, verticality). Each sub-benchmark utilizes carefully constructed inputs—image grids or labeled single images—with forced-choice answers. Notably, CompareBench includes:

• TallyBench: 2,000 single-image counting tasks spanning biological and artificial objects with exact integer ground truth.
• HistCaps: 515 historical images for temporally anchored comparison.
• CompareBench QA: 1,000 QA pairs, split across CTally (quantity), CTemp (temporal), CGeom (geometry), and CSpat (spatial).

Vision-centric Perceptual Reasoning: BLINK-Twice

BLINK-Twice (Ye et al., 10 Oct 2025) targets deception and misperception, using seven illusion classes (misleading, dislocation, art illusion, occlusion, forced perspective, physical illusion, motion illusion) and pairs of original/adversarial images. Models must provide both yes/no answers and token-level reasoning chains.

Interactive Spatial Planning: iVISPAR

iVISPAR (Mayer et al., 5 Feb 2025) evaluates sequential and spatial planning via a vision-language adaptation of the sliding-tile puzzle in 2D, 3D, and text modalities. Models must execute valid move sequences, optimally reconfigure object layouts, and demonstrate path efficiency.

Compositional and Abstract Reasoning: CVR, Raven/PGM Derivatives

CVR (Zerroug et al., 2022) and PGM/I-RAVEN (Mondal et al., 2023) compose visual rules from a vocabulary of elementary relations (e.g., shape, color, count, spatial inclusion), generating puzzles that require the model to infer and transfer combinations of rules (∧, ∨) in “odd-one-out” or matrix completion formats. Performance is evaluated not just by accuracy but also by sample efficiency and compositional transfer.

Chart and Math Reasoning: ChartMuseum, VisAidMath

ChartMuseum (Tang et al., 19 May 2025) and VisAidMath (Ma et al., 30 Oct 2024) require models to perform fine-grained visual extraction from data visualizations and diagrams, where the correct answer often cannot be deduced without explicit visual reasoning (e.g., reading bar heights, constructing geometric aids).

Selective Vision and Argument Structure: VisArgs

VisArgs (Chung et al., 27 Jun 2024) introduces argument-annotated images with visual premises localized by bounding boxes, and the requirement to link only argument-relevant image regions in reasoning trees.

Multi-step, Multi-modal Output: RBench-V, RISEBench

RBench-V (Guo et al., 22 May 2025) and RISEBench (Zhao et al., 3 Apr 2025) go beyond text responses, requiring reasoning steps that involve generating new images—drawing auxiliary lines, tracing paths, or editing real/synthetic scenes according to complex, logic-bound instructions.

3. Core Methodologies and Metrics

Visual reasoning benchmarks impose diverse protocols tailored to isolating genuine reasoning skill:

• Answer Modalities: Single-choice, open-ended, binary (yes/no), or generative (image output).
• Evaluation Metrics: Accuracy, mean path deviation, sample efficiency (AULC, SES), reasoning-chain fidelity (CoT-Score, reasoning fidelity), transfer/generalization scores, and domain-specific metrics (Jaccard/F-measure for segmentation, BLEU/ROUGE for narrative VQA).
• Protocols: Zero-shot single-prompts (e.g., CompareBench), multi-turn interaction with environment feedback (iVISPAR), chain-of-thought prompting and reasoning-trace analysis (BLINK-Twice, VERIFY (Bi et al., 14 Mar 2025), VisuLogic (Xu et al., 21 Apr 2025)), and “LLM-as-Judge” for subjective or image-based outputs.

Benchmark construction strategies include both human curation for naturalistic and diverse edge cases, and synthetic generation (often with programmatic control over abstract relations, spatial layouts, or distractor design) for exhaustive coverage and compositional depth.

4. Empirical Findings and Model Limitations

Consistent trends across modern benchmarks:

• Persistent Human-Machine Gap: Even leading closed-source models lag human ceiling (typically 92–99%) by 20–60 points depending on benchmark and sub-task, especially in vision-only, spatial, planning, and multi-modal output regimes. On CompareBench, the best API achieves 85% overall, but only 64% on temporal ordering, whereas humans reach 92% overall, 30% (vision-only) on temporal tasks.
• Scaling Law Trends: Model accuracy typically increases with parameter count, but performance saturates below human baselines on advanced visual reasoning tasks. For instance, open-source models chronically underperform leading APIs in CompareBench and SpatialViz-Bench (Wang et al., 10 Jul 2025).
• Key Error Modes:
  • Counting and Visual Comparison: Systematic under- or over-counting, confusion of visually similar categories, and semantic overreliance (e.g., “building” interpreted as “tall”).
  • Spatial/Geometric Reasoning: Confusion between geometric dimensions, errors in folding/unfolding, or spatial alignment. Notable 2D → 3D performance cliffs observed throughout spatial and folding tasks.
  • Temporal and Illusory Reasoning: Reliance on memorized knowledge as opposed to vision-based cues; inability to resolve illusions or physically mismatched realities.
  • Selective Attention and Argumentation: Failure to isolate or ground inference on relevant visual premises; attention diffusion across irrelevant scene regions.
  • Multi-modal Output Failure: Models routinely ignore instructions to generate visual outputs or substitute textual descriptions for required drawings.
• Task-Specific Discoveries:
  • Active Visual Interaction and Multi-turn Observation: Repeated visual engagement (BLINK-Twice’s re-seeing protocol) often yields 5–10% absolute accuracy gains, particularly for weaker visual encoders.
  • Reasoning Chain Fidelity: Chain-of-thought prompting aids some QA performance but often leads to verbose, unstable, or tangential explanations, with true reasoning quality (CoT-Score, VERIFY’s fidelity) lagging far behind answer accuracy.

5. Directions for Future Benchmarking and Model Design

Contemporary and next-stage benchmarks articulate a clear agenda:

1. Hybrid and Modular Reasoning: Explicit decoupling of perceptual and symbolic/knowledge reasoning via neuro-symbolic or modular architectures, particularly for tasks requiring external world knowledge (historical ordering), compositional transfer, or geometry hybridization (Cai et al., 25 Sep 2025, Zerroug et al., 2022).
2. Task-specific and Curriculum-based Fine-tuning: Integration of synthetic, compositional, or CLEVR-style curricula for geometry and spatial relations, followed by adaptation to real-world images and plans (Cai et al., 25 Sep 2025).
3. Self-calibration and Uncertainty Quantification: Automatic detection when visual evidence is insufficient or ambiguous, prompting deferral or active request for additional input (Cai et al., 25 Sep 2025).
4. Depth, Multi-view, and Dynamic Input Fusion: Improved handling of 3D perception and multi-modal integration to overcome 2D/3D cliffs and video/dynamic reasoning scenarios (Mayer et al., 5 Feb 2025, Wang et al., 10 Jul 2025, Shen et al., 17 May 2025).
5. Multi-modal Chain-of-Thought (M-CoT): Enabling interleaved reasoning that alternates between visual and text output at each step; benchmarking progress via RBench-V (Guo et al., 22 May 2025).
6. Reinforcement Learning with Visual Grounding: ViGoRL (Sarch et al., 29 May 2025) demonstrates substantial gains via RL formulations anchoring reasoning steps to spatial coordinates with active zoom and subgoal setting.

Ongoing directions include formulating better semantic metrics for generative or diagram-based tasks, evaluating dynamic and embodied interaction (iVISPAR, VisualWebArena), and scaling up agent-based or tool-using paradigms where visual reasoning is not merely passive.

6. Benchmark Landscape: Comparative Synopsis

Benchmark	Core Skill(s)	#Items	Input	Output	SOTA (Model)	Human	Random
CompareBench	Quantity/Temp./Geo./Spatial Compar.	1,000	Images (grids)	Forced choice (A–D)	85.4% (Gemini-2.5 Pro)	92%	25%
BLINK-Twice	Illusion/Perceptual Reasoning	896 Qs	Image	Yes/No + CoT	66.7% (Gemini-2.5 Pro*)	–	50%
iVISPAR	Spatial Planning/Alignment	900	2D/3D/Text	“move ...” sequences	54.6% (Sonnet-3.5, 2D)	≈90%	<10%
CVR	Compositional Abstraction	103 rules	Synthetic images	“odd-one-out” choice	67.7% (ResNet-50, SSL)	78.7%	25%
ChartMuseum	Chart Visual/Textual Reasoning	1,000	Real charts	Short answer	63% (Gemini-2.5 Pro)	93%	25%
VisAidMath	Math-Visual Fusion/Construction	1,200	Diagrams/Equations	Numeric/Derivation	45.3% (GPT-4-Vision)	–	–
VisArgs	Selective Vis. Premise, Argument QA	1,611	Image, tree	Box/conclusion/region	79.5% (GPT-4-O, Select.)	98%	–
SpatialViz-Bench	Spatial Visualization	1,180	Multi-panel	Forced choice	44.7% (Gemini-2.5 Pro)	90%+	25%
RBench-V	Multi-modal Output (draw/trace)	803	Multi-modal	Image/text gen.	25.8% (o3)	82.3%	–

*Definitive human and random baselines and item counts given only if explicitly available.

7. Significance and Role in Model Development

Visual reasoning benchmarks have systematically exposed core limitations of contemporary VLMs—such as the inability to abstract visual patterns without language crutches, poor spatial/geometric generalization, or failure to ground reasoning on argument-relevant visual content. They also provide a controlled substrate for rapid iteration and diagnosis of architectural, training, and prompting interventions. By structuring tasks to stress compositionality, interactivity, and hybrid judgment, these benchmarks shift the field toward agents with robust, vision-centric, and human-aligned reasoning skills.

The future trajectory will demand datasets with even richer, more interactive scenarios, multi-turn and tool-based problem solving, dynamic vision, and explicit integration of visual attention and memory mechanisms—a direction already presaged by emerging benchmarks and novel agentic frameworks.