CompareBench: Visual Comparison Benchmark

• CompareBench is a diagnostic benchmark that isolates visual comparison reasoning by evaluating tasks in quantity, temporal, geometric, and spatial domains under controlled conditions.
• It leverages fixed A–D answer formats and standardized prompts to ensure strict, zero-shot evaluation without training splits.
• The benchmark enables researchers to compare VLM performance across specific tasks using controlled datasets derived from TallyBench and HistCaps.

CompareBench is a benchmark for evaluating visual comparison reasoning in vision-LLMs (VLMs), defined as the ability to make relative judgments such as which image contains more instances, which scene occurred earlier, which object is longer, or which marked point is closer to the camera. It contains 1,000 single-choice question-answer pairs across four tasks—quantity, temporal, geometric, and spatial—and is designed as a controlled diagnostic evaluation rather than a broad visual question answering suite (Cai et al., 25 Sep 2025). In adjacent literature, the term also appears more broadly as a label for comparative benchmarking systems and methodologies, especially where the central object of study is not absolute prediction but structured comparison across models, systems, plans, or outputs (Kil et al., 2024).

1. Definition and research scope

CompareBench was introduced to evaluate a capability that is described as fundamental yet understudied in current VLM evaluation: visual comparison reasoning. The benchmark directly probes four comparison abilities that go beyond recognition and captioning: quantity comparison, temporal ordering, geometric property comparison, and spatial relation reasoning. Its design addresses a gap left by broader multimodal evaluations such as VQA, GQA, MMBench, and MM-Vet, which seldom isolate and systematically test questions of the form “which is more,” “which is earlier,” “which is longer,” or “which is closer” (Cai et al., 25 Sep 2025).

The benchmark is framed as a pure evaluation corpus. No train/dev/test splits are reported, and evaluation is conducted in a standardized, zero-shot setting using unified instruction templates per task. The benchmark is derived from two auxiliary datasets constructed for this purpose: TallyBench, a counting dataset with 2,000 image-question pairs, and HistCaps, a collection of 515 historical images with bilingual captions. This construction makes CompareBench simultaneously diagnostic and controlled: diagnostic because it isolates specific comparative operations, and controlled because prompts, answer spaces, and target entities are tightly specified.

A related but broader benchmark lineage is visible in multimodal comparative reasoning work. CompBench, introduced as MLLM-CompBench, evaluates multimodal LLMs on around 39.8K image-pair triplets across eight dimensions—visual attribute, existence, state, emotion, temporality, spatiality, quantity, and quality—using horizontally concatenated image pairs and constrained outputs. CompareBench is narrower in taxonomy but more tightly focused on four core comparison skills and a unified A–D answer format (Kil et al., 2024).

2. Dataset composition and source corpora

CompareBench comprises four sub-benchmarks with fixed sample counts and distinct input structures.

Sub-benchmark	Samples	Core format
CompareTallyBench	600	4-image grid, output A/B/C/D
CompareTemporalBench	100	4-image grid, output A/B/C/D
CompareGeometryBench	200	Single image with four labeled objects A–D
CompareSpatialBench	100	Single image with four labeled points/objects A–D

The quantity subset, CompareTallyBench, uses a 4-image grid at 1600×1600 derived from TallyBench. Questions ask which image shows the most instances of a target category, with examples such as “Which image shows the most dogs?” or “Which image contains the most spoons?” The model must output one of A, B, C, or D. TallyBench itself contains 2,000 counting image-question pairs spanning around 50 fine-grained categories under two top-level groups—Biology and Artificial Objects—and each item includes JSON metadata with image_name, vlm_question, gt_answer, categories, and image_type (Cai et al., 25 Sep 2025).

The temporal subset, CompareTemporalBench, uses a 4-image grid at 1920×1440 built from HistCaps. It asks which scene appears earlier in history. The task depends on visual cues such as clothing, architecture, and technology, together with world knowledge. HistCaps supplies temporally anchored historical content with temporal tags and long and short captions in English and Chinese, but does not itself contain question-answer annotations; these are constructed in CompareBench.

The geometric subset, CompareGeometryBench, consists of single images containing four labeled objects A–D. It targets intrinsic dimensional properties—length, width, height, thickness, and diameter—with equal coverage of the five properties at 40 samples each. Questions take forms such as “Which object is longer in length?” The spatial subset, CompareSpatialBench, also uses single images with four labeled targets and is divided evenly between depth and vertical height, with 50 samples for each. Representative prompts include “Which object is closer to the camera?” and “Which marked point is higher above ground?” (Cai et al., 25 Sep 2025).

Across all four subsets, prompts enforce strict comparison rules: only marked entities are to be considered, unmarked context is to be ignored, exactly one correct answer is expected, and resizing distortions in grid-based tasks are explicitly acknowledged in the instructions.

3. Prompting, outputs, and evaluation protocol

CompareBench standardizes both prompts and outputs. Counting prompts in TallyBench require exact integer answers with explicit inclusion and exclusion rules: all identifiable instances are to be counted, including drawings or prints, and partially occluded, cropped, or blurred objects are to be included if recognizable; reflections are excluded. CompareBench itself converts the problem into single-choice comparative QA, requiring a single letter A–D with no additional text (Cai et al., 25 Sep 2025).

The benchmark is evaluated in zero-shot mode using off-the-shelf systems rather than task-specific fine-tuning. Closed-source APIs include OpenAI models from GPT-4o mini through GPT-5 and o3/o4 variants, Gemini 2.5 Flash-Lite, 2.5 Flash, and 2.5 Pro, and Claude Sonnet 4. Open-source evaluations cover Qwen2.5-VL-3B, -7B, -32B, -72B, and Qwen3-VL-235B-A22B. The paper emphasizes standardized prompts and fixed image inputs; no fine-tuning is reported (Cai et al., 25 Sep 2025).

Accuracy is the sole reported metric. For TallyBench, exact-count accuracy is

$$\mathrm{Acc}_{\text{Tally}} = \frac{1}{N}\sum_{i=1}^{N} \mathbb{1}\!\left(\hat{y}_i = y_i\right)\times 100\%.$$

For CompareBench, single-choice accuracy is

$$\mathrm{Acc}_{\text{CompareBench}} = \frac{1}{M}\sum_{j=1}^{M} \mathbb{1}\!\left(\hat{c}_j = c_j^{*}\right)\times 100\%.$$

Per-subset accuracies are reported for CompareTallyBench, CompareTemporalBench, CompareGeometryBench, and CompareSpatialBench, and overall CompareBench accuracy is a micro average over all 1,000 questions. The benchmark does not report robustness metrics, explanation metrics, or train/test generalization measures. Its robustness claims instead derive from controlled task construction, strict prompts, labeled targets, and standardized grids.

4. Empirical performance and scaling behavior

The reported results show substantial but incomplete capability. On overall CompareBench accuracy, Gemini 2.5 Pro is best at 85.40%, followed by OpenAI GPT-5 at 79.20%, OpenAI o3-pro at 79.10%, OpenAI o4-mini at 79.00%, and Gemini 2.5 Flash at 78.90%. Among open-source models, Qwen3-VL-235B-A22B reaches 65.40%, while Qwen2.5-VL-72B reaches 57.50% (Cai et al., 25 Sep 2025).

Task-specific performance is uneven. In quantity comparison, Gemini 2.5 Pro achieves 90.83%, ahead of Gemini 2.5 Flash at 86.33%, OpenAI o4-mini at 85.83%, GPT-5 mini at 84.17%, and GPT-5 at 81.17%. In temporal ordering, GPT-5 is best at 74.00%, followed by o3-pro at 72.00%, o3 at 65.00%, and Gemini 2.5 Pro at 64.00%. In geometric comparison, Gemini 2.5 Pro leads at 82.00%, followed by GPT-5 mini at 74.50%, GPT-5 at 72.50%, and GPT-4.1 at 70.50%. In spatial relations, GPT-5 reaches 86.00%, while Gemini 2.5 Pro and o4-mini are tied at 81.00%.

Scaling trends are explicit. Larger models, both proprietary and open-source, consistently achieve higher accuracy across tasks. On spatial reasoning, Qwen2.5-VL rises from 36.00% at 3B to 68.00% at 72B. On the underlying TallyBench counting task, Qwen2.5-VL increases from 56.05% at 3B to 75.60% at 72B. These trends indicate that scale improves comparative reasoning, but the benchmark also shows that scale does not eliminate core weaknesses.

Human-model contrasts are asymmetric. Humans are near-perfect on counting, geometry, and spatial comparisons, but score only 30.00% on temporal ordering when restricted to vision-only reasoning. Large VLMs can outperform that human vision-only baseline on temporal questions because they can exploit world knowledge learned during pretraining. This makes CompareTemporalBench diagnostically unusual: it is partly a perception task and partly a knowledge-mediated inference task (Cai et al., 25 Sep 2025).

5. Failure modes, limitations, and interpretive cautions

The paper’s error analysis identifies systematic failure modes rather than isolated mistakes. In counting, models confuse visually similar instances, miss partially occluded or stacked objects, and misread small repetitive objects under clutter or blur. In geometry, they misclassify the dimension being compared—for example, length versus height or thickness versus width—and may rely on texture or appearance rather than geometric structure. In spatial reasoning, common errors include foreground/background confusion, incorrect camera-relative depth judgments, and treating marked points in the sky or ground as object features rather than spatial coordinates. In temporal ordering, errors arise when scenes are visually similar and lack reliable era-specific cues, making predictions sensitive to prompt wording and pretrained background knowledge (Cai et al., 25 Sep 2025).

Several benchmark limitations are explicit. There is no train/dev/test split, since the corpus is evaluation-only. The single-choice format emphasizes diagnostic correctness rather than generative explanation. The temporal subset intentionally mixes perceptual cues with world knowledge, which complicates strict vision-only interpretation. Potential biases arise from TallyBench category distribution, from curated historical narratives in HistCaps, and from bilingual captioning, which may carry translation biases. The paper also notes that resizing distortions in grid tasks are acknowledged in the instructions rather than eliminated by construction.

These limitations matter for interpretation. A high score on CompareBench is evidence of competence in constrained visual comparison under standardized prompts; it is not by itself evidence of calibrated explanation, causal reasoning, or robustness to prompt reframing. Conversely, poor performance on a subtask does not necessarily isolate a single deficit: temporal errors may reflect missing historical knowledge, while spatial errors may reflect failures in depth estimation, marker grounding, or both. This suggests that CompareBench is best used as a diagnostic layer within a broader evaluation stack rather than as a standalone measure of multimodal general intelligence.

6. Comparative benchmarking ecosystem and methodological extensions

CompareBench belongs to a broader methodological family in which the benchmarked object is a relative judgment rather than an absolute label. In multimodal reasoning, MLLM-CompBench operationalizes this family at larger scale by mining and pairing images across 14 datasets and around 39.8K triplets, with eight comparison dimensions and constrained outputs such as Left, Right, or Same. CompareBench is more compact and more controlled, whereas MLLM-CompBench is broader in coverage and pairwise in structure (Kil et al., 2024).

In systems benchmarking, DBenVis provides a distinct but conceptually related example: an interactive visual analytics system that acts as a unified “CompareBench” for DBMS evaluation. It ingests sysbench and TPC-H outputs, normalizes DBMS-specific EXPLAIN plans across PostgreSQL, MariaDB, and MySQL, and supports side-by-side analysis of throughput, query rate, latency, per-query durations, and normalized query-plan trees. Here, “CompareBench” denotes not a fixed dataset but an end-to-end comparative evaluation workflow (Choi et al., 2024).

At the meta-evaluation level, Benchmark$^2$ proposes three metrics for assessing benchmark quality itself: Cross-Benchmark Ranking Consistency, Discriminability Score, and Capability Alignment Deviation. For a benchmark $B_i$, CBRC is the average Kendall’s $\tau$ between its induced model ranking and peer benchmarks in the same domain; DS combines the normalized spread $\sigma_i/\bar{s}_i$ with the fraction of model pairs separated by more than $\epsilon=0.02$; CAD maps within-family inversion rate to a score in $(0,1]$ via $e^{-12\cdot \text{inv\_rate}}$. For CompareBench, adopting these diagnostics would test whether its rankings agree with peer benchmarks, whether it separates models effectively, and whether stronger family members systematically dominate weaker ones on individual items (Qian et al., 7 Jan 2026).

JudgmentBench addresses a different but relevant problem: evaluation in domains without verifiable ground truth. On 30 real-world legal tasks, comparative judgments recover the intended quality ordering substantially better than rubrics, with mean Spearman’s rank correlation 0.908 versus 0.150, while requiring less than half the annotation time. A plausible implication is that if future CompareBench-style resources move toward expert, judgment-rich domains, pairwise preference elicitation may be a stronger primary supervision signal than rubric scoring (Yang et al., 24 May 2026).

Earlier comparative benchmarking work supplies additional methodological precedents. Nested performance profiles show that standard solver benchmarking can be biased by ranking methods solely relative to the best solver, and propose layered recomputation and averaging to stabilize rankings among non-top solvers (Hekmati et al., 2018). SentiBench illustrates an apple-to-apple comparison design in NLP, evaluating 24 sentiment analysis methods across 18 labeled datasets and exposing strong domain dependence, rank variability, and the importance of coverage alongside accuracy (Ribeiro et al., 2015). Taken together, these works indicate that CompareBench is not only a single benchmark name but also part of a broader research program: constructing evaluations in which relative comparison is the object of measurement, the mechanism of annotation, or the basis of benchmark validation.