TallyBench: Vision-Language Counting Benchmark

• TallyBench is a dedicated counting dataset pairing 2,000 real-world images with a single counting question to evaluate numerosity skills in vision-language models.
• It employs strict annotation protocols and an exact-match accuracy metric, emphasizing precise counting in complex, cluttered scenes.
• The dataset forms the quantity subtask of CompareBench, revealing model failure modes in occlusion handling, object overlap, and scale variation.

Searching arXiv for TallyBench / CompareBench and related naming ambiguity. TallyBench is a counting dataset for evaluating the numerosity skills of vision–LLMs in real-world imagery. It consists of 2 000 real-world images, each paired with a single counting question, and was introduced as one of two auxiliary datasets used to derive CompareBench, a benchmark for visual comparison reasoning in vision-LLMs. Within that broader framework, TallyBench serves both as a standalone benchmark and as the foundation for the quantity-comparison subtask in CompareBench (Cai et al., 25 Sep 2025).

1. Position within CompareBench

CompareBench evaluates visual comparison reasoning in vision-LLMs across four tasks: quantity, temporal, geometric, and spatial. It consists of 1000 QA pairs and is derived from two auxiliary datasets: TallyBench and HistCaps. In this construction, TallyBench provides the counting substrate from which quantity-oriented comparison instances are formed (Cai et al., 25 Sep 2025).

TallyBench is therefore not a general-purpose multimodal corpus but a diagnostic evaluation resource targeting counting under realistic visual variability. Its design isolates exact object numerosity estimation rather than free-form description, open-ended reasoning, or approximate quantity judgment. This suggests that TallyBench occupies a specific methodological niche: it operationalizes counting as a tightly controlled QA problem in order to expose failure modes that may be obscured in broader VLM benchmarks.

A naming ambiguity is possible because TailBench++ is a separate benchmark suite for latency-critical cloud workloads. TailBench++ extends TailBench with dynamic multi-client and multi-server benchmarking, whereas TallyBench denotes the counting-image dataset used in CompareBench; the two resources address unrelated problem domains (Li et al., 6 May 2025).

2. Dataset composition and sourcing

TallyBench consists of 2 000 real-world images, each paired with a single counting question. The images were curated to span two broad domains—“Biology” with 900 images and “Artificial Objects” with 1 100 images—and are further subdivided into about fifty fine-grained object classes (Cai et al., 25 Sep 2025).

Examples of fine-grained classes include animals such as Dog, Cat, Chicken, Sheep/Goat, and Birds; plants and food/beverage categories such as Apple, Banana, and Cups/Glasses; and man-made objects such as Books, Spoons, Knives, Electronics, Clothing, Transportation, Household items, and Balls. Reported per-class counts range from 40, as in Dog, up to 100, as in Chicken and Book (Cai et al., 25 Sep 2025).

Although the precise web sources are not itemized, the images were selected to provide challenging real-world variability, including occlusion, perspective changes, various lighting and resolution, and mixtures of physical objects and printed or drawn representations. This choice is significant because it prevents the benchmark from collapsing into a sanitized counting task dominated by canonical object views. A plausible implication is that TallyBench probes robustness to the kinds of visual clutter and representational heterogeneity that commonly degrade VLM counting performance.

Property	Value
Total images	2 000
Total QA pairs	2 000
Domains	Biology: 900; Artificial Objects: 1 100

The paper does not describe separate training, validation, or test splits for TallyBench. It is treated as a single held-out evaluation set, and all reported model accuracies use the full 2 000 images (Cai et al., 25 Sep 2025).

3. Question format and annotation protocol

For each image, annotators wrote a single, unambiguous counting question of the form “How many X are in the image?” where $X$ is the target category. A ground-truth integer answer was recorded in a JSON entry together with metadata fields: image_name, vlm_question, gt_answer, categories, and image_type (Cai et al., 25 Sep 2025).

The dataset enforces a unified instruction template for all questions. This template defines the operative counting policy through five rules: count every clearly identifiable, distinct instance of the target object; include both physical objects and visual representations; count partially occluded, cropped, or blurred objects if they are still recognizable; do not count reflections; and provide exactly one integer with no additional words or explanations (Cai et al., 25 Sep 2025).

These rules are central to the benchmark’s interpretability. They reduce annotation variance by fixing edge-case behavior for visual representations, partial visibility, and reflections—three recurrent sources of disagreement in counting tasks. During dataset assembly, the same rules were used both to guide annotators and to perform cross-checks. A small subset of images was annotated by multiple workers to verify inter-annotator agreement, which is reported qualitatively to be near 100 % on clear cases, with most disagreements arising only on highly ambiguous occlusions (Cai et al., 25 Sep 2025).

The requirement that models output exactly one integer is methodologically important. It suppresses answer-format variability and turns evaluation into a pure exact-match problem. This suggests that TallyBench is designed not merely to test recognition of the queried category, but to decouple numerical correctness from linguistic fluency.

4. Evaluation criterion and metric

TallyBench uses Accuracy (%) under an exact-match criterion. A prediction is correct if and only if the model’s output integer exactly equals the ground-truth integer. Let $N = 2\,000$ be the number of samples, and let $c$ denote the number of exact matches. Then the accuracy is defined as

$A = \frac{c}{N} \times 100\%.$

The paper also gives the equivalent indicator-function form:

$$A = \frac{1}{N} \displaystyle\sum_{i=1}^N \mathbf{1}\{ \hat{y}_i = y_i \}\times 100\%,$$

where $y_i$ is the ground-truth count for sample $i$, $\hat{y}_i$ is the predicted count, and $\mathbf{1}\{\cdot\}$ is the indicator function (Cai et al., 25 Sep 2025).

No precision/recall or other continuous-error metrics, such as mean absolute error, are reported. TallyBench therefore privileges exact numerosity over approximate count quality. This evaluation choice has two consequences. First, it makes the benchmark stringent: an answer off by one is fully incorrect. Second, it emphasizes whether a model can produce the exact discrete count under realistic clutter rather than whether it can estimate count magnitude coarsely.

Because all reported accuracies are computed on the full held-out set, the benchmark functions as a direct comparative probe of model counting behavior rather than as a trainable supervised learning task with standard split-based generalization analysis (Cai et al., 25 Sep 2025).

5. Difficulty profile and representative failure modes

The paper reports six “hard cases” on which all four strong models failed. These include questions such as “How many spoons are in the image?” with ground-truth answer 12 and model outputs like 9 or 15; “How many Labubu plush toys are in the image?” with ground-truth answer 7; “How many chickens are in the image?” with ground-truth answer 9; “How many books are in the image?” with ground-truth answer 5; “How many birds are in the image?” with ground-truth answer 8; and another chicken scene with ground-truth answer 6 (Cai et al., 25 Sep 2025).

The associated challenges are explicit: partially occluded spoons arranged in stacks, visually intricate stuffed toys, overlapping legs and eggs in the scene, stacked books with very narrow visible spines, perches with varying scales and partial occlusion, and dense arrangements with camouflage against the background. From these cases, the paper identifies four principal failure modes: missing or double-counting partially occluded objects, confusing visually similar instances, failing to parse thin or stacked objects, and over- or under-estimating when objects vary dramatically in scale (Cai et al., 25 Sep 2025).

These failure modes are informative because they indicate that counting errors are not reducible to a single deficit. They implicate object individuation, occlusion handling, scale normalization, and category discrimination. A plausible implication is that TallyBench measures a compound competence spanning detection, segmentation-like separation of instances, and cardinality extraction, rather than a narrow arithmetic skill.

6. Reported model performance and interpretive significance

Reported accuracies on TallyBench span both closed-source APIs and open-source vision-LLMs. Closed-source APIs range from approximately 52 % up to approximately 87 %, with Gemini 2.5 Pro reported as the best at 87.35 % and GPT-5 (mini) at 80.25 %. Open-source models span approximately 56 % to approximately 82 % as they scale up, with Qwen2.5-VL-3B at 56.05 %, Qwen2.5-VL-72B at 75.60 %, and Qwen3-VL-235B-A22B at 82.25 %. Human performance, under the same rules, is measured at 98.00 % (Cai et al., 25 Sep 2025).

System category	Example model	Accuracy
Closed-source API	Gemini 2.5 Pro	87.35 %
Closed-source API	GPT-5 (mini)	80.25 %
Open-source VLM	Qwen2.5-VL-3B	56.05 %
Open-source VLM	Qwen2.5-VL-72B	75.60 %
Open-source VLM	Qwen3-VL-235B-A22B	82.25 %
Human	Annotators following the same rules	98.00 %

The results show clear scaling trends, but the remaining gap to human performance is substantial. The reported interpretation is that contemporary VLMs still err on counting tasks that humans deem trivial, often missing small, occluded, or overlapping objects (Cai et al., 25 Sep 2025).

Within CompareBench more broadly, these results contribute to the claim that visual comparison remains a systematic blind spot for current VLMs. TallyBench is especially relevant to that conclusion because quantity comparison presupposes reliable extraction of exact counts. This suggests that weak performance on TallyBench can propagate into downstream comparison failures even when the linguistic form of the comparison question is simple.

7. Methodological role and limitations

TallyBench is described as a rigorously annotated, 2 000-image counting dataset designed to isolate and evaluate the numerosity skills of vision-LLMs. Its strict single-integer output protocol and broad category coverage define it as both a standalone benchmark and an enabling resource for CompareBench’s quantity subtask (Cai et al., 25 Sep 2025).

Several methodological constraints follow directly from its design. Because the benchmark uses one counting question per image, it does not assess multi-question consistency on the same visual scene. Because it reports only exact-match accuracy, it does not characterize error magnitude. Because no training, validation, or test splits are described, it functions as a held-out evaluation resource rather than a conventional supervised benchmark with split-wise learning protocols (Cai et al., 25 Sep 2025).

These limitations should not be conflated with weaknesses in annotation rigor. Rather, they delimit the benchmark’s purpose. TallyBench is optimized for controlled diagnosis of counting behavior under realistic visual conditions. Its contribution lies in making counting failures measurable in a format compatible with modern VLM evaluation pipelines, while preserving enough real-world complexity to reveal systematic breakdowns in occlusion handling, visual disambiguation, and exact cardinality estimation.