CountBench: Multimodal Numeracy Benchmark

• CountBench is a multimodal benchmark that measures a model's ability to align spelled-out numeric expressions with the corresponding number of objects in images.
• It has been repurposed for category-specific object counting in multimodal reasoning systems, emphasizing detect-then-count protocols and structured reasoning.
• A distinct text-to-video variant of CountBench tests numeral-to-instance alignment in generated videos using metrics like CountAcc, temporal consistency, and CLIP scores.

Searching arXiv for the most relevant CountBench papers and later works that use or redefine the benchmark. CountBench is a benchmark name used in more than one recent research context, but it most commonly denotes the image–text counting benchmark introduced in “Teaching CLIP to Count to Ten” for probing whether a vision–LLM aligns textual count expressions with visible object multiplicity (Paiss et al., 2023). In later work, the same benchmark is reused as a category-specific object-counting testbed for multimodal reasoning systems (Liu et al., 17 May 2025, Xu et al., 8 Jun 2026). Separately, “When Numbers Speak: Aligning Textual Numerals and Visual Instances in Text-to-Video Diffusion Models” introduces another benchmark also called CountBench, this time for numeral-to-instance alignment in text-to-video diffusion outputs (Sun et al., 9 Apr 2026). The shared name therefore refers not to a single fixed artifact, but to at least two distinct numeracy benchmarks spanning image–text alignment and text-to-video generation.

1. Historical origin and nomenclature

The historically earlier CountBench was introduced with “Teaching CLIP to Count to Ten” as a small, carefully curated, test-only benchmark targeting a persistent compositional weakness of CLIP-like vision–LLMs: recognizing and aligning the number of object instances in an image with count expressions in text (Paiss et al., 2023). Its task formulation is image–text alignment under explicit numerals, not generic VQA. The benchmark uses image–text pairs where the caption explicitly states a count and the image clearly depicts that count.

Subsequent papers adopt this original CountBench as an object-counting benchmark for multimodal reasoning. VisionReasoner treats CountBench as a category-specific object counting benchmark in which success depends on multi-object localization, de-duplication, and tallying, explicitly framing counting as “detect-then-count” (Liu et al., 17 May 2025). Visual Para-Thinker++ similarly uses CountBench as a canonical benchmark for precise object counting in natural images and evaluates on its official test split with accuracy as the reported metric (Xu et al., 8 Jun 2026).

A later and distinct reuse of the name appears in the text-to-video literature. “When Numbers Speak” introduces a purpose-built CountBench to expose and measure the failure of text-to-video diffusion models to align textual numerals with the actual number of visual instances generated (Sun et al., 9 Apr 2026). This benchmark targets generated videos rather than natural images, and its design and metrics differ substantially from the original image–text CountBench.

This multiplicity of meanings is central to interpreting claims about “CountBench” in papers. A plausible implication is that benchmark citations must be read together with task modality and protocol, because the name alone is no longer unambiguous.

2. The original image–text CountBench

The original CountBench is curated from the publicly available LAION-400M dataset and is designed as a focused probe of numeric compositionality for contrastive vision–LLMs (Paiss et al., 2023). Its scope is narrow by design: it covers spelled numbers two to ten, excludes zero and one, and excludes non-spelled numerals such as “2” or “11.” Captions contain a spelled-out number plus a noun phrase for the counted object, and natural pluralization appears as in typical web captions.

Its construction pipeline is explicitly filtered. First, captions are retained only if they contain spelled numbers in $\{\text{two},\dots,\text{ten}\}$, while digits and higher numbers are discarded to avoid time, dates, and other non-count references. Second, visual verification with an off-the-shelf detector requires the spelled number to match the number of instances of the most prevalent detected object in the image. Third, a manual verification pass removes ambiguities and errors. Fourth, the final set is rebalanced so that each count bucket has an equal number of samples (Paiss et al., 2023).

The resulting benchmark contains 540 image–caption pairs, with exactly 60 pairs for each spelled count from two to ten, yielding nine balanced buckets and no train/validation/test split. It is explicitly test-only. The object vocabulary is broad and open-vocabulary, including animals, foods, and everyday objects. At higher counts, the dataset skews toward simpler 2D collections and grid-like layouts, reflecting what is available and reliably annotatable in web data.

A key design choice is that the released benchmark contains only correct image–text pairs. Counterfactual captions with incorrect counts are generated on the fly by swapping the number word while keeping the rest of the caption fixed. This makes the number token the only changed linguistic variable, isolating count alignment rather than broader caption semantics (Paiss et al., 2023).

3. Task formulation and evaluation in the original benchmark

The original CountBench supports a multi-way count-classification protocol. For each image, the evaluator forms nine candidate captions by replacing the original number with each word in $\{\text{two},\dots,\text{ten}\}$ and computes an image–text similarity score $s(I,T_n)$. The predicted count is the one attached to the highest-scoring caption (Paiss et al., 2023).

The reported primary metrics are top-1 accuracy and mean absolute error:

$$\mathrm{Acc} = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\!\left[\arg\max_n s(I_i,T_i(n)) = n_i^{\text{true}}\right]$$

and

$$\mathrm{MAE} = \frac{1}{N}\sum_{i=1}^{N}\left|\hat{n}_i - n_i\right|.$$

An alternative equivalent summary is pairwise count-consistency accuracy, defined by comparing the score of the correct caption against a single counterfactual caption with an incorrect number. No retrieval metrics such as $R@k$ are defined for CountBench (Paiss et al., 2023).

The benchmark was introduced together with a counting-contrastive finetuning objective for CLIP-like models. In the reported results, CLIP-B/32 improves from 31.67% accuracy and 1.53 MAE for the official baseline to 75.93% accuracy and 0.49 MAE for the proposed filtered counting set plus counting-contrastive loss. BASIC-S improves from 17.97% accuracy and 2.13 MAE to 69.02% accuracy and 0.64 MAE under the same proposed training recipe (Paiss et al., 2023). The accompanying ablations show that both clean filtering and the counting-contrastive loss are necessary, and that performance tends to drop for higher counts where training data is scarcer.

The benchmark’s intended interpretation is therefore diagnostic rather than encyclopedic in coverage: it is a compact, high-precision probe of whether a model truly aligns spelled number words with visual multiplicity.

4. Repurposing in multimodal reasoning systems

Later multimodal reasoning papers reuse CountBench in a different operational form. In VisionReasoner, CountBench is one of ten tasks spanning detection, segmentation, and counting, and the evaluation is purely zero-shot: no CountBench samples are used for training (Liu et al., 17 May 2025). The paper reports evaluation only on its test split, with 504 test samples, under a unified multi-modal conversation format. For CountBench, explicit numerical references are removed from the text descriptions to prevent number parroting, and the model is required to output a structured reasoning trace enclosed by > ... and a final answer enclosed by <answer> ... </answer>, with only the final count being scored (Liu et al., 17 May 2025).

For counting in VisionReasoner, the sole metric is exact-match count accuracy,

$$\mathrm{Acc} = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}[\hat{y}_i = y_i].$$

The model approaches counting as detect-then-count: given image $I$ and text query $T$, it predicts a structured set of bounding boxes and center points $\{(B_i,P_i)\}_{i=1..N}$ for all objects matching $\{\text{two},\dots,\text{ten}\}$0, produces a reasoning chain, and then outputs the count $\{\text{two},\dots,\text{ten}\}$1 (Liu et al., 17 May 2025). VisionReasoner-7B reaches 87.6% accuracy on CountBench-test, compared with 78.8% for LLaVA-OneVision-7B, 76.5% for Qwen2-VL-7B, and 76.0% for Qwen2.5-VL-7B (Liu et al., 17 May 2025).

Visual Para-Thinker++ uses CountBench in another reasoning-centric setting. The task is an open-ended question of the form “How many [objects] are in the image?”, and the model must output a single integer as the final answer (Xu et al., 8 Jun 2026). Its counting protocol uses $\{\text{two},\dots,\text{ten}\}$2 Worker Agents with scan-order allocation: left-to-right, right-to-left, top-to-bottom, and bottom-to-top. Workers operate under context isolation, each seeing only the image, question, and the Main segment, while the Summary Agent reconciles full Worker reasoning traces rather than majority-voting on final labels (Xu et al., 8 Jun 2026). On CountBench, Visual Para-Thinker++ reports 74.8% test accuracy with Qwen2.5-VL-3B and 88.3% with Qwen2.5-VL-7B, outperforming the corresponding base Qwen2.5-VL models and several single-trajectory or inference-time parallel baselines (Xu et al., 8 Jun 2026).

These later works reposition CountBench from an image–text similarity probe into a benchmark for explicit visual reasoning, localization, grouping, and count reconciliation. This suggests that CountBench has become a transferable evaluation target for object-centric numeracy, even when the modeling paradigm shifts from contrastive scoring to structured reasoning and RL-trained multi-object cognition.

5. The text-to-video CountBench

A distinct benchmark with the same name is introduced in “When Numbers Speak” for text-to-video diffusion models (Sun et al., 9 Apr 2026). This CountBench is purpose-built to expose and measure a specific failure mode: whether the integer in the prompt, ranging from 1 to 8, matches the number of object instances rendered in the generated video, including multi-category compositions where more than one count must be satisfied.

Its benchmark design differs fundamentally from the original image–text version. It contains 210 curated prompts spanning instance counts from 1 to 8 and involving 1–3 object categories per prompt. Candidate prompts are first produced by a LLM to ensure variety and simple, dynamic descriptions, then manually reviewed to remove repetitive or illogical items. The benchmark is intentionally compositional and stresses numeracy in scenes where counts for multiple categories must be satisfied simultaneously. Temporal dynamics are implicit because outputs are videos and the evaluation considers frame-wise counts and their stability over time; the prompts themselves are text-only synthetic prompts for generation rather than paired with real video data. The paper does not define train/validation/test splits; all 210 prompts form a single evaluation set (Sun et al., 9 Apr 2026).

The evaluation protocol uses three complementary video-level metrics. Counting Accuracy (CountAcc) measures exact adherence to the specified counts by using GroundingDINO with category-specific text queries to obtain per-frame counts, scoring each frame and category as 1 if the detected count equals the prompt’s count and 0 otherwise, then averaging across categories and frames. Temporal Consistency (TC) quantifies count stability over adjacent frame pairs. CLIP Score averages frame-wise CLIP-based text–visual alignment scores across the video, following CLIPScore. Supplementary analysis also reports VBench “Subject-Consistency,” based on DINO features per instance per frame and cosine similarity to the first and previous frames (Sun et al., 9 Apr 2026).

On this CountBench, NUMINA is evaluated as a training-free identify-then-guide framework. Across Wan scales, it improves CountAcc from 42.3% to 49.7% on Wan2.1-1.3B, from 47.8% to 52.7% on Wan2.2-5B, and from 53.6% to 59.1% on Wan2.1-14B, while maintaining or modestly improving temporal coherence and CLIP alignment (Sun et al., 9 Apr 2026). The paper’s per-numeral analysis shows that failures amplify with higher counts: for Wan2.1-1.3B, baseline accuracy is 68.7% for two objects but 44.5% for three, and at eight objects baseline is 11.3% while NUMINA reaches 20.7% (Sun et al., 9 Apr 2026).

This CountBench therefore operationalizes a different notion of numeracy: not recognition of counts in existing images, but alignment between textual numerals and latent instance layouts in generated videos.

6. Variants, limitations, and recurrent points of confusion

The two CountBench variants can be summarized as follows.

Variant	Domain	Core protocol
Original CountBench	Image–text counting	540 image–caption pairs, counts two–ten, nine-way caption substitution, top-1 accuracy and MAE (Paiss et al., 2023)
Reused CountBench in LVLM reasoning	Natural-image object counting	Official test split, exact-match accuracy, question-answer or detect-then-count formulations (Liu et al., 17 May 2025, Xu et al., 8 Jun 2026)
Text-to-video CountBench	T2V numeral alignment	210 prompts, counts 1–8, 1–3 categories, CountAcc, TC, and CLIP Score (Sun et al., 9 Apr 2026)

Each version also has explicit limitations. The original CountBench is modest in size, limited to counts two through ten, excludes zero and one, and shows a bias toward simpler or grid-like scenes at higher counts (Paiss et al., 2023). The text-to-video CountBench is limited to 210 prompts, count ranges 1–8, and up to 3 categories, and does not cover extreme-density scenes or exhaustive domain diversity; detector reliability and attention maps focusing on salient parts rather than whole instances may introduce noise or over-segmentation (Sun et al., 9 Apr 2026).

A notable ambiguity concerns dataset size in the reused image-counting setting. The original CLIP paper describes CountBench as 540 image–caption pairs and test-only (Paiss et al., 2023), whereas VisionReasoner evaluates “CountBench-test” with 504 samples (Liu et al., 17 May 2025). The provided materials do not reconcile this discrepancy. A plausible implication is that later papers may adopt a particular processed test split or standardized evaluation subset, but that interpretation is not specified in the cited texts.

Another recurrent confusion is terminological rather than methodological. CounterBench is a different benchmark for formal counterfactual reasoning in LLMs and is sometimes misspelled “CountBench” in discussion, but it targets structural causal model reasoning rather than visual counting (Chen et al., 16 Feb 2025). For technical reading, this distinction is essential.

CountBench thus functions less as a single immutable benchmark than as a family name for numeracy-focused evaluation artifacts. In the image–text line of work, it probes whether models align number words with visible multiplicity. In the text-to-video line, it probes whether generated instance counts match prompt numerals over time. Across both lines, the common research objective is the same: isolating numerical alignment as a first-class evaluation axis rather than treating it as a minor subcase of general semantic matching.