UNICBench: UNIfied Counting Benchmark for MLLM

Abstract: Counting is a core capability for multimodal LLMs (MLLMs), yet there is no unified counting dataset to rigorously evaluate this ability across image, text, and audio. We present UNICBench, a unified multimodal, multi level counting benchmark and evaluation toolkit with accurate ground truth, deterministic numeric parsing, and stratified reporting. The corpus comprises 5,300 images (5,508 QA), 872 documents (5,888 QA), and 2,069 audio clips (2,905 QA), annotated with a three level capability taxonomy and difficulty tags. Under a standardized protocol with fixed splits/prompts/seeds and modality specific matching rules, we evaluate 45 state-of-the-art MLLMs across modalities. Results show strong performance on some basic counting tasks but significant gaps on reasoning and the hardest partitions, highlighting long-tail errors and substantial headroom for improving general counting. UNICBench offers a rigorous and comparable basis for measurement and a public toolkit to accelerate progress.

• The paper introduces UNICBench, a unified benchmark with 14,301 evidence-grounded questions spanning image, text, and audio counting across pattern, semantic, and reasoning tasks.
• Evaluations of 45 MLLMs show sharp accuracy declines on hard, high-count cases, with only three image models exceeding 50% Hit@80% and Gemini-2.5-Pro-Thinking leading text counting at 76.1%.
• The benchmark demonstrates why success rate must accompany error metrics, as refusals can artificially lower MAE, while long-tailed scenes, dense speech, and formatting failures remain major challenges.

Motivation and scope

Counting is a foundational numerical capability that current multimodal LLMs (MLLMs) are expected to exhibit, yet the evaluation landscape treats it in a fragmented way. Visual counting datasets (ShanghaiTech, NWPU-Crowd, FSC-147) use heterogeneous annotation formats (points, boxes, density maps) and are not expressed as QA tasks; document QA benchmarks such as DocVQA and ChartQA touch on counting only as subproblems without addressing semantic deduplication or cross-segment aggregation; audio resources are dominated by sound event detection corpora like AudioSet rather than counting tasks. No existing benchmark simultaneously spans image, text, and audio counting under a single protocol.

UNICBench (2603.00595) addresses this gap with three design commitments: (i) comprehensive modality and task coverage, (ii) a canonical QA-evidence schema with evidence-first ground truth, and (iii) a standardized evaluation protocol with fixed splits, prompts, seeds, deterministic numeric parsing, and stratified reporting. The authors identify four obstacles to rigorous counting evaluation—coverage gaps, annotation heterogeneity, inconsistent protocols, and evaluation cost—and structure the benchmark around them.

Corpus construction

The corpus comprises 8,241 samples and 14,301 questions across three tracks:

Modality	Samples	Questions	Categories	Avg count	Count range
Image	5,300	5,508	49	63.53	[0, 10,294]
Text	872	5,888	12	120.40	[0, 71,176]
Audio	2,069	2,905	2	30.32	[0, 635]

The image track aggregates established sources (FSC-147, NWPU-MOC, CARPK, JHU-CROWD++, UCF-QNRF, ShanghaiTech, IOCfish5K, Global Wheat Head Detection, Snapshot Serengeti) plus manually annotated categories (screen panels, pens, birds, books, marbles, etc.), with resolutions from 234×180 to 6736×4640 pixels. Count distributions are heavily long-tailed, from sparse scenes (airplane mean 5.58) to extreme density (crowd mean 355.58, max 10,294). The text track is entirely self-collected across 12 categories—code, HTML, JSON, MusicXML, LaTeX, news, literary works, CSV, exam papers, official documents, ancient texts, and law—with lengths spanning 584 to over 8 million characters. Notably, ancient texts preserve classical Chinese and literary samples retain source-language phrasing (80 bilingual samples, 9.2%). The audio track combines DESED environmental sounds (sparse, 1.56 events/sample) and AliMeeting conversational speech (dense, 81.51 counts/sample), with sub-second temporal annotations at 0.01-second granularity.

Quality control uses dual independent annotation with arbitration, reported as achieving 100% annotation consistency, plus automated pre-counting tools (regex, syntax parsers) followed by manual verification for text. Every ground truth stores both gt_count and structured evidence (pixel coordinates, character spans, or temporal ranges) in a unified JSON schema.

Taxonomy

Two orthogonal axes organize the benchmark. The capability taxonomy formalizes levels by the operation required on the entity set $E$: Pattern (L1) is direct perceptual counting ($y = |E|$); Semantic (L2) applies attribute filters or identity aggregation ($y = |\{e \in E \mid P(e)\}|$); Reasoning (L3) imposes explicit rules or structural constraints ($y = g(|S_1|, \dots)$). The difficulty taxonomy stratifies by measurable attributes into Easy (counts 1–10), Medium (11–100), and Hard (>100), with modality-specific correlates such as occlusion, repetition, and event overlap stored per-sample in data cards.

The level distribution differs sharply by modality: images are dominated by Pattern-level questions (85.4%), while text skews toward Reasoning (43.7%) and Semantic (29.9%) levels, and audio balances Pattern (64.0%) against Semantic/Reasoning (36.0%). This asymmetry is deliberate—it reflects where each modality's genuine counting challenges lie—but it also means cross-modality comparisons of aggregate scores are not directly meaningful without conditioning on level.

Evaluation protocol

All models receive a uniform system prompt demanding a single numeric answer, with inference standardized at temperature 0.0, max_tokens 4096, and a 120-second timeout (with documented exceptions, e.g., GPT-5 family run at default temperature with minimal reasoning effort). Answer extraction follows a deterministic priority: numeric tokens inside <answer> tags first, then trailing standalone numbers, then the first parseable numeric token. Non-parsable outputs are excluded from error metrics but penalized through Success Rate—a design choice whose consequences become visible in the audio results.

Metrics comprise Success Rate, MAE/MSE, and Hit Rate at relative tolerances of 100% (exact match), 90%, and 80%, all reported overall and stratified by difficulty and capability level. In total, 45 state-of-the-art MLLMs were evaluated: 21 on images, 22 on text, and 13 on audio.

Results

Image track. Success rates are near-universal (many models at 100%), establishing that producing numeric output is not the bottleneck. Accuracy is: only three models exceed 50% Hit@80%—InternVL3_5-241B-A28B (57.1%), Gemini-2.5-Pro-Thinking (51.3%), and GPT-5-mini (50.5%)—and exact-match rates remain low throughout. Errors grow monotonically from Easy to Hard; hard-partition MAE reaches hundreds even for frontier models (e.g., 444.6 for Claude-Sonnet-4, 694.2 for Gemini-2.5-Flash). A notable result is that open-source models remain competitive: Qwen2.5-VL-7B achieves an overall MAE of 41.8, comparable to o4-mini (42.9) and better than several larger closed models. By capability, Reasoning tasks show the lowest absolute errors, but this is partly an artifact of small true counts rather than genuine competence—the paper states this explicitly. The authors attribute long-tail failures to a supervision gap (no per-instance supervision during instruction tuning), representation limits (patch-based tokenization compressing densely packed objects), and calibration/distribution shift.

Text track. Gemini-2.5-Pro-Thinking leads decisively with 63.3% exact match and 76.1% Hit@80%, well ahead of DeepSeek-R1-0528 (59.7%/71.3%) and o3 (61.1%/72.0%). Semantic-level tasks yield the smallest errors, indicating modern LLMs handle filtering and aggregation well, while Pattern-level strict-format matching is more error-prone and Reasoning remains the principal bottleneck and largest source of inter-model variance. An instructive comparison between DeepSeek-V3.1 thinking and non-thinking variants shows nearly identical median behavior but an overall MAE gap (46.9 vs. 75.3) driven almost entirely by a small set of extreme high-count samples where thinking mode succeeds—evidence that aggregate metrics on long-tailed count distributions can be dominated by a handful of cases.

Audio track. This track exposes a methodological subtlety: several GPT-Audio variants report MAE below 1, but only because their success rates hover near 60–64%—difficult items produce refusals or non-numeric outputs that are excluded from MAE. Models with near-perfect success rates (Qwen2.5-Omni-7B at 100%, Voxtral-mini at 99.8%) absorb these failures into much larger MAEs (29.2 and 29.5 respectively). The paper flags this explicitly, and it is arguably the clearest demonstration of why Success Rate must be read jointly with error metrics. Among fully comparable high-success models, Qwen2.5-Omni-7B's combination of 100% success and moderate MAE stands out. Error analysis further shows that audio errors are categorical rather than graded: exact-match, 10%, and 20% tolerance metrics barely separate because ground-truth counts are small, so any miss exceeds tolerance. Conversational speech ("talk") induces disproportionately larger errors than environmental sounds, implicating temporal segmentation of overlapping utterances—not numerosity magnitude—as the dominant difficulty.

Cross-cutting failure modes

Beyond numeric error, the paper catalogs format-level failures: Out-of-Context responses (13,290 instances total, concentrated in text and audio, including Azure's 20 MB file-size limit truncating long audio), Out-of-Thinking traces breaking template extraction (1,821), and Incorrect Format refusals (3,298). Text exhibits the highest formatting-failure rate, attributed to longer prompts and linguistically complex questions. Category-level heatmaps consistently identify crowd, tree, and bottle caps (image) and LaTeX/code (text) as persistent weak spots across nearly all models, pointing to structural causes rather than model-specific idiosyncrasies.

Limitations and open questions

The paper acknowledges dataset bias and evolving tooling as limitations. Several specific caveats bear on interpretation. First, the audio track contains only two categories (environment, talk), limiting generalization of audio conclusions. Second, the capability-level confound—Reasoning tasks having small true counts—means per-capability MAE comparisons across levels should not be read as a pure difficulty ranking. Third, some models could not be run under identical decoding settings (temperature defaults, reduced reasoning effort for cost), introducing residual protocol heterogeneity despite the unified framework. Fourth, the claim of "100% annotation consistency" reflects the dual-annotation-with-arbitration procedure rather than independent inter-annotator agreement, so annotation subjectivity on ambiguous instance-inclusion rules (a problem the paper itself identifies in prior datasets) is managed but not quantified. Open questions include whether hybrid detector-plus-LLM pipelines close the dense-scene gap, how evidence-first outputs affect numeric accuracy, and whether improved cross-modal alignment transfers counting skill across modalities.

Conclusion

UNICBench provides the first modality-spanning, taxonomy-stratified counting benchmark for MLLMs, with 14,301 QA pairs, evidence-first ground truth, and a reproducible evaluation toolkit. Its central empirical finding is consistent across modalities: MLLMs handle many Pattern and Semantic counting tasks adequately but degrade sharply on Reasoning-level and Hard partitions, with long-tail high-count cases dominating aggregate error. The benchmark's stratified reporting and joint Success-Rate/error accounting expose failure modes—refusal-driven metric inflation, tolerance-insensitive categorical audio errors, thinking-mode gains confined to extreme counts—that unstratified evaluation would obscure.