PairTally: Fine-Grained Visual Counting

• PairTally is a benchmark dataset for fine-grained, intent-driven visual counting, designed to test models' ability to count the specified object among similar distractors.
• The dataset comprises 681 high-resolution images with exactly two object categories per image, enabling evaluation in both inter-category and intra-category settings.
• Empirical findings reveal that models often overcount or default to priors, highlighting challenges in prompt sensitivity and discriminating subtle visual attributes.

PairTally is a benchmark dataset for fine-grained, intent-driven visual counting. It was introduced to test whether a model can count the specific object that a prompt refers to when multiple similar objects are present in the same scene, rather than merely counting repeated or salient objects. The benchmark contains 681 high-resolution images, and each image contains exactly two object categories. It includes both inter-category and intra-category settings, so it evaluates selective counting both when the competing categories are clearly different and when they are closely related subcategories distinguished by subtle differences in shape, size, color, or semantics (Nguyen et al., 17 Sep 2025).

1. Definition and motivation

PairTally was motivated by a limitation in prior counting datasets and methods: many systems can estimate how many objects appear in an image in a broad sense, but do not reliably count what a prompt actually specifies when visually confusable alternatives co-occur. The benchmark therefore targets fine-grained, intent-driven counting rather than generic object numerosity (Nguyen et al., 17 Sep 2025).

The task formulation is explicitly selective. A model is asked to count one category in an image that also contains a distractor category. The benchmark is intended to expose whether a system can follow the counting prompt, distinguish the intended object from a distractor object in the same scene, and handle fine-grained differences such as shape, size, color, or semantics. The underlying concern is that many systems default to counting the most salient or frequent objects rather than the user’s intended target.

PairTally operationalizes this problem with natural, real-world scenes rather than synthetic composites. Because each image contains exactly two object categories, the benchmark does not reduce to ordinary open-vocabulary detection or single-class counting. Instead, it tests prompt-conditioned discrimination under co-occurrence, clutter, and fine-grained ambiguity.

2. Relation to prior counting benchmarks

The benchmark is positioned against several earlier datasets that evaluate counting under different assumptions (Nguyen et al., 17 Sep 2025). FSC-147 and FSCD-147 provide broad category diversity, but each image mostly contains instances from a single object class, so multiclass selective counting is only weakly tested. OmniCount-191, CountBench, FSCD-LVIS, and MCAC are broader than FSC-147, but they do not provide controlled, real-world images that require discrimination between two object types simultaneously, including visually similar subtypes. PrACo probes prompt sensitivity, but its images are synthetically composed mosaics rather than naturally co-occurring objects in the same physical scene.

The central novelty of PairTally is therefore not only that it is a counting dataset, but that it uses natural scenes in which one of two categories must be counted while the other acts as a distractor. This makes the benchmark diagnostic for selective counting failure modes that can remain hidden on single-class or synthetically composed data.

A common misconception is that strong performance on standard counting datasets implies robust prompt following in mixed scenes. PairTally was introduced precisely because that implication does not hold in general. Its design isolates the discrepancy between counting “objects in general” and counting “the object meant by the prompt.”

3. Dataset composition and scene design

PairTally is organized into five supercategories: Food, Fun, Household, Office, and Other (Nguyen et al., 17 Sep 2025). Across these, it covers 54 object categories and 98 subcategories total. The benchmark construction began with 100 subcategory pairs, consisting of 50 intra-category pairs and 50 inter-category pairs. Three intra-category pairs were then removed because they were too ambiguous, leaving a final benchmark of 97 subcategory pairs.

The images are mainly drawn from tabletop and household contexts and were captured with consumer smartphones, specifically an iPhone 12 and a Samsung S21 Plus. The scenes were designed to include natural variation in lighting, perspective, occlusion, and scene complexity. These acquisition choices matter because the benchmark is intended to reflect real-world selective counting rather than laboratory-style isolation of target instances.

The subcategory distinctions are explicitly fine-grained. The paper states that the split is based on color in 43.5% of cases, texture/shape in 42.5%, and size in 14.1%. Representative examples include spiral pasta vs. penne pasta, citrus vs. calamansi lime, black vs. white checker pieces, big vs. small marbles, AAA vs. AA batteries, 5¢ vs. 10¢ coins, hex vs. mushroom bolts, hex vs. square nuts, and pens with vs. without cap. The paper also notes that a few categories have no subcategories.

The inter-category/intra-category distinction is fundamental. Inter-category scenes involve two distinct categories in one image, such as mahjong tiles vs. poker chips. Intra-category scenes instead require discrimination among closely related subcategories of the same broader class. The latter setting is intended to be harder because the model must attend to a subtle attribute rather than broad object identity.

4. Task formulation and evaluation protocol

The benchmark evaluates 10 models in three broad classes (Nguyen et al., 17 Sep 2025). The exemplar-based counting models are FamNet, DAVE, GeCo, and LoCA. The language-prompted or open-vocabulary counting systems are CountGD, evaluated in both exemplar-plus-text and text-only modes, and LLMDet, evaluated in text-only mode. The large vision-LLMs are Ovis2, Qwen2.5-VL, LLaMA-3.2, and InternVL3, specifically the 16B, 7B, 11B, and 14B variants listed in the paper.

For exemplar-based methods, each scene provides three exemplar bounding boxes. For VLMs, the prompt template is:

“Count the number of {object} in this image. Provide only the total count in this format: <count>N</count>. If you see no {object} or are unsure, respond with <count\>0</count>.”

The placeholder is filled with the target reference, such as “red poker chips.”

The paper formalizes several evaluation quantities. If category $A$ has ground-truth count $a$, the absolute error for a prompt targeting $A$ is $|f(A) - a|$. If the prompt asks for both $A$ and $B$, with total count $a+b$, the corresponding error is $|f(A+B) - (a+b)|$. To probe distractor sensitivity and prompt responsiveness, the benchmark also uses $|f(A) - (a+b)|$, which tests whether a model effectively counts both categories when only $A$ is requested, and $a$0, which tests whether the model’s prediction changes when the prompt changes.

The reported scene-level metrics are MAE and NAE, with

$a$1

NAE is used to compare performance across scenes with different count magnitudes. The benchmark also separates performance by inter-category and intra-category subsets, and further stratifies intra-category cases by the attribute that differs.

5. Empirical findings and characteristic failure modes

The benchmark results indicate that current models struggle to count what the prompt intends in mixed, fine-grained scenes (Nguyen et al., 17 Sep 2025). The paper identifies two broad and distinct failure patterns.

First, specialist counting models frequently overcount. Exemplar-based methods such as DAVE, GeCo, LoCA, and FamNet often produce predictions larger than the target count for the requested category, suggesting that they drift toward counting all repeated objects rather than the specified subset. The reported overcount rates $a$2 are 74.0% for DAVE, 84.7% for GeCo, 83.1% for LoCA, and 73.1% for FamNet. For many of these models, the inequality $a$3 occurs often, meaning that the nominal single-category prediction is closer to the total number of both categories than to the requested subset.

Second, the VLMs are generally weaker at enumeration itself. The paper describes their outputs as erratic and unstable, with guesses often weakly correlated with the true count. Their overcount rates are much lower—5.0% for Ovis2, 26.1% for Qwen2.5-VL, 12.9% for LLaMA-3.2, and 9.1% for InternVL3—but this lower overcount rate does not indicate strong selective counting. Rather, the models often default to mid-to-high count guesses regardless of the image, which the paper interprets as reliance on priors rather than reliable counting.

The inter-category/intra-category split further sharpens the diagnosis. Most models perform worse in the intra-category setting. The paper highlights DAVE and CountGD (Text) as among the most robust, while CountGD and GeCo show large performance drops in intra-category scenes, indicating distractor sensitivity. LLMDet is reported to perform especially poorly on intra-category cases.

Attribute type also matters. On the intra-category subsets, color is the easiest distinction overall, size is the hardest, and texture/shape is intermediate. The paper gives representative NAE values illustrating this ordering: for DAVE, color NAE is 0.738, size NAE is 1.293, and texture/shape NAE is 0.693; for CountGD (Text), the corresponding values are 0.760, 1.410, and 0.735; for GeCo they are 0.791, 1.345, and 0.946.

The study also reports that prompt engineering did not solve the underlying problem. Negation prompts, more detailed descriptions, and bounding-box coordinates expressed as textual references did not improve performance and often made it worse. This is important because it rules out a simple explanation in which the observed failures arise only from underspecified prompts.

6. Subsequent use as a transfer benchmark

A later paper, "The MixCount Dataset: Bridging the Data Gap for Open-Vocabulary Object Counting" (Dumery et al., 18 May 2026), uses PairTally as a principal real-world benchmark for mixed-object counting. In that work, PairTally is described as a test set for fine-grained object counting with 681 dense images and mixed categories, and it is singled out because FSC-147 labels only one object type per image whereas PairTally contains images with two object types labeled per image. This makes PairTally a more direct test of whether a model can distinguish visually similar categories and count the correct one in cluttered mixed scenes.

The MixCount study evaluates CountGD++ on PairTally using MAE and RMSE, following prior counting papers. The baseline CountGD++ model is trained on FSC-147 + 1000 mosaicked FSC-147 images, while CountGD++ (+ MixCount) is trained on FSC-147 + the 1000 mosaicked images + 1000 MixCount training images. At inference, the same adaptive cropping and test-time normalization from CountGD++ are used. For PairTally, confidence thresholds are tuned on the MixCount validation set and set to 0.75 when only positive prompts are provided and 0.4 when both positive and negative prompts are provided. The PairTally prompting setup uses positive text $a$4, positive internal exemplars $a$5, negative text $a$6, and negative internal exemplars $a$7.

The reported PairTally results are:

Setting	Baseline CountGD++	CountGD++ (+ MixCount)
Full prompting MAE / RMSE	35.33 / 60.83	28.87 / 54.82
Weaker-prompt MAE / RMSE	46.41 / 69.52	43.36 / 67.02

The reduction from 35.33 to 28.87 MAE corresponds to an 18.3% reduction in MAE, while the weaker-prompt setting improves by 6.6%. The paper presents this as evidence that synthetic training on MixCount transfers to a real benchmark that the original FSC-147-trained model struggles with. In that sense, PairTally functions not only as a diagnostic dataset but also as a stress test for whether new training data or modeling strategies actually improve fine-grained, open-vocabulary counting in realistic mixed-object scenes.