YOLO-Count: Differentiable Object Counting

• The paper introduces a regression-based cardinality map that ensures scale-invariant count supervision and interpretable per-pixel contributions.
• It employs a vision–language fusion architecture that integrates YOLOv8-L features with CLIP text embeddings via T-CSPLayer modules for robust multimodal learning.
• The model facilitates gradient-based count control in text-to-image pipelines, reducing average count error from ~30% to below 10% in benchmark tests.

YOLO-Count is a differentiable, open-vocabulary object counting model designed to address general counting challenges and provide precise quantity control for text-to-image (T2I) generation. Its core innovations include a regression-based “cardinality map” for interpretable and scale-aware counting, a vision–language fusion architecture, and a hybrid supervision framework. YOLO-Count achieves state-of-the-art accuracy on standard counting and T2I-guided benchmarks and enables fully differentiable, gradient-based end-to-end control over object counts in image generation pipelines (Zeng et al., 1 Aug 2025).

1. Cardinality Map Formalism

YOLO-Count introduces the “cardinality map” as a regression target distinct from Gaussian density maps. For an image containing $K$ annotated object instances, with each instance $i$ having a binary mask $M_i$ ($M_i \in \{0,1\}^{H\times W}$), let $N_i = |M_i|$ denote its area. The pixelwise cardinality contribution is defined as: $$y_{\text{pixel}}(x, y) = \sum_{i=1}^K \frac{1}{N_i} M_i(x, y)$$ ensuring every object contributes a unit mass regardless of size. The image is partitioned into an $U \times V$ grid; the cardinality map at grid cell $(u, v)$ is

$$y_{\mathrm{car}}(u, v) = \sum_{(x, y) \in \Omega_{u,v}} y_{\text{pixel}}(x, y)$$

with $$\sum_{u=1}^U \sum_{v=1}^V y_{\mathrm{car}}(u, v) = K$$. This construction provides an unambiguous, scale-invariant, and fully differentiable ground truth for count supervision.

2. Model Architecture

YOLO-Count extends YOLOv8-L as its visual backbone, passing $i$0 to yield features at three resolutions: $i$1 Feature fusion occurs via the Vision–Language Path Aggregation Network (VLPAN), which integrates visual features with CLIP-encoded category text embedding $i$2. VLPAN uses T-CSPLayer modules, integrating top-down and bottom-up attention pathways modulated by $i$3, producing two distinct multimodal feature maps $i$4 for classification and counting, respectively: $i$5

Prediction is carried out by two heads:

• Classification head: Locates the open-vocabulary category.

$i$6

• Counting (cardinality regression) head: Sums over $i$7 grid for the count estimate.

$i$8

3. Supervision and Training

YOLO-Count employs a two-phase supervised regime:

3.1 Strong Supervision

• Datasets: Instance mask annotated (e.g. LVIS).
• Object function: Combine losses for cardinality regression and localization:

$i$9

where

$M_i$0

$M_i$1

with hyperparameters $M_i$2.

3.2 Weak Supervision

• Datasets: Point-level count (e.g. FSC147).
• Object function: Combine sparse classification and count consistency:

$M_i$3

with

$M_i$4

$M_i$5

with typical settings $M_i$6.

3.3 Training Schedule

The backbone is pretrained on LVIS for 250 epochs, then finetuned on FSC147 for up to 500 epochs with a mixing ratio $M_i$7 of LVIS batches. The CLIP text encoder is frozen throughout. Learning rates are set to $M_i$8 (backbone) and $M_i$9 (new modules).

4. Integration with Text-to-Image Generation

YOLO-Count provides differentiable count guidance for T2I models. After a denoising pass in the T2I pipeline, the current image is passed to YOLO-Count to produce $M_i \in \{0,1\}^{H\times W}$0. A guidance loss enforces quantity control: $M_i \in \{0,1\}^{H\times W}$1 where $M_i \in \{0,1\}^{H\times W}$2 is the target count.

A learnable “count token” $M_i \in \{0,1\}^{H\times W}$3 is inserted into the text prompt embedding. Gradients of $M_i \in \{0,1\}^{H\times W}$4 with respect to $M_i \in \{0,1\}^{H\times W}$5 are backpropagated through the (frozen) T2I model and YOLO-Count, iteratively updating $M_i \in \{0,1\}^{H\times W}$6 (max 150 steps, LR = 5e-3). This enables fine-grained control of generated object counts, facilitating end-to-end differentiability across both the counting and generative components.

5. Experimental Evaluation

YOLO-Count achieves strong results across counting and T2I quantity control tasks.

5.1 Counting Accuracy

MAE/RMSE metrics across several benchmarks are presented below.

Model	FSC-Test MAE↓	RMSE↓	LVIS MAE↓	RMSE↓	OpenImg7-New MAE↓	RMSE↓	Obj365-New MAE↓	RMSE↓
YOLO-Count	14.80	96.14	1.65	6.08	3.72	11.96	3.28	9.15
CountGD	12.98	98.35	4.84	12.45	6.09	29.92	3.53	10.61
DAVE	14.90	103.42	5.29	11.48	5.31	14.24	4.89	13.22
CLIP-Count	17.78	106.62	10.81	22.61	14.01	30.16	15.48	30.28

5.2 Ablation Analysis

Variant	Test MAE	Test RMSE	Val MAE	Val RMSE
Full YOLO-Count	14.80	96.14	15.43	58.36
– w/o strong pretrain	18.42	111.45	19.50	88.64
– w/o weak finetune	43.91	150.40	43.86	124.33
– density map (no car.)	16.71	107.24	17.87	76.42
– no cls alignment	17.01	110.41	17.57	85.54

Removal of either supervision phase or cardinality map yields consistent degradation in error rates, affirming the necessity and complementarity of each design choice.

5.3 T2I Quantity-Control Control

Across LargeGen and LargeGen-New, YOLO-Count guidance reduces average count error from ∼30% (prior methods) to below 10%. Plots of requested versus generated counts reveal near-slope-1 alignment, indicating robust and effective quantity control.

6. Usage and Implementation

Inputs required for YOLO-Count–controlled generation are:

• Text prompt with a placeholder token (e.g., “<COUNT> red apples”)
• Desired count $M_i \in \{0,1\}^{H\times W}$7
• Pretrained YOLO-Count weights and CLIP text encoder

Recommended training settings are:

• Strong pretrain: $M_i \in \{0,1\}^{H\times W}$8
• Weak finetune: $M_i \in \{0,1\}^{H\times W}$9
• Mixing ratio $N_i = |M_i|$0 (FSC147 style) or $N_i = |M_i|$1 (diverse categories)

A typical inference/generation pseudocode is: $N_i = |M_i|$3

Optional postprocessing via thresholding the classification branch ($N_i = |M_i|$2) controls the tradeoff between strict inclusion and leniency in count estimation.

---

YOLO-Count’s differentiable, regression-based approach enables robust, fine-grained object counting and quantity conditioning in generative pipelines, with verified state-of-the-art empirical performance and practical integration strategies (Zeng et al., 1 Aug 2025).