---
title: 'YOLO-Count: Differentiable Object Counting'
url: https://www.emergentmind.com/topics/yolo-count
type: topic
---

# YOLO-Count: Differentiable Object Counting

YOLO-Count is a differentiable, open-vocabulary object counting model designed to address general counting challenges and provide precise quantity control for text-to-image (T2I) generation. Its core innovations include a regression-based “cardinality map” for interpretable and scale-aware counting, a vision–language fusion architecture, and a hybrid supervision framework. YOLO-Count achieves state-of-the-art accuracy on standard counting and T2I-guided benchmarks and enables fully differentiable, gradient-based end-to-end control over object counts in image generation pipelines [2508.00728].

## 1. Cardinality Map Formalism

YOLO-Count introduces the “cardinality map” as a regression target distinct from Gaussian density maps. For an image containing \(K\) annotated object instances, with each instance \(i\) having a binary mask \(M_i\) (\(M_i \in \{0,1\}^{H\times W}\)), let \(N_i = |M_i|\) denote its area. The pixelwise cardinality contribution is defined as:
\[
y_{\text{pixel}}(x, y) = \sum_{i=1}^K \frac{1}{N_i} M_i(x, y)
\]
ensuring every object contributes a unit mass regardless of size. The image is partitioned into an \(U \times V\) grid; the cardinality map at grid cell \((u, v)\) is
\[
y_{\mathrm{car}}(u, v) = \sum_{(x, y) \in \Omega_{u,v}} y_{\text{pixel}}(x, y)
\]
with \(\sum_{u=1}^U \sum_{v=1}^V y_{\mathrm{car}}(u, v) = K\). This construction provides an unambiguous, scale-invariant, and fully differentiable ground truth for count supervision.

## 2. Model Architecture

YOLO-Count extends YOLOv8-L as its visual backbone, passing \(I \in \mathbb{R}^{640 \times 640 \times 3}\) to yield features at three resolutions:
\[
[f_{80\times 80}, f_{40 \times 40}, f_{20 \times 20}] = \mathrm{VisualBackbone}(I).
\]
Feature fusion occurs via the Vision–Language Path Aggregation Network (VLPAN), which integrates visual features with CLIP-encoded category text embedding \(f_T\). VLPAN uses T-CSPLayer modules, integrating top-down and bottom-up attention pathways modulated by \(f_T\), producing two distinct multimodal feature maps \([f^1, f^2]\) for classification and counting, respectively:
\[
[f^1, f^2] = \mathrm{VLPAN}([f_{80}, f_{40}, f_{20}], f_T).
\]

Prediction is carried out by two heads:
- **Classification head:** Locates the open-vocabulary category.
  \[
  \hat{y}_{\mathrm{cls}} = \sigma(\langle o_{\mathrm{cls}}(p), f_T \rangle), \quad \hat{y}_{\mathrm{cls}} \in [0,1]^{U \times V}
  \]
- **Counting (cardinality regression) head:** Sums over \(U \times V\) grid for the count estimate.
  \[
  \hat{y}_{\mathrm{cnt}} = \mathrm{CountingHead}(f^2) \in \mathbb{R}^{U \times V}
  \]

## 3. Supervision and Training

YOLO-Count employs a two-phase supervised regime:

### 3.1 Strong Supervision

- **Datasets:** Instance mask annotated (e.g. LVIS).
- **Object function:** Combine losses for cardinality regression and localization:
  \[
  \mathcal{L}^{\mathrm{strong}} = \alpha_1 \mathcal{L}_{\mathrm{cnt}}^{\mathrm{strong}} + \beta_1 \mathcal{L}_{\mathrm{cls}}^{\mathrm{strong}}
  \]
  where
  \[
  \mathcal{L}_{\mathrm{cnt}}^{\mathrm{strong}} = |\hat{y}_{\mathrm{cnt}} - y_{\mathrm{car}}|_1,
  \]
  \[
  \mathcal{L}_{\mathrm{cls}}^{\mathrm{strong}} = \mathrm{BCE}(\hat{y}_{\mathrm{cls}}, y_{\mathrm{cls}})
  \]
  with hyperparameters \(\alpha_1=1.0, \beta_1=0.1\).

### 3.2 Weak Supervision

- **Datasets:** Point-level count (e.g. FSC147).
- **Object function:** Combine sparse classification and count consistency:
  \[
  \mathcal{L}^{\mathrm{weak}} = \alpha_2 \mathcal{L}_{\mathrm{cnt}}^{\mathrm{weak}} + \beta_2 \mathcal{L}_{\mathrm{cls}}^{\mathrm{weak}}
  \]
  with
  \[
  \mathcal{L}_{\mathrm{cnt}}^{\mathrm{weak}} = \left| \sum_p \hat{y}_{\mathrm{cnt}}(p) - K \right|
  \]
  \[
  \mathcal{L}_{\mathrm{cls}}^{\mathrm{weak}} = -\frac{1}{|\Omega|} \sum_{p \in \Omega} [M_{\mathrm{pos}}(p)\log\hat{y}_{\mathrm{cls}}(p) + M_{\mathrm{neg}}(p)\log (1-\hat{y}_{\mathrm{cls}}(p))]
  \]
  with typical settings \(\alpha_2=1.0, \beta_2=0.1\).

### 3.3 Training Schedule

The backbone is pretrained on LVIS for 250 epochs, then finetuned on FSC147 for up to 500 epochs with a mixing ratio \(\gamma\) of LVIS batches. The CLIP text encoder is frozen throughout. Learning rates are set to \(5\times 10^{-9}\) (backbone) and \(1\times10^{-5}\) (new modules).

## 4. Integration with Text-to-Image Generation

YOLO-Count provides differentiable count guidance for T2I models. After a denoising pass in the T2I pipeline, the current image is passed to YOLO-Count to produce \(\hat{y}_{\mathrm{cnt}}\). A guidance loss enforces quantity control:
\[
\mathcal{L}_{\mathrm{guide}} = \left| \sum_p \hat{y}_{\mathrm{cnt}}(p) - Q_{\mathrm{req}} \right|
\]
where \(Q_{\mathrm{req}}\) is the target count. 

A learnable “count token” \(\tau\) is inserted into the text prompt embedding. Gradients of \(\mathcal{L}_{\mathrm{guide}}\) with respect to \(\tau\) are backpropagated through the (frozen) T2I model and YOLO-Count, iteratively updating \(\tau\) (max 150 steps, LR = 5e-3). This enables fine-grained control of generated object counts, facilitating end-to-end differentiability across both the counting and generative components.

## 5. Experimental Evaluation

YOLO-Count achieves strong results across counting and T2I quantity control tasks.

### 5.1 Counting Accuracy

MAE/RMSE metrics across several benchmarks are presented below.

| Model         | FSC-Test MAE↓ | RMSE↓ | LVIS MAE↓ | RMSE↓ | OpenImg7-New MAE↓ | RMSE↓ | Obj365-New MAE↓ | RMSE↓ |
|---------------|---------------|-------|-----------|-------|-------------------|-------|-----------------|-------|
| YOLO-Count    | **14.80**     |**96.14**| **1.65** |**6.08**| **3.72**         |**11.96**| **3.28**      |**9.15**|
| CountGD       | 12.98         | 98.35 | 4.84      | 12.45 | 6.09              | 29.92 | 3.53          | 10.61  |
| DAVE          | 14.90         |103.42 | 5.29      | 11.48 | 5.31              | 14.24 | 4.89          | 13.22  |
| CLIP-Count    | 17.78         |106.62 |10.81      | 22.61 |14.01              | 30.16 |15.48          | 30.28  |

### 5.2 Ablation Analysis

| Variant                  | Test MAE | Test RMSE | Val MAE | Val RMSE |
|--------------------------|----------|-----------|---------|----------|
| Full YOLO-Count          | 14.80    | 96.14     | 15.43   | 58.36    |
| – w/o strong pretrain    | 18.42    |111.45     | 19.50   | 88.64    |
| – w/o weak finetune      | 43.91    |150.40     | 43.86   |124.33    |
| – density map (no car.)  | 16.71    |107.24     | 17.87   | 76.42    |
| – no cls alignment       | 17.01    |110.41     | 17.57   | 85.54    |

Removal of either supervision phase or cardinality map yields consistent degradation in error rates, affirming the necessity and complementarity of each design choice.

### 5.3 T2I Quantity-Control Control

Across LargeGen and LargeGen-New, YOLO-Count guidance reduces average count error from ∼30% (prior methods) to below 10%. Plots of requested versus generated counts reveal near-slope-1 alignment, indicating robust and effective quantity control.

## 6. Usage and Implementation

Inputs required for YOLO-Count–controlled generation are:
- Text prompt with a placeholder token (e.g., “<COUNT> red apples”)
- Desired count \(Q_{\mathrm{req}}\)
- Pretrained YOLO-Count weights and CLIP text encoder

Recommended training settings are:
- Strong pretrain: \(\alpha_1=1.0, \beta_1=0.1\)
- Weak finetune: \(\alpha_2=1.0, \beta_2=0.1\)
- Mixing ratio \(\gamma \approx 0.05\) (FSC147 style) or \(\gamma \approx 0.5\) (diverse categories)

A typical inference/generation pseudocode is:
```python
Given prompt with token τ, target Q_req
Initialize τ from CLIP glove
for step in 1…150 do
    I ← T2I_model.generate(prompt(τ))
    y_cnt ← YOLO-Count(I, prompt)
    L_guide ← | sum(y_cnt) – Q_req |
    if L_guide plateaued 20 steps: break
    τ ← τ – lr * ∂L_guide/∂τ
end
return final image I
```

Optional postprocessing via thresholding the classification branch (\(\hat{y}_{\mathrm{cls}}(p) > \kappa\)) controls the tradeoff between strict inclusion and leniency in count estimation.

---

YOLO-Count’s differentiable, regression-based approach enables robust, fine-grained object counting and quantity conditioning in generative pipelines, with verified state-of-the-art empirical performance and practical integration strategies [2508.00728].

Source: https://www.emergentmind.com/topics/yolo-count