Vision-Text Quantity Alignment Loss

Updated 6 July 2026

The paper introduces L_align, a loss that enforces ordinal ranking by aligning image features with count-based text prompts, emphasizing numerical proximity.
It employs quantity-oriented text prompts to create factual and counterfactual vision-text pairs, ensuring the model distinguishes fine-grained count differences.
Empirical results show that incorporating structured numerical supervision significantly improves count accuracy and zero-shot generalizability.

Vision-text quantity alignment loss is the objective explicitly introduced in QUANet, the “QUantity-Aware neural Network for text-promptable object counting,” to make vision-language representations sensitive to object numerosity rather than only to category semantics. Denoted $L_{align}$ , it is applied after constructing quantity-oriented text prompts and extracting visual and textual embeddings with the VLM encoders, and its stated role is to inject quantity awareness into the vision encoder before downstream density decoding. In this formulation, alignment is not limited to matching an image with the correct object category; it also encodes relative count structure, so that prompts with numerals closer to the ground-truth count should be more similar to the image than prompts with more distant numerals (Shi et al., 9 Jul 2025).

1. Problem setting and conceptual motivation

Text-promptable object counting methods typically condition counting on category-bearing prompts such as “a photo of [class]”, or on richer prompts that still emphasize semantic attributes such as category, color, and position. QUANet argues that this is adequate for object recognition but insufficient for count discrimination, because the counting problem depends not only on whether image and text refer to the same category, but also on whether the visual representation reflects how many objects are present (Shi et al., 9 Jul 2025).

The motivation for $L_{align}$ is therefore narrower and more structured than ordinary multimodal contrastive learning. Prior prompt-based counting formulations can distinguish that different numerals correspond to different quantities, but QUANet argues that this remains too coarse because it does not teach the model that some wrong counts are closer to the truth than others. The paper states that “Knowing how much is the difference between numbers, as the key to exact object counting, was however untouched in previous works.” The intended supervisory signal is thus ordinal and distance-sensitive rather than merely categorical.

A central implication is that the loss targets a quantity-aware geometry in the joint embedding space. The image representation is not trained only to be “close to the correct text and far from incorrect text”; it is trained so that similarity varies with numerical proximity. In the paper’s counting example, an image containing 14 kiwis should align more strongly with “a photo of 12 kiwis” and “a photo of 16 kiwis” than with “a photo of 10 kiwis” or “a photo of 18 kiwis.” This makes the alignment objective explicitly sensitive to relative numerical error rather than only to positive-versus-negative identity.

2. Quantity-oriented prompts and representation design

QUANet constructs quantity-oriented text prompts $P^Q$ using the template

$\text{``a photo of [num] [class]''}.$

Here, $[class]$ is the target category name, and $[num]$ is replaced by the ground-truth count $a^p$ to form the factual prompt. Counterfactual prompts are generated by replacing $[num]$ with incorrect counts drawn from

$\left \{ a^p\pm k\Delta \mid k \in [1,\frac{N}{2}] \right \},$

where $\Delta$ is a dynamic interval determined from the ground-truth count bin (Shi et al., 9 Jul 2025).

The paper’s concrete example is an image containing 14 kiwis. The factual prompt is “a photo of 14 kiwis,” and counterfactual prompts include “a photo of 10/12/16/18 kiwis.” The image paired with the factual prompt is treated as a positive vision-text pair, while the same image paired with each counterfactual prompt is treated as a negative pair. Quantity information is therefore encoded directly in the text branch through the embedding of the numerical token inside the sentence.

QUANet separates quantity-bearing and category-bearing text representations. The quantity-aware textual embedding is denoted $L_{align}$ 0. For category-conditioned density prediction, the model removes the word embedding of the $L_{align}$ 1 token from $L_{align}$ 2 to obtain a category-only embedding $L_{align}$ 3. This means that the quantity-bearing representation is used for alignment, whereas the quantity-stripped category representation is used for visual-text fusion in decoding. The design explicitly decouples quantity supervision during alignment from category guidance during density prediction.

The prompt schedule is scale-adaptive. The number of counterfactual prompts is $L_{align}$ 4. The dynamic count bins for $L_{align}$ 5 are

$L_{align}$ 6

with corresponding values

$L_{align}$ 7

The paper also reports that fixed $L_{align}$ 8 is worse, which indicates sensitivity to count scale.

3. Mathematical form of the alignment loss

QUANet does not define $L_{align}$ 9 as a CLIP-style batched InfoNCE objective. The paper does not provide logits, temperature scaling, explicit normalization terms, or a batched contrastive formulation. Instead, it defines a ranking-style loss directly on cosine similarities: $P^Q$ 0 where $P^Q$ 1 is the ReLU function (Shi et al., 9 Jul 2025).

The paper defines $P^Q$ 2 as the cosine similarity between the visual embedding and the textual embedding of the positive vision-text pair, and $P^Q$ 3 as the cosine similarities for the negative pairs. The negative set is ordered in a structured way. The first half corresponds to lower-than-truth counts,

$P^Q$ 4

and the second half corresponds to higher-than-truth counts,

$P^Q$ 5

Within each half, $P^Q$ 6 is ordered by descending $P^Q$ 7, so prompts with larger count error appear first.

This ordering determines the semantics of the two terms. The first term enforces that every negative similarity should be below the positive similarity. The second term imposes ranking within the negative set itself: numerically closer wrong counts should have higher similarity than numerically farther wrong counts. In the kiwi example, the similarity for “12 kiwis” should exceed that for “10 kiwis,” and the similarity for “16 kiwis” should exceed that for “18 kiwis.”

The paper explicitly relates the first term to standard contrastive alignment. It states that the first term is similar to the conventional contrastive constraint that forces the positive pair to have higher similarity than negative pairs, and the ablation section says that this term “essentially plays the same role to a standard vision–text contrastive loss as in $P^Q$ 8, which treats all negative samples equally.” The novelty lies in the second term, which adds quantity-aware ranking among negatives. This is the component that encodes ordinal quantity structure rather than simple semantic discrimination.

4. Position within QUANet and optimization pipeline

The inputs to $P^Q$ 9 are the visual embeddings $\text{``a photo of [num] [class]''}.$ 0 and textual embeddings $\text{``a photo of [num] [class]''}.$ 1, extracted from the image and the quantity-oriented prompts by the VLM encoders. The paper specifies a pre-trained DINOv2 with ViT-B/14 backbone as the vision encoder and a pre-trained BERT-base using the top 9 blocks as the text encoder. During training, the text encoder is frozen, and the last six layers of the vision encoder are fine-tuned. The paper states that the loss “helps enhance the quantity-awareness in the vision encoder,” and later reports cosine similarities with the corresponding image global feature, which strongly indicates that alignment is performed with global image-level visual embeddings rather than local patch features, although the exact pooling or projection details are not given in the text (Shi et al., 9 Jul 2025).

After alignment, QUANet strips the number token from $\text{``a photo of [num] [class]''}.$ 2 to obtain $\text{``a photo of [num] [class]''}.$ 3, then uses four cross-attention blocks to produce category-prompted visual features $\text{``a photo of [num] [class]''}.$ 4. These are passed to the dual-stream adaptive counting decoder, or DAC-decoder, which consists of a Transformer stream, a CNN stream, several Transformer-to-CNN enhancement adapters (T2C-adapters), and a gating net for combining the two predicted density maps. The T2C-adapters transfer Transformer-derived contextual and global information into the CNN branch at the patch level through a cross-attention block and a channel-excitation block.

The two streams produce density maps $\text{``a photo of [num] [class]''}.$ 5 and $\text{``a photo of [num] [class]''}.$ 6, which are fused into the final prediction

$\text{``a photo of [num] [class]''}.$ 7

The counting loss is

$\text{``a photo of [num] [class]''}.$ 8

QUANet further introduces the cross-stream quantity ranking loss

$\text{``a photo of [num] [class]''}.$ 9

where $[class]$ 0 is ReLU and $[class]$ 1 is a patch interval. The full training objective is

$[class]$ 2

with $[class]$ 3. In this decomposition, $[class]$ 4 supervises the embedding space, while $[class]$ 5 supervises quantity ordering in the decoder outputs.

Implementation details tied to this stage are explicit. The image size is $[class]$ 6, the patch number for $[class]$ 7 is $[class]$ 8, the ranking interval is $[class]$ 9, the optimizer is AdamW, the learning rate is $[num]$ 0, training runs for 200 epochs with batch size 32, and the reported hardware is an NVIDIA A40 GPU. At inference time, quantity-oriented prompts are not available because the true count is unknown. QUANet therefore drops the QTP pathway and instead uses a standard category prompt such as “a photo of birds” to obtain $[num]$ 1 for category-guided counting. The quantity alignment mechanism is thus a training-time representation-shaping device rather than a test-time prompting requirement.

5. Empirical evidence and ablation results

The paper reports extensive experiments on FSC-147, CARPK, PUCPR+, and ShanghaiTech and attributes strong generalizability for zero-shot class-agnostic counting to the full QUANet design. For the vision-text quantity alignment loss itself, the most direct evidence comes from the FSC-147 ablations on loss functions, where removing either term of $[num]$ 2 degrades performance relative to the full model (Shi et al., 9 Jul 2025).

Setting	Validation MAE	Test MAE
QUANet( $[num]$ 3 w/o FT)	15.29	13.95
QUANet( $[num]$ 4 w/o ST)	15.01	13.81
Full QUANet	14.22	13.24

Here FT and ST denote removal of the first and second terms of $[num]$ 5, respectively. Both components therefore contribute, and the full ranking-aware objective performs best. The comparison to standard vision-text contrastive loss is also explicit: QUANet( $[num]$ 6) yields validation MAE 15.42 and test MAE 13.84, whereas full QUANet reaches validation MAE 14.22 and test MAE 13.24. This supports the paper’s claim that standard CLIP-like contrastive alignment is weaker than the proposed quantity-aware ordering objective.

Ablations on prompting provide indirect support. Removing quantity-oriented text prompts entirely, denoted “QUANet w/o QTPs,” degrades FSC-147 test MAE from 13.24 to 15.10 and RMSE from 97.24 to 103.16. The paper also reports that the average cosine similarity over positive vision-text pairs increases by 6% after training, and that numerical-rank preservation across images improves in mAP from 80.10 to 85.70 when using QTPs. These observations are consistent with stronger quantity sensitivity in the learned representation.

The empirical interpretation offered by the paper is specific. Using only the first term of $[num]$ 7 behaves similarly to replacing the full loss with standard $[num]$ 8, because both enforce positive-over-negative separation while treating negatives coarsely. The second term is the part that adds structured numerical supervision. This suggests that the principal gain does not arise from introducing quantity words into prompts alone, but from imposing an ordering constraint over the similarities induced by those prompts.

Within current vision-language literature, “quantity alignment” can refer to different mechanisms, and QUANet’s formulation is narrower than several superficially related objectives. In QUANet, the term refers specifically to count-sensitive vision-text alignment for object counting, with quantity represented by the numeral token in prompts and with supervision defined by numerical proximity to the ground-truth count (Shi et al., 9 Jul 2025).

This scope differs from several adjacent formulations. $[num]$ 9-CLIP introduces a multi-granular contextualized contrastive alignment loss for caption-, sentence-, and phrase-level text units and their corresponding text-conditioned visual regions, but it is not an explicit count- or cardinality-aware objective; its contribution is hierarchical multi-positive intra-image alignment rather than count discrimination (Zohra et al., 14 Dec 2025). WCA refines inference-time alignment scores in a frozen VLM by constructing a similarity matrix between multiple local image regions and multiple fine-grained text descriptions, yet it introduces no new trainable loss and is framed as weighted local cross-alignment scoring rather than quantity alignment in the counting sense (Li et al., 2024). VISTA addresses a different “quantity” issue: the degradation of vision-text alignment as text sequence length grows in decoder-only MLLMs, adding an explicit alignment regularizer motivated by token-quantity effects rather than by object numerosity (Li et al., 16 May 2025).

Several assumptions and limitations are either stated or implicit in QUANet’s formulation. The method depends on constructing counterfactual counts around the ground truth using a manually designed adaptive interval $a^p$ 0, and its effectiveness therefore assumes that these intervals provide meaningful hardness levels across datasets. The paper indicates that fixed $a^p$ 1 is worse, suggesting sensitivity to count scale. Because inference uses only category prompts, the benefit of $a^p$ 2 must transfer from training-time quantity supervision into an improved visual representation; the paper demonstrates this empirically but does not provide a theoretical analysis of that transfer. The paper also notes remaining failure cases when objects are heavily stacked or occluded, or when object instance definition is ambiguous, as in spectacles counted as two eyeglasses. These cases indicate that quantity-aware embedding alignment improves count sensitivity without fully resolving severe visual ambiguity.

The formulation also leaves several low-level details unspecified in the text. The paper explicitly provides the prompt template, positive and negative pair construction, the exact equation for $a^p$ 3, the definitions of $a^p$ 4 and $a^p$ 5, the adaptive $a^p$ 6 schedule, and the ablation evidence. It does not provide exact formulas for visual or textual embedding pooling, feature normalization, projection heads, or any temperature parameter, nor does it define a batched contrastive setup for $a^p$ 7. For these details, the paper text alone is insufficient.