FitAP: Confidence-Free AP for OV-RefOD

• FitAP is a confidence-free Average Precision metric for OV-RefOD that deterministically ranks predicted boxes using a score based on normalized area and the best IoU with ground truth.
• It preserves the standard AP evaluation by aggregating predictions globally and applying greedy one-to-one matching over multiple COCO-style IoU thresholds.
• Empirical results show that applying techniques like Reverse Contrast Attention can boost FitAP performance, albeit with limitations such as scale sensitivity and reliance on oracle IoU.

Searching arXiv for FitAP and the cited paper to ground the article. FitAP is a confidence-free Average Precision metric introduced for Open Vocabulary Referring Object Detection (OV-RefOD), a setting in which a model receives an image $I$ and a free-form referring expression $Q$ and returns a set of predicted boxes $P(I,Q)=\{b_1,\dots,b_{N_P}\}$, often without per-box confidence scores. FitAP preserves the standard AP precision-recall protocol by replacing learned confidence with a deterministic ranking score,

$$s(b;I,Q)=A_{\mathrm{box}}(b;I)\cdot \mathrm{IoU}^\star(b),$$

where $A_{\mathrm{box}}(b;I)=|b|/|I|$ is the normalized box area and $$\mathrm{IoU}^\star(b)=\max_{g\in G(I,Q)} \mathrm{IoU}(b,g)$$ is the best overlap with the ground-truth set $G(I,Q)=\{g_1,\dots,g_{N_G}\}$. FitAP is then defined as the mean AP over the ten COCO-style IoU thresholds $\{0.50,0.55,\dots,0.95\}$ (Juanico et al., 26 Jul 2025).

1. Formal definition in OV-RefOD

For a prediction $b\in P(I,Q)$, FitAP first computes the standard intersection-over-union

$$\mathrm{IoU}(b,g)=\frac{|b\cap g|}{|b\cup g|}\in[0,1],$$

then identifies the best-overlap ground truth through

$Q$0

The confidence-free score used only for ranking is

$Q$1

This construction retains AP-style ranking and matching even when the VLM does not emit classifier scores or proposal confidences (Juanico et al., 26 Jul 2025).

At a fixed IoU threshold $Q$2, all predictions from all $Q$3 pairs are aggregated into a single list and sorted in descending order by $Q$4. Greedy one-to-one matching then follows standard detection practice: a prediction is a true positive if $Q$5 and the ground-truth indexed by $Q$6 has not yet been matched by any higher-ranked prediction from the same image-query pair; otherwise it is a false positive. With $Q$7 denoting the total number of ground-truth boxes across the evaluation set, cumulative precision and recall at rank $Q$8 are

$Q$9

The interpolated precision envelope is

$P(I,Q)=\{b_1,\dots,b_{N_P}\}$0

and

$P(I,Q)=\{b_1,\dots,b_{N_P}\}$1

Finally,

$P(I,Q)=\{b_1,\dots,b_{N_P}\}$2

2. Ranking, matching, and metric mechanics

FitAP is global rather than per-image: it aggregates all predictions from all $P(I,Q)=\{b_1,\dots,b_{N_P}\}$3 pairs into one ranked list, then reuses that ranking for every IoU threshold. The metric therefore separates ranking from thresholding. IoU enters twice. First, it governs matching through the condition $P(I,Q)=\{b_1,\dots,b_{N_P}\}$4. Second, it affects rank through the multiplicative score $P(I,Q)=\{b_1,\dots,b_{N_P}\}$5. Box area enters only through the normalized area term, which is constrained to $P(I,Q)=\{b_1,\dots,b_{N_P}\}$6 by division through image area $P(I,Q)=\{b_1,\dots,b_{N_P}\}$7 (Juanico et al., 26 Jul 2025).

The paper’s worked example makes the mechanics explicit. For one $P(I,Q)=\{b_1,\dots,b_{N_P}\}$8 pair with two ground-truth boxes and three predictions, the predictions are ranked by score as $P(I,Q)=\{b_1,\dots,b_{N_P}\}$9, $$s(b;I,Q)=A_{\mathrm{box}}(b;I)\cdot \mathrm{IoU}^\star(b),$$0, $$s(b;I,Q)=A_{\mathrm{box}}(b;I)\cdot \mathrm{IoU}^\star(b),$$1 even though $$s(b;I,Q)=A_{\mathrm{box}}(b;I)\cdot \mathrm{IoU}^\star(b),$$2 has the worst IoU, because its normalized area is largest. At $$s(b;I,Q)=A_{\mathrm{box}}(b;I)\cdot \mathrm{IoU}^\star(b),$$3, this yields one false positive followed by two true positives and an interpolated $$s(b;I,Q)=A_{\mathrm{box}}(b;I)\cdot \mathrm{IoU}^\star(b),$$4. At $$s(b;I,Q)=A_{\mathrm{box}}(b;I)\cdot \mathrm{IoU}^\star(b),$$5, no prediction satisfies the threshold, so $$s(b;I,Q)=A_{\mathrm{box}}(b;I)\cdot \mathrm{IoU}^\star(b),$$6. Averaging only these two thresholds for illustration gives $$s(b;I,Q)=A_{\mathrm{box}}(b;I)\cdot \mathrm{IoU}^\star(b),$$7; in actual use, FitAP averages over all ten thresholds from $$s(b;I,Q)=A_{\mathrm{box}}(b;I)\cdot \mathrm{IoU}^\star(b),$$8 to $$s(b;I,Q)=A_{\mathrm{box}}(b;I)\cdot \mathrm{IoU}^\star(b),$$9 (Juanico et al., 26 Jul 2025).

This construction penalizes duplicate or overlapping predictions in the same way as standard AP. The highest-ranked prediction matching a given ground-truth box receives the true positive, while later predictions hitting the same box become false positives. Multiple predicted boxes per referring expression are therefore allowed, but the one-to-one matching rule prevents overcounting.

3. Computation procedure and implementation details

The evaluation pipeline begins by precomputing, for every predicted box in every image-query pair, the full IoU vector against all ground-truth boxes, the best-overlap value $A_{\mathrm{box}}(b;I)=|b|/|I|$0, the corresponding index $A_{\mathrm{box}}(b;I)=|b|/|I|$1, the normalized area $A_{\mathrm{box}}(b;I)=|b|/|I|$2, and the ranking score $A_{\mathrm{box}}(b;I)=|b|/|I|$3. These per-prediction records are then pooled across the dataset and sorted in descending order by score. The same ordering is reused for all ten IoU thresholds, which makes the threshold sweep inexpensive once ranking has been fixed (Juanico et al., 26 Jul 2025).

For each threshold $A_{\mathrm{box}}(b;I)=|b|/|I|$4, all ground-truth boxes are initialized as unmatched. The evaluator traverses the globally sorted prediction list once, assigns TP or FP labels under greedy one-to-one matching, and accumulates precision-recall points. AP is computed by integrating the interpolated precision envelope, either by summation over recall-change points or on a fixed recall grid. The default settings given are the ten COCO-style IoU thresholds, area normalization $A_{\mathrm{box}}(b;I)=|b|/|I|$5 with $A_{\mathrm{box}}(b;I)=|b|/|I|$6 in pixels, and a precision-envelope integral consistent with standard AP implementations (Juanico et al., 26 Jul 2025).

The stated complexity is $A_{\mathrm{box}}(b;I)=|b|/|I|$7 for IoU computation over image-query pairs, $A_{\mathrm{box}}(b;I)=|b|/|I|$8 for sorting the global prediction list with $A_{\mathrm{box}}(b;I)=|b|/|I|$9, and $$\mathrm{IoU}^\star(b)=\max_{g\in G(I,Q)} \mathrm{IoU}(b,g)$$0 for the ten threshold traversals. The implementation guidance is correspondingly concrete: vectorize IoU computation per image, cache image area, use a stable sort, and break score ties by higher $$\mathrm{IoU}^\star(b)=\max_{g\in G(I,Q)} \mathrm{IoU}(b,g)$$1, then larger $$\mathrm{IoU}^\star(b)=\max_{g\in G(I,Q)} \mathrm{IoU}(b,g)$$2, then a deterministic index.

4. Design rationale and relation to standard AP

The defining motivation for FitAP is that many VLMs used in OV-RefOD do not emit per-box confidence scores. Standard AP and mAP require a confidence score per prediction for ranking and are therefore sensitive to score calibration and class priors. FitAP avoids confidence altogether by ranking detections with a deterministic, geometry-derived score. In the intended use case, this makes it especially suitable for prompt-based VLMs and OV-RefOD, where confidences are missing, inconsistent, or not comparable (Juanico et al., 26 Jul 2025).

The metric is also presented as reducing dependence on uncalibrated textual or logit outputs and instruction-following idiosyncrasies. Instead, it emphasizes two interpretable geometric factors: alignment, through IoU, and sizing, through normalized box area. The paper reports strong empirical correlations with ground-truth area: Pearson $$\mathrm{IoU}^\star(b)=\max_{g\in G(I,Q)} \mathrm{IoU}(b,g)$$3 for $$\mathrm{IoU}^\star(b)=\max_{g\in G(I,Q)} \mathrm{IoU}(b,g)$$4 and $$\mathrm{IoU}^\star(b)=\max_{g\in G(I,Q)} \mathrm{IoU}(b,g)$$5 for $$\mathrm{IoU}^\star(b)=\max_{g\in G(I,Q)} \mathrm{IoU}(b,g)$$6. Precision-recall curves built from $$\mathrm{IoU}^\star(b)=\max_{g\in G(I,Q)} \mathrm{IoU}(b,g)$$7 show the expected precision-recall trade-off, monotone AP decrease as $$\mathrm{IoU}^\star(b)=\max_{g\in G(I,Q)} \mathrm{IoU}(b,g)$$8 increases, and zigzag empirical PR curves with standard interpolation envelopes (Juanico et al., 26 Jul 2025).

The stability and fairness arguments are correspondingly limited and explicit. Deterministic ranking from geometry is said to reduce metric variance due to decoding randomness in VLMs, and normalization by image area keeps $$\mathrm{IoU}^\star(b)=\max_{g\in G(I,Q)} \mathrm{IoU}(b,g)$$9, making the score commensurate across varying resolutions and aspect ratios. At the same time, the metric’s ranking is evaluation-time and oracle-like, because $G(I,Q)=\{g_1,\dots,g_{N_G}\}$0 is computed against ground truth. A plausible implication is that FitAP is best understood as a confidence-free localization metric rather than a drop-in replacement for confidence-ranked mAP.

5. Empirical use with Reverse Contrast Attention

FitAP was introduced and evaluated on OV-RefOD using a novel split of COCO val2017 containing 2,064 $G(I,Q)=\{g_1,\dots,g_{N_G}\}$1 pairs. The paper reports FitAP as a single scalar, namely the average of $G(I,Q)=\{g_1,\dots,g_{N_G}\}$2 over $G(I,Q)=\{g_1,\dots,g_{N_G}\}$3, and applies Reverse Contrast Attention (RCA) at inference time to 15 open-source VLMs. The main result is that RCA improved FitAP in 11 of 15 models, with gains up to $G(I,Q)=\{g_1,\dots,g_{N_G}\}$4 (Juanico et al., 26 Jul 2025).

Model	FitAP post vs. pre	Change
Qwen2.5-VL-7B	46.8535 vs 37.0005	+26.6%
DeepSeek-VL2	3.99586 vs 3.38530	+18.0%
SAIL-VL-1.6-8B	5.67149 vs 4.84873	+17.0%
Gemma3-27B	3.01913 vs 2.74179	+10.1%
PaliGemma2-3B-mix-448	41.1179 vs 38.7982	+5.98%

The broader result set also includes gains for Ovis2-34B, Kimi-VL-A3B, VARCO-VISION-14B, valley2_dpo, and Moondream2, while WeThink-Qwen2.5VL-7B, Ristretto-3B, POINTS1.5-Qwen2.5-7B, and Valley-Eagle show decreases. The paper relates these outcomes to attention sharpness and the timing and structure of vision-language fusion: late-fusion models benefit consistently, while DeepSeek-VL2, described as an early-fusion MoE, also improves, indicating that capacity and disentanglement matter. The accompanying analyses report negative correlations between the count of subthreshold hidden-state components and a central attention sharpness metric $G(I,Q)=\{g_1,\dots,g_{N_G}\}$5 in models that improved with RCA, including Qwen2.5-VL-7B ($G(I,Q)=\{g_1,\dots,g_{N_G}\}$6) and DeepSeek-VL2 ($G(I,Q)=\{g_1,\dots,g_{N_G}\}$7), both statistically significant; WeThink’s correlation was insignificant and RCA did not help it (Juanico et al., 26 Jul 2025).

6. Limitations, extensions, and nomenclature

FitAP has three explicit limitations. First, ranking uses $G(I,Q)=\{g_1,\dots,g_{N_G}\}$8 computed against ground truth, so evaluation uses an oracle ranking independent of model confidence. Second, multiplying by $G(I,Q)=\{g_1,\dots,g_{N_G}\}$9 introduces scale sensitivity: at equal $\{0.50,0.55,\dots,0.95\}$0, larger boxes can be favored. Third, FitAP is box-based and therefore does not evaluate mask quality or fine-grained shape. These limitations are not hidden by the formulation; they are part of the metric’s definition and are central to interpreting results (Juanico et al., 26 Jul 2025).

The edge-case behavior is similarly explicit. Multiple predicted boxes per referring expression are allowed; duplicates beyond the first correct match become false positives. If $\{0.50,0.55,\dots,0.95\}$1 but predictions exist, these contribute only false positives. For benchmarks, such pairs can either be excluded from $\{0.50,0.55,\dots,0.95\}$2 if the dataset guarantees no positives, or retained to test specificity. Small GT or prediction misalignments near the IoU threshold can flip TP and FP labels, but averaging over thresholds mitigates threshold brittleness. Ties in the ranking score should be broken deterministically, and the paper recommends stable sorting for reproducibility (Juanico et al., 26 Jul 2025).

The proposed extensions retain the confidence-free principle while changing the ranking function or evaluation granularity. These include a scale-robust score

$\{0.50,0.55,\dots,0.95\}$3

with $\{0.50,0.55,\dots,0.95\}$4 and $\{0.50,0.55,\dots,0.95\}$5; an IoU-only baseline $\{0.50,0.55,\dots,0.95\}$6; multi-scale reporting as FitAPs, FitAPm, and FitAPl; a segmentation-based “FitAP-mask”; and per-query analysis for multi-instance referring expressions. Finally, the label “FitAP” is not unique across arXiv. An unrelated use appears in cell-free massive MIMO-aided ISAC systems, where “FitAP” denotes joint Access Point activation and power allocation rather than a detection metric (Tung et al., 12 Jul 2025). In the OV-RefOD literature, however, FitAP specifically denotes the confidence-free AP construction defined by $\{0.50,0.55,\dots,0.95\}$7 and averaged over the standard ten IoU thresholds (Juanico et al., 26 Jul 2025).