Class-Agnostic Average Precision (CAAP)

• Class-Agnostic Average Precision (CAAP) is an evaluation metric that measures localization performance by matching predicted and ground-truth bounding boxes without considering semantic labels.
• It computes the area under the precision–recall curve across varied IoU thresholds, offering an AP-style assessment of spatial accuracy in object detection.
• CAAP is crucial for open-world applications as it isolates localization quality and is reported over sensitivity intervals on datasets like COCO-Val and LVIS-Minival.

Searching arXiv for the cited papers to ground the article in current records. Class-Agnostic Average Precision (CAAP) is an evaluation metric introduced in "LLM-Guided Agentic Object Detection for Open-World Understanding" for object detection settings in which localization must be assessed independently of semantic naming (Mumcu et al., 14 Jul 2025). It measures the area under a precision–recall curve obtained by matching predicted and ground-truth bounding boxes using only spatial overlap and ignoring all class labels. The metric is motivated by open-world and open-vocabulary detection, where valid classes may change per image and where it is useful to evaluate pure localization performance even when labels are incorrect, novel, or generated on the fly. Earlier work on class-agnostic detection defined the task of predicting boxes for all objects irrespective of class, but evaluated it primarily with Average Recall at fixed numbers of detections, AR@K, rather than any AP-style formulation (Jaiswal et al., 2020).

1. Task setting and motivation

Class-agnostic detection focuses on detecting objects irrespective of their object-classes. In the earlier formulation, the goal is to predict bounding boxes for all objects in an image but not their object-classes, with the predicted boxes then consumed by another system for application-specific classification, retrieval, or related downstream processing. The motivation given for that task includes settings such as large-scale object matching, visual grounding, visual relation prediction, and obstacle detection, where determining the presence and location of objects may matter more than assigning them to a fixed taxonomy (Jaiswal et al., 2020).

CAAP arises in a later open-world formulation in which semantic labels may be generated dynamically by a LLM and passed to an open-vocabulary detector. In that context, the metric is intended to isolate localization from naming. The stated motivation is threefold: in open-world or open-vocabulary settings the set of valid classes may change per image; pure localization performance may be of interest even if the semantic label is incorrect or novel; and decoupling localization from naming quality, measured separately by Semantic Naming Average Precision (SNAP), gives clearer insight into where failures occur (Mumcu et al., 14 Jul 2025).

2. Formal definition

The metric is defined over the set of all predicted bounding boxes and the set of all ground-truth bounding boxes. Let $B_P=\{b_{p,1},b_{p,2},\dots,b_{p,N_P}\}$ denote the predictions, where each prediction $b_{p,i}$ has confidence score $s_i$, and let $B_{GT}=\{b_{gt,1},b_{gt,2},\dots,b_{gt,N_{GT}}\}$ denote the ground-truth boxes. Let $\theta_{iou}$ be an Intersection-over-Union threshold for spatial matching, and let $\theta_s$ be a confidence-score threshold (Mumcu et al., 14 Jul 2025).

A predicted box $b_{p,i}$ is matched to a ground-truth box $b_{gt,j}$ if

$\mathrm{IoU}(b_{p,i},b_{gt,j})\;\ge\;\theta_{iou},$

ignoring all class labels. Each ground-truth box may be matched to at most one prediction, specifically the one with highest score among candidates.

The indicator functions for true and false positives are

$$\mathrm{TP}(b_{p,i}) = \begin{cases} 1 & \text{if }b_{p,i}\text{ matches a unique }b_{gt},\ 0 & \text{otherwise}, \end{cases}$$

$b_{p,i}$0

For fixed $b_{p,i}$1 and $b_{p,i}$2, precision and recall are

$b_{p,i}$3

$b_{p,i}$4

The Class-Agnostic Average Precision at IoU $b_{p,i}$5 is then the area under the precision–recall curve:

$b_{p,i}$6

In practice this is approximated by sampling many $b_{p,i}$7. To report a single number over multiple IoU thresholds $b_{p,i}$8, the paper uses

$b_{p,i}$9

where each $s_i$0 is itself averaged over the chosen set of score thresholds $s_i$1 (Mumcu et al., 14 Jul 2025).

3. Computation protocol

The computation procedure follows the standard AP pattern, but with all labels collapsed into a single undifferentiated object category. Across the dataset, one first collects all detections $s_i$2 and all ground-truth boxes $s_i$3. For each IoU threshold in $s_i$4, predictions are sorted by descending score, all ground-truth boxes are initialized as unmatched, and each prediction is compared against every unmatched ground-truth box. If the best overlap satisfies the IoU threshold, the prediction is declared a true positive and that ground-truth box is marked matched; otherwise the prediction is a false positive. After all predictions have been processed, any ground-truth boxes that remain unmatched count as false negatives (Mumcu et al., 14 Jul 2025).

The precision–recall curve is then constructed by sweeping the confidence threshold $s_i$5 over a fine grid, given in the paper summary as, for example, 101 values. Integrating that curve yields $s_i$6, and averaging over the IoU thresholds yields $s_i$7. The implied pseudocode permits either the 101-point interpolated AP or all-points interpolation as in COCO (Mumcu et al., 14 Jul 2025).

This protocol differs from a mere top-$s_i$8 recall calculation because it explicitly tracks both true positives and false positives as the confidence threshold changes. That property is central to its status as an AP-style metric rather than a recall-only measure.

4. Relation to standard AP and to earlier class-agnostic evaluation

Standard AP is computed separately per class, matches only predictions whose predicted label equals the ground-truth class, and then averages across classes. CAAP instead ignores all class labels and treats every box as belonging to the same class. The metric therefore removes any dependence on whether the predicted semantic label matches the ground truth and evaluates only spatial localization under a one-class matching regime (Mumcu et al., 14 Jul 2025).

The distinction is especially important because the earlier paper "Class-agnostic Object Detection" did not introduce or define any Class-Agnostic Average Precision metric. Its evaluation protocol for generalization to unseen object-types used Average Recall at varying numbers of allowed detections, AR@K with IoU $s_i$9. Concretely, for each image the top $B_{GT}=\{b_{gt,1},b_{gt,2},\dots,b_{gt,N_{GT}}\}$0 scored boxes were taken, those boxes were matched greedily to all ground-truth boxes of all classes with classes ignored, true positives were counted for matched ground-truth boxes, and recall was computed as $B_{GT}=\{b_{gt,1},b_{gt,2},\dots,b_{gt,N_{GT}}\}$1. The metric was then averaged over the validation set to obtain AR@K, with reported values for $B_{GT}=\{b_{gt,1},b_{gt,2},\dots,b_{gt,N_{GT}}\}$2, separately on seen classes, unseen classes, and their harmonic mean across multiple datasets (Jaiswal et al., 2020).

A common misconception is therefore to attribute CAAP to the 2020 class-agnostic detection paper. The available record does not support that attribution: the earlier work formulates the task and a recall-based evaluation, whereas the AP-style class-agnostic metric appears in the later open-world setting (Jaiswal et al., 2020).

5. Benchmark reporting and sensitivity intervals

The reported experiments present CAAP on LVIS-Minival, COCO-Val, and COCO-OOD. Table 1 is organized by three “sensitivity intervals,” defined as Low (LO): $B_{GT}=\{b_{gt,1},b_{gt,2},\dots,b_{gt,N_{GT}}\}$3–$B_{GT}=\{b_{gt,1},b_{gt,2},\dots,b_{gt,N_{GT}}\}$4, Mid (MI): $B_{GT}=\{b_{gt,1},b_{gt,2},\dots,b_{gt,N_{GT}}\}$5–$B_{GT}=\{b_{gt,1},b_{gt,2},\dots,b_{gt,N_{GT}}\}$6, and High (HI): $B_{GT}=\{b_{gt,1},b_{gt,2},\dots,b_{gt,N_{GT}}\}$7–$B_{GT}=\{b_{gt,1},b_{gt,2},\dots,b_{gt,N_{GT}}\}$8. These intervals summarize behavior at progressively stricter localization tolerances (Mumcu et al., 14 Jul 2025).

Dataset	Interval	CAAP
LVIS-Minival	HI	0.06
LVIS-Minival	MI	0.13
LVIS-Minival	LO	0.15
COCO-Val	HI	0.08
COCO-Val	MI	0.22
COCO-Val	LO	0.25
COCO-OOD (unknown only)	HI	0.26
COCO-OOD (unknown only)	MI	0.56
COCO-OOD (unknown only)	LO	0.61

The paper states that on COCO-Val the model achieves $B_{GT}=\{b_{gt,1},b_{gt,2},\dots,b_{gt,N_{GT}}\}$9, $\theta_{iou}$0, and $\theta_{iou}$1. On LVIS-Minival, described as long-tail, results drop to $\theta_{iou}$2, $\theta_{iou}$3, and $\theta_{iou}$4 for LO, MI, and HI, respectively. On COCO-OOD, described as entirely novel objects, CAAP is highest, with $\theta_{iou}$5 at LO, $\theta_{iou}$6 at MI, and $\theta_{iou}$7 at HI, which the paper interprets as showing that the detector can localize unseen categories well (Mumcu et al., 14 Jul 2025).

6. Advantages, limitations, and interpretive use

The stated advantages of CAAP are that it purely measures localization accuracy under open-world assumptions, that class-agnostic matching avoids penalizing novel labels, and that when used alongside SNAP it disentangles spatial from semantic performance. These properties make the metric suitable when the system’s label space is dynamic or partly unknown and when localization and naming need to be diagnosed separately (Mumcu et al., 14 Jul 2025).

The limitations noted or implied are equally important. CAAP alone does not reflect whether the model picked the “right” name; it records only whether something was localized. In cases where fine-grained semantics influence localization, such as part-versus-whole distinctions, IoU-only matching can be ambiguous. The computational cost and complexity are also similar to standard AP because the procedure requires multiple IoU thresholds and multiple score thresholds (Mumcu et al., 14 Jul 2025).

The practical recommendations are to choose a sufficiently fine grid of IoU and score thresholds for stable estimates, to report CAAP together with a semantic metric such as SNAP, and to be aware that very low IoU thresholds may inflate CAAP for sloppy boxes. For that reason, reporting over LO, MI, and HI intervals provides additional nuance beyond a single averaged score (Mumcu et al., 14 Jul 2025).