Semantic Naming AP (SNAP)

• SNAP is an AP-style metric for evaluating naming quality in open-world detection, decoupling semantic evaluation from localization performance.
• It computes cosine similarity in CLIP text-embedding space to compare predicted names with ground-truth labels, bypassing exact string matches and IoU constraints.
• Paired with CAAP, SNAP isolates naming errors from localization issues, offering a nuanced assessment of autonomous open-world detection systems.

Searching arXiv for the cited SNAP paper and closely related AP/semantic-ranking work. Semantic Naming Average Precision (SNAP) is an AP-style evaluation metric introduced for open-world object detection systems that autonomously generate object names, rather than operating over a fixed category vocabulary. It is designed to measure the semantic correctness of predicted object names independently of spatial localization, using cosine similarity in CLIP text-embedding space instead of exact string identity or box overlap. In the LAOD framework, SNAP is paired with Class-Agnostic Average Precision (CAAP): CAAP isolates localization quality, while SNAP isolates naming quality, reflecting the paper’s view that localization and naming can fail independently in open-world settings (Mumcu et al., 14 Jul 2025).

1. Definition and scope

SNAP is defined in “LLM-Guided Agentic Object Detection for Open-World Understanding” as a metric for evaluating “the semantic correctness of the names produced by the system,” independently of whether the detector localized the objects well (Mumcu et al., 14 Jul 2025). The metric arises from a pipeline in which a LLM generates scene-specific object names and an open-vocabulary detector localizes objects corresponding to those names. Because the system has two distinct capabilities—naming and localization—the paper argues that a single conventional detection metric is insufficient.

The metric is explicitly described as semantic rather than lexical. Predicted class names are compared to ground-truth class names using semantic similarity in CLIP text-embedding space. The paper states that SNAP is not based on IoU matching, box overlap, exact string equality, synonym tables, WordNet, or LLM-as-a-judge matching. Instead, it computes a precision–recall curve over confidence-ranked predicted labels and defines SNAP as the area under that curve (Mumcu et al., 14 Jul 2025).

A central property of the formulation is decoupling from localization. The paper states that SNAP is “decoupled from box-level IoU constraints” and evaluates naming “regardless of spatial localization” (Mumcu et al., 14 Jul 2025). In consequence, SNAP is not an end-to-end detection score; it is a naming score intended to complement, rather than replace, localization metrics.

2. Formal construction

For each image, the paper defines the set of ground-truth object labels as

$G = \{l_{gt,1}, l_{gt,2}, \dots, l_{gt,N_{GT}\}$

and the set of predicted labels as

$P = \{l_{p,1}, l_{p,2}, \dots, l_{p,N_P}\},$

where each predicted label $l_{p,i}$ has an associated confidence score $s_i$ (Mumcu et al., 14 Jul 2025).

Class names are mapped into CLIP text-embedding space via

$\text{Emb}(\cdot): \text{Text} \to \mathbb{R}^d,$

and semantic similarity is defined as cosine similarity: $$\text{sim}(l_{p,i}, l_{gt,j}) = \frac{\text{Emb}(l_{p,i}) \cdot \text{Emb}(l_{gt,j})} {\|\text{Emb}(l_{p,i})\| \|\text{Emb}(l_{gt,j})\|}.$$ A predicted label is counted as a true positive if it can be uniquely matched to a ground-truth label such that

$\text{sim}(l_{p,i}, l_{gt,j}) \geq \tau_s,$

where $\tau_s \in [0,1]$ is a semantic similarity threshold, and where the matched ground-truth label has not already been matched by another prediction (Mumcu et al., 14 Jul 2025).

The one-to-one constraint is explicit: each ground-truth label can be matched to at most one prediction. Unmatched predictions are false positives, and unmatched ground-truth labels are false negatives. After sorting all predicted labels across the dataset by confidence, precision and recall at confidence threshold $\theta_s$ are defined as

$$P(\tau_s,\theta_s) = \frac{\sum_{l_{p,i}\in P, s_i \ge \theta_s} \text{TP}(l_{p,i})} {\sum_{l_{p,i}\in P, s_i \ge \theta_s} \text{TP}(l_{p,i})+\text{FP}(l_{p,i})}$$

and

$P = \{l_{p,1}, l_{p,2}, \dots, l_{p,N_P}\},$0

with SNAP then defined as

$P = \{l_{p,1}, l_{p,2}, \dots, l_{p,N_P}\},$1

The paper notes that the printed equations contain some malformed LaTeX, but the intended AP-style meaning is clear: predictions are ranked by confidence, matched one-to-one against ground-truth labels by semantic threshold, and scored by area under the resulting precision–recall curve (Mumcu et al., 14 Jul 2025).

3. Evaluation protocol

The reported protocol constructs, for each image, a ground-truth label set $P = \{l_{p,1}, l_{p,2}, \dots, l_{p,N_P}\},$2 from semantic class annotations and a predicted label set $P = \{l_{p,1}, l_{p,2}, \dots, l_{p,N_P}\},$3 with confidence scores. Semantic similarity is computed for each predicted–ground-truth label pair using CLIP text embeddings. At a chosen threshold $P = \{l_{p,1}, l_{p,2}, \dots, l_{p,N_P}\},$4, each prediction is tested against currently unmatched ground-truth labels in the same image; if at least one label satisfies the threshold, the prediction is counted as a true positive and matched, otherwise it is a false positive. Ground-truth labels that remain unmatched contribute false negatives (Mumcu et al., 14 Jul 2025).

Across the whole dataset, predictions are sorted by descending confidence. Sweeping the confidence threshold $P = \{l_{p,1}, l_{p,2}, \dots, l_{p,N_P}\},$5 yields a precision–recall curve, and integrating under that curve produces SNAP. For reporting an average over multiple semantic thresholds, the paper states: “For reporting an average over multiple semantic similarity thresholds (e.g., [email protected]:.9), we compute the Semantic Name Average Precision (SNAP) at each threshold value between 0.6 and 0.9, incremented by 0.1.” The thresholds are therefore $P = \{l_{p,1}, l_{p,2}, \dots, l_{p,N_P}\},$6, $P = \{l_{p,1}, l_{p,2}, \dots, l_{p,N_P}\},$7, $P = \{l_{p,1}, l_{p,2}, \dots, l_{p,N_P}\},$8, and $P = \{l_{p,1}, l_{p,2}, \dots, l_{p,N_P}\},$9 (Mumcu et al., 14 Jul 2025).

The paper also reports interval groups labeled LO, MI, and HI. For SNAP, these intervals correspond to increasing semantic similarity strictness. However, the exact numeric partition of LO, MI, and HI is not explicitly enumerated for SNAP in the provided text. The text specifies only that higher intervals correspond to stricter semantic acceptance (Mumcu et al., 14 Jul 2025).

The same source also establishes a practical restriction: SNAP requires datasets with semantic ground-truth labels. It is reported on LVIS-Minival and COCO-Val, but not on COCO-OOD, because “it lacks semantic labeling” (Mumcu et al., 14 Jul 2025).

4. Relationship to AP, CAAP, and semantic ranking metrics

SNAP is structurally AP-like but differs from standard object-detection AP in what constitutes a match. Standard AP in conventional detection jointly entangles localization and exact class correctness under a fixed vocabulary. SNAP replaces exact class identity with semantic similarity in embedding space and removes IoU from the matching rule altogether (Mumcu et al., 14 Jul 2025).

The paper’s contrast with CAAP is explicit. CAAP measures localization only: class labels are ignored, matching is based on IoU, and AP is computed over box detections. SNAP measures naming only: localization is ignored, matching is based on semantic similarity in CLIP text space, and AP is computed over predicted labels. The authors argue for “the joint use of CAAP and SNAP over a single combined metric, which may obscure this distinction” (Mumcu et al., 14 Jul 2025).

This design places SNAP within a broader family of semanticized ranking metrics. A related but distinct example is Semantic R-Precision (SemR-p), which integrates semantic similarity into an R-precision framework for keyphrase evaluation (Venturini et al., 5 Jun 2026). SemR-p evaluates only the top $l_{p,i}$0 predictions, where $l_{p,i}$1 is the number of references, and assigns each prediction either full credit for an exact stem match or a semantic score obtained by averaging cosine similarities to the top-$l_{p,i}$2 most similar reference phrases. Unlike SNAP, SemR-p is not AP-style: it is rank-aware through top-$l_{p,i}$3 truncation rather than prefix-sensitive cumulative precision, and it does not perform one-to-one assignment between predictions and references (Venturini et al., 5 Jun 2026).

This comparison suggests a useful conceptual distinction. SemR-p provides a semantic relevance function over ranked outputs, but SNAP adds AP-style cumulative ranking behavior and one-to-one matching. A plausible implication is that SNAP is more directly aligned with retrieval-style notions of duplicate control and confidence-ranked precision, while SemR-p is more naturally interpreted as average semantic relevance over a fixed top segment (Venturini et al., 5 Jun 2026).

Work on AP optimization is also relevant at the objective-design level. “Rethinking the Optimization of Average Precision: Only Penalizing Negative Instances before Positive Ones is Enough” argues that AP loss is fundamentally driven by negative-before-positive ranking violations and proposes the PNP family of losses as surrogates for AP optimization (Li et al., 2021). SNAP is not discussed there, but because SNAP is formally an AP-style ranking metric, the paper’s decomposition of AP into ranking violations is pertinent to any attempt to optimize SNAP rather than merely compute it. This suggests that, if SNAP were used as a training objective, the essential optimization target would be semantically incorrect names ranked ahead of semantically acceptable ones, although the binary positive/negative structure assumed in PNP would require adaptation to semantic thresholds or graded similarity (Li et al., 2021).

5. Reported results and empirical behavior

The main reported SNAP values appear in Table 1 of the LAOD paper. On LVIS-Minival, the paper reports

$l_{p,i}$4

On COCO-Val, it reports

$l_{p,i}$5

On COCO-OOD, SNAP is not reported because the dataset lacks semantic labels (Mumcu et al., 14 Jul 2025).

The paper interprets COCO-Val as yielding the highest SNAP scores overall, citing “strong semantic alignment between LLM-generated labels and ground truth categories.” By contrast, LVIS-Minival scores are described as comparatively lower due to the long-tail distribution and dense object annotations in LVIS, which make both localization and naming more difficult (Mumcu et al., 14 Jul 2025).

No direct baseline comparison is provided for SNAP. The paper explicitly states that “there is currently no existing method that can be directly compared in terms of CAAP and SNAP results” (Mumcu et al., 14 Jul 2025). This is an important methodological point: the reported numbers establish the behavior of the proposed system under the new metric, but they do not yet define a comparative benchmark landscape in the way that mAP or Recall@k do in more established evaluation regimes.

The qualitative examples illustrate the kind of behavior SNAP is meant to reward. The paper describes examples in which the method detects and names objects such as “broom,” “faucet,” “glasses,” and “cardboard box,” including cases where the named objects are not part of the fixed known-vocabulary regime (Mumcu et al., 14 Jul 2025). These examples support the paper’s claim that semantic naming requires a metric distinct from unknown-object detection or prompt-conditioned open-vocabulary AP.

6. Assumptions, ambiguities, and limitations

SNAP’s central assumption is that CLIP text-space cosine similarity is an adequate operationalization of semantic naming correctness. The paper relies entirely on CLIP embeddings for semantic matching and does not use WordNet, manually curated synonym dictionaries, LLM adjudication, or human evaluation (Mumcu et al., 14 Jul 2025). This yields a concrete and reproducible matching rule, but it also means the metric inherits the strengths and weaknesses of CLIP’s text representation.

Several implementation details remain underspecified. The paper does not fully explain how confidence scores $l_{p,i}$6 are obtained for naming-only predictions, especially if names originate in an LLM-generated vocabulary rather than detector outputs. It is also not explicit about whether repeated labels from multiple boxes are treated as separate prediction instances or collapsed into unique image-level names. Likewise, when multiple unmatched ground-truth labels exceed the semantic threshold for a prediction, the exact tie-breaking strategy is not specified; the paper states only that a valid unmatched ground-truth label must exist (Mumcu et al., 14 Jul 2025).

The LO/MI/HI groupings for SNAP are not numerically spelled out in the provided text, and the AP interpolation method is not further specified. These gaps do not obscure the conceptual structure of the metric, but they do limit exact reproducibility.

A further limitation is dependence on dataset label strings. Because matches are defined against the semantic names present in the benchmark, dataset taxonomy remains consequential even when exact lexical identity is relaxed. The paper explicitly notes that SNAP cannot be computed on datasets such as COCO-OOD that provide unknown-object annotations without semantic names (Mumcu et al., 14 Jul 2025).

Finally, the metric does not resolve ontological ambiguity beyond what CLIP similarity already encodes. If a benchmark labels an object as “car” while a prediction names “wheel,” or labels “person” while a system predicts “shirt,” the metric reduces that discrepancy to embedding similarity rather than reasoning over part–whole relations or taxonomic hierarchy. This suggests that SNAP softens label mismatch, but does not solve the deeper problem of object granularity.

7. Significance and broader context

SNAP addresses a specific evaluation gap in open-world and open-vocabulary detection: traditional metrics can measure localization and, under fixed vocabularies, class correctness, but they do not directly measure whether an autonomous system has named objects appropriately when the names are generated dynamically rather than supplied as prompts (Mumcu et al., 14 Jul 2025). In that sense, SNAP formalizes semantic naming as a first-class evaluative dimension.

Its significance lies partly in decomposition. By splitting evaluation into CAAP for localization and SNAP for naming, the LAOD framework treats open-world perception as a composite capability rather than a single score. This decomposition is consistent with broader trends in evaluation research that replace exact lexical matching with semantic similarity while preserving ranking structure. SemR-p exemplifies this tendency in keyphrase generation, where semantic equivalence and ranking both matter but are not well captured by exact-match metrics alone (Venturini et al., 5 Jun 2026).

At the same time, SNAP remains a proposal rather than a standardized benchmark instrument. The absence of direct baselines, the dependence on CLIP as the semantic judge, and several underspecified implementation details mean that the metric is conceptually clear but not yet institutionally stabilized. A plausible implication is that future work may refine the matching rule, specify duplicate handling more precisely, or investigate alternatives to thresholded CLIP similarity.

The AP literature also indicates possible future directions. If SNAP were to be optimized during training rather than used only for evaluation, AP-oriented surrogate losses such as PNP provide a potential starting point for designing semantic ranking objectives that focus on incorrect predictions outranking correct ones (Li et al., 2021). Because SNAP uses semantic acceptance rather than fixed categorical relevance, any such extension would likely need to reconcile AP’s binary ranking logic with semantic thresholds or graded similarity.

Taken together, SNAP can be understood as an AP-style metric for semantic naming quality in autonomous open-world detection, grounded in confidence-ranked one-to-one matching in CLIP embedding space. Its main contribution is to separate naming from localization without reverting to exact lexical identity, thereby making semantic label generation measurable in its own right (Mumcu et al., 14 Jul 2025).