NegRefine: Zero-Shot OOD Framework

• NegRefine is a zero-shot OOD detection framework that systematically refines negative labels to reduce false positives in vision-language models.
• It employs an LLM-driven filtering mechanism to remove proper nouns and subcategory overlaps, enhancing reliability by mitigating label confusion.
• The method leverages a multi-matching-aware scoring function and shows improved AUROC and FPR95 metrics on benchmark datasets compared to prior approaches.

NegRefine is a zero-shot out-of-distribution (OOD) detection framework for vision-LLMs (VLMs), motivated by limitations in negative label-based methods that rely on large lexicons of negative textual labels. NegRefine systematically refines the negative label set and introduces a multi-matching-aware scoring function to mitigate false positives caused by label overlaps and improper negative sampling, enabling robust separation of in-distribution (ID) and OOD examples without extra fine-tuning (Ansari et al., 13 Jul 2025).

1. Limitations of Prior Negative Label-Based OOD Methods

Negative label-based OOD detection methods, such as NegLabel and CSP, operate by constructing a large negative lexicon $Y_{\text{neg}}$ (typically from WordNet) and contrasting a VLM’s similarity to $Y_{\text{in}}$ (ID labels) versus $Y_{\text{neg}}$. While this strategy is effective on broad classes, three pathologies degrade performance:

• Subcategory Overlap: Many negative labels are in fact subcategories of ID classes; since CLIP typically assigns higher similarity to specific (subcategory) prompts, images of e.g. “daisy” can match more strongly to “blue-eyed african daisy” (which is treated as OOD), causing misclassification.
• Proper-Noun Matches: Negative sets drawn from WordNet contain frequent proper noun entries (e.g., country names, places) that CLIP associates with generic concepts (e.g., “costa rica” matches “beach”). This increases false OOD detections when such negatives are spuriously matched.
• Multi-Label Matching: Real-world images often trigger both ID and (semantically related) negative labels, and in naive softmax-ratio scoring, negatives can dominate the score even when the ID class is present (e.g., a photo of a “bee” matches “tickseed sunflower,” driving OOD decision).

These effects result in high ID false positive rates and reduced robustness for existing zero-shot OOD methods (Ansari et al., 13 Jul 2025).

2. Negative Label Filtering Mechanism

NegRefine filters the negative label set by identifying and removing proper nouns and subcategory labels. Filtering is executed as a one-time offline procedure (Alg. 1 in (Ansari et al., 13 Jul 2025)), leveraging both CLIP and a LLM:

• Proper Noun Detection: Each candidate negative label $w$ is posed to the LLM (“Is $w$ a proper noun (name of an entity)?”). Labels for which the LLM returns “Yes” are collected into $\mathcal{N}$ and excluded.
• Subcategory Detection: For non-proper-noun $w$, the algorithm finds its top-$n$ nearest ID class labels $L(w)$ using CLIP text-embedding similarity, then poses to the LLM for each $l\in L(w)$: “Is $Y_{\text{in}}$0 a subcategory of $Y_{\text{in}}$1?” If true for any $Y_{\text{in}}$2, $Y_{\text{in}}$3 is added to the subcategory set $Y_{\text{in}}$4.

The filtered negative set is

$Y_{\text{in}}$5

Empirically, this removes ∼20.6% of negatives as proper nouns and ∼3.6% as subcategories, reducing adverse overlap and ID confusion (Ansari et al., 13 Jul 2025).

3. Multi-Matching-Aware Scoring Function

The NegRefine scoring system augments the standard softmax-ratio (NegLabel-style) OOD score to explicitly account for cases where images match multiple labels. The overall ID score is: $Y_{\text{in}}$6 where $Y_{\text{in}}$7 are ID labels, $Y_{\text{in}}$8 are filtered negatives ($Y_{\text{in}}$9), $Y_{\text{neg}}$0 is CLIP cosine similarity, $Y_{\text{neg}}$1.

To address multi-label matching, NegRefine selects the top-$Y_{\text{neg}}$2 ID ($Y_{\text{neg}}$3) and negative ($Y_{\text{neg}}$4) labels for each image. For each $Y_{\text{neg}}$5, it constructs a composite caption “$Y_{\text{neg}}$6 and $Y_{\text{neg}}$7” and computes: $Y_{\text{neg}}$8 The multi-matching term is

$Y_{\text{neg}}$9

The final ID score is: $w$0 An image is classified as in-distribution if $w$1, where $w$2 is calibrated so that 95% of ID validation samples are accepted.

4. Algorithmic Structure and Computational Complexity

NegRefine’s workflow consists of an offline filtering stage and an efficient, fully zero-shot scoring scheme:

• Filtering Stage (NegFilter): For each negative label, LLM queries (for proper-noun and subcategory) are executed, and CLIP text similarity is computed to retrieve $w$3 most similar in-distribution classes for subcategory assessment. Complexity is $w$4.
• Inference per Image:

1. Compute CLIP similarities to all ID and filtered negative labels. 2. Compute $w$5 via (1). 3. Select top-$w$6 from each set. 4. For each of $w$7 pairs, compute composite caption similarity and joint scores ($w$8 via (2)). 5. Combine as $w$9. The per-image computational cost is $w$0 CLIP calls, with K=number of ID labels, M=filtered negatives, and $w$1.

5. Experimental Evaluation and Effectiveness

NegRefine is evaluated on ImageNet-1K (ID) with challenging OOD sets: iNaturalist, OpenImage-O, Clean, and NINCO. Metrics include AUROC (higher is better) and FPR95 (lower is better). The method is compared to CSP and ablated versions. Main results (ViT-B/16, zero-shot):

Method	AUROC (avg)	FPR95 (avg)
CSP	89.97	34.47
NegRefine	91.79	30.12

Component-wise ablation (ImageNet-1K, FPR95 avg):

Variant	NegFilter	$w$2	FPR95
CSP (none)	–	–	34.47
+ NegFilter only	✓	–	31.88
+ $w$3 only	–	✓	31.70
NegRefine (both)	✓	✓	30.12

Both filtering and multi-matching components yield measurable additive improvements.

Additional findings:

• Proper-noun filtering removes ~20.6% of raw negatives; subcategory filtering ~3.6%.
• Extended ablations show little effect from LLM choice, consistent gains across architectures (ResNet50, ViT variants), and optimal scoring settings at $w$4.

6. Hyperparameters and Tuning Procedures

Relevant parameters include:

• Negative fraction $w$5 of WordNet classes, balancing OOD coverage and noise.
• CLIP temperature $w$6.
• LLM filter neighbor pool $w$7.
• Matching parameters $w$8, $w$9; grid search on $\mathcal{N}$0, $\mathcal{N}$1 is recommended.
• Decision threshold $\mathcal{N}$2 set for 95% ID recall.

This configuration is robust across OOD sets and CLIP backbones. A plausible implication is that performance is minimally sensitive to LLM model and moderately sensitive to matching hyperparameters.

7. Key Insights, Limitations, and Future Directions

Empirical evidence indicates that filtering “bad” negatives (proper nouns, subcategories) substantially reduces false OOD alarms for ID data; modeling multi-label matching with CLIP-style joint captions further addresses false positives when images match both ID and (otherwise problematic) negatives. NegRefine achieves fully zero-shot operation and is complementary to prompt tuning and test-time adaptation strategies (e.g., LAPT, AdaNeg).

Limitations include the reliance on LLMs for label filtering and the restriction to binary (“y and $\mathcal{N}$3”) composite captions—richer n-ary constructs may provide additional robustness. A plausible implication is that as taxonomic resources like WordNet evolve, the cost of LLM-based filtering may be reduced. Extension to few-shot or supervised OOD, or application to alternative vision-language backbones (e.g., ALIGN), is outlined as future research.