---
title: 'NegRefine: Zero-Shot OOD Framework'
url: https://www.emergentmind.com/topics/negrefine
type: topic
---

# NegRefine: Zero-Shot OOD Framework

NegRefine is a zero-shot out-of-distribution (OOD) detection framework for vision-language models (VLMs), motivated by limitations in negative label-based methods that rely on large lexicons of negative textual labels. NegRefine systematically refines the negative label set and introduces a multi-matching-aware scoring function to mitigate false positives caused by label overlaps and improper negative sampling, enabling robust separation of in-distribution (ID) and OOD examples without extra fine-tuning [2507.09795].

## 1. Limitations of Prior Negative Label-Based OOD Methods

Negative label-based OOD detection methods, such as NegLabel and CSP, operate by constructing a large negative lexicon \(Y_{\text{neg}}\) (typically from WordNet) and contrasting a VLM’s similarity to \(Y_{\text{in}}\) (ID labels) versus \(Y_{\text{neg}}\). While this strategy is effective on broad classes, three pathologies degrade performance:

- **Subcategory Overlap**: Many negative labels are in fact subcategories of ID classes; since CLIP typically assigns higher similarity to specific (subcategory) prompts, images of e.g. “daisy” can match more strongly to “blue-eyed african daisy” (which is treated as OOD), causing misclassification.
- **Proper-Noun Matches**: Negative sets drawn from WordNet contain frequent proper noun entries (e.g., country names, places) that CLIP associates with generic concepts (e.g., “costa rica” matches “beach”). This increases false OOD detections when such negatives are spuriously matched.
- **Multi-Label Matching**: Real-world images often trigger both ID and (semantically related) negative labels, and in naive softmax-ratio scoring, negatives can dominate the score even when the ID class is present (e.g., a photo of a “bee” matches “tickseed sunflower,” driving OOD decision).

These effects result in high ID false positive rates and reduced robustness for existing zero-shot OOD methods [2507.09795].

## 2. Negative Label Filtering Mechanism

NegRefine filters the negative label set by identifying and removing proper nouns and subcategory labels. Filtering is executed as a one-time offline procedure (Alg. 1 in [2507.09795]), leveraging both CLIP and a large language model (LLM):

- **Proper Noun Detection**: Each candidate negative label \(w\) is posed to the LLM (“Is \(w\) a proper noun (name of an entity)?”). Labels for which the LLM returns “Yes” are collected into \(\mathcal{N}\) and excluded.
- **Subcategory Detection**: For non-proper-noun \(w\), the algorithm finds its top-\(n\) nearest ID class labels \(L(w)\) using CLIP text-embedding similarity, then poses to the LLM for each \(l\in L(w)\): “Is \(w\) a subcategory of \(l\)?” If true for any \(l\), \(w\) is added to the subcategory set \(\mathcal{S}\).

The filtered negative set is 
\[
Y_{\rm neg}' = \mathcal{L}^- \setminus (\mathcal{N} \cup \mathcal{S}).
\]
Empirically, this removes ∼20.6% of negatives as proper nouns and ∼3.6% as subcategories, reducing adverse overlap and ID confusion [2507.09795].

## 3. Multi-Matching-Aware Scoring Function

The NegRefine scoring system augments the standard softmax-ratio (NegLabel-style) OOD score to explicitly account for cases where images match multiple labels. The overall ID score is:
\[
S_{\rm base}(x) = 
\frac
{\sum_{i=1}^K \exp(\phi(x, y_i)/\tau)}
{\sum_{i=1}^K \exp(\phi(x, y_i)/\tau) + \sum_{j=1}^M \exp(\phi(x, \tilde y_j)/\tau)}
\tag{1}
\]
where \(y_i\) are ID labels, \(\tilde y_j\) are filtered negatives (\(Y_{\text{neg}}'\)), \(\phi\) is CLIP cosine similarity, \(\tau=0.01\).

To address multi-label matching, NegRefine selects the top-\(k\) ID (\(T_{\rm in}(x)\)) and negative (\(T_{\rm neg}(x)\)) labels for each image. For each \((y_i, \tilde y_j)\), it constructs a composite caption “\(y_i\) and \(\tilde y_j\)” and computes:
\[
p_{i,j}(x) = 
\frac{\exp(\phi(x, t_{i,j})/\tau)}
{\exp(\phi(x, t_{i,j})/\tau) + \exp(\phi(x, \tilde y_j)/\tau)}
\]
The multi-matching term is
\[
S_{MM}(x) = \max_{(i,j) \in T_{\rm in} \times T_{\rm neg}} p_{i,j}(x)
\tag{2}
\]
The final ID score is:
\[
S(x) = S_{\rm base}(x) + \alpha S_{MM}(x),\quad \alpha \approx 2
\tag{3}
\]
An image is classified as in-distribution if \(S(x) \geq \gamma\), where \(\gamma\) is calibrated so that 95% of ID validation samples are accepted.

## 4. Algorithmic Structure and Computational Complexity

NegRefine’s workflow consists of an offline filtering stage and an efficient, fully zero-shot scoring scheme:

- **Filtering Stage (NegFilter)**: For each negative label, LLM queries (for proper-noun and subcategory) are executed, and CLIP text similarity is computed to retrieve \(n = 10\) most similar in-distribution classes for subcategory assessment. Complexity is \(O(|Y_{\text{neg}}|(n + \text{LLM}) + |Y_{\text{neg}}|\log|Y_{\text{in}}|)\).
- **Inference per Image**: 
    1. Compute CLIP similarities to all ID and filtered negative labels.
    2. Compute \(S_{\text{base}}\) via (1).
    3. Select top-\(k\) from each set.
    4. For each of \(k^2\) pairs, compute composite caption similarity and joint scores (\(S_{MM}\) via (2)).
    5. Combine as \(S(x)\).
  The per-image computational cost is \(O(K + M + k^2)\) CLIP calls, with K=number of ID labels, M=filtered negatives, and \(k \sim 5\).

## 5. Experimental Evaluation and Effectiveness

NegRefine is evaluated on ImageNet-1K (ID) with challenging OOD sets: iNaturalist, OpenImage-O, Clean, and NINCO. Metrics include AUROC (higher is better) and FPR95 (lower is better). The method is compared to CSP and ablated versions. Main results (ViT-B/16, zero-shot):

|       Method        | AUROC (avg) | FPR95 (avg) |
|:-------------------|:-----------:|:-----------:|
| CSP                |   89.97     |   34.47     |
| NegRefine          |   91.79     |   30.12     |

Component-wise ablation (ImageNet-1K, FPR95 avg):

| Variant                 | NegFilter | $S_{MM}$ | FPR95             |
|-------------------------|-----------|----------|-------------------|
| CSP (none)              |     –     |    –     | 34.47             |
| + NegFilter only        |     ✓     |    –     | 31.88             |
| + $S_{MM}$ only         |     –     |    ✓     | 31.70             |
| NegRefine (both)        |     ✓     |    ✓     | 30.12             |

Both filtering and multi-matching components yield measurable additive improvements.

Additional findings:
- Proper-noun filtering removes ~20.6% of raw negatives; subcategory filtering ~3.6%.
- Extended ablations show little effect from LLM choice, consistent gains across architectures (ResNet50, ViT variants), and optimal scoring settings at \(k=5, \alpha=2\).

## 6. Hyperparameters and Tuning Procedures

Relevant parameters include:
- Negative fraction \(p = 15\%\) of WordNet classes, balancing OOD coverage and noise.
- CLIP temperature \(\tau=0.01\).
- LLM filter neighbor pool \(n=10\).
- Matching parameters \(k=5\), \(\alpha=2\); grid search on \(\alpha\in[1,4]\), \(k\in[3,7]\) is recommended.
- Decision threshold \(\gamma\) set for 95% ID recall.

This configuration is robust across OOD sets and CLIP backbones. A plausible implication is that performance is minimally sensitive to LLM model and moderately sensitive to matching hyperparameters.

## 7. Key Insights, Limitations, and Future Directions

Empirical evidence indicates that filtering “bad” negatives (proper nouns, subcategories) substantially reduces false OOD alarms for ID data; modeling multi-label matching with CLIP-style joint captions further addresses false positives when images match both ID and (otherwise problematic) negatives. NegRefine achieves fully zero-shot operation and is complementary to prompt tuning and test-time adaptation strategies (e.g., LAPT, AdaNeg).

Limitations include the reliance on LLMs for label filtering and the restriction to binary (“y and $\tilde{y}$”) composite captions—richer n-ary constructs may provide additional robustness. A plausible implication is that as taxonomic resources like WordNet evolve, the cost of LLM-based filtering may be reduced. Extension to few-shot or supervised OOD, or application to alternative vision-language backbones (e.g., ALIGN), is outlined as future research.

Source: https://www.emergentmind.com/topics/negrefine