Activation Matters: Test-time Activated Negative Labels for OOD Detection with Vision-Language Models

Abstract: Out-of-distribution (OOD) detection aims to identify samples that deviate from in-distribution (ID). One popular pipeline addresses this by introducing negative labels distant from ID classes and detecting OOD based on their distance to these labels. However, such labels may present poor activation on OOD samples, failing to capture the OOD characteristics. To address this, we propose \underline{T}est-time \underline{A}ctivated \underline{N}egative \underline{L}abels (TANL) by dynamically evaluating activation levels across the corpus dataset and mining candidate labels with high activation responses during the testing process. Specifically, TANL identifies high-confidence test images online and accumulates their assignment probabilities over the corpus to construct a label activation metric. Such a metric leverages historical test samples to adaptively align with the test distribution, enabling the selection of distribution-adaptive activated negative labels. By further exploring the activation information within the current testing batch, we introduce a more fine-grained, batch-adaptive variant. To fully utilize label activation knowledge, we propose an activation-aware score function that emphasizes negative labels with stronger activations, boosting performance and enhancing its robustness to the label number. Our TANL is training-free, test-efficient, and grounded in theoretical justification. Experiments on diverse backbones and wide task settings validate its effectiveness. Notably, on the large-scale ImageNet benchmark, TANL significantly reduces the FPR95 from 17.5\% to 9.8\%. Codes are available at \href{https://github.com/YBZh/OpenOOD-VLM}{YBZh/OpenOOD-VLM}.

• The paper presents a novel adaptive framework (TANL) that dynamically selects negative labels based on test-time activation, improving OOD detection.
• The paper demonstrates significant performance gains, notably reducing ImageNet FPR95 by 7.7% compared to prior negative label methods.
• The method leverages FIFO queues and activation-aware scoring, ensuring robust adaptability across diverse benchmarks and real-world conditions.

Test-time Activated Negative Labels for OOD Detection with Vision-LLMs

Introduction and Motivation

Out-of-distribution (OOD) detection constitutes a critical challenge in reliable artificial intelligence, particularly in open-world deployment scenarios. Modern vision-LLMs (VLMs), such as CLIP, have significantly advanced zero-shot recognition by leveraging unified representations of image and textual modalities. However, their generalization is imperfect: OOD samples can still be misclassified with high confidence, threatening system robustness and safety.

Negative label-based approaches to OOD detection—most notably, methods that mine negative labels far from the in-distribution (ID) classes (e.g., NegLabel)—represent a line of work in which labels semantically distant from the ID classes are treated as negative anchors. While these negative labels can theoretically help identify OOD samples, their effectiveness is modulated by their "activation": the extent to which images (ID or OOD) are likely to be assigned to these negative classes by the VLM. Empirical analysis reveals that many negative labels are weakly activated on OOD data, resulting in noise and degradation in detection performance (Figure 1).

Figure 1: Distribution and impact of activation scores across ID/OOD samples and the relationship to FPR95.

This motivates a dynamic and adaptive strategy for negative label selection, aligned to the activation statistics encountered during deployment, rather than a priori mining. The TANL framework addresses this gap.

TANL: Test-time Activated Negative Labels

Label Activation Metric and its Role

TANL’s core insight is that the utility of a negative label for OOD detection is governed by its test-time activation. Formally, the activation score for a label on a dataset is defined as the average normalized assignment probability to that label across the dataset. High OOD detection efficacy arises from negative labels with high OOD activation and low ID activation. Experimental evidence underscores that the majority of static negative labels exhibit poor OOD activation, and including such weakly-activated labels impairs detection performance.

Activation-aware Negative Label Mining

TANL introduces an adaptive mechanism for mining negative labels at test time, guided by ongoing estimates of label activation. This is enabled via FIFO queues that cache recent high-confidence positive (ID) and negative (OOD) image instances. Activation metrics are dynamically updated using these queues, approximating the unknown test-time OOD distribution.

For each batch, a batch-adaptive variant further refines label activation estimation by aggregating statistics over both historical and current batch samples, mediated by a hyperparameter $\alpha$ that trades off temporal smoothness and responsiveness to abrupt shifts.

Figure 2: Overview of the TANL framework. Activated negative labels are dynamically identified and mined; label activations inform adaptive selection and scoring during test time.

Activation-aware Score Function

Recognizing the varied importance of negative labels, TANL introduces a scoring function that weights negative labels according to their activation. The score prioritizes negative labels with stronger OOD activation, mitigating sensitivity to the label set size and emphasizing robust detection. This adaptation mechanism ensures that new or shifting OOD distributions can be recognized, as the set of influential negative labels can update online.

(TANL’s approach to activation-aware scoring fundamentally differs from static or activation-agnostic aggregation, providing improved robustness and reduced error—see Figure 3.)

Figure 3: Influence of the number of negative labels on FPR95 for various selection and scoring methods. Activation-aware strategies demonstrate both improved performance and reduced sensitivity to label count.

Empirical Evaluation

ImageNet and Standard Benchmarks

Extensive benchmarks on ImageNet and multiple OOD datasets show that TANL delivers state-of-the-art performance in both AUROC and FPR95 across diverse settings, including zero-shot, training-free scenarios. For instance, on ImageNet, TANL achieves a remarkable reduction in FPR95 from 17.5% (CSP) and 25.4% (NegLabel) to 9.8%—a 7.7% improvement over the previously leading approach.

OpenOOD, Full-Spectrum, and Medical Benchmarks

The framework maintains its efficacy across near-OOD, far-OOD, full-spectrum, and challenging medical datasets, further validating the robustness and generality of the approach. Notably, the improvements persist under severe covariate shift and distributional variations (Figure 4).

Figure 4: Visualization of the corpus dataset, highlighting the adaptively selected, highly-activated negative labels for a far-OOD regime.

Ablation and Analysis

Component ablation demonstrates the pivotal role of dynamic activation-based negative label selection and activation-aware scoring. Batch-adaptive updating, label selection criteria, and the choice of activation metrics are systematically studied (see Figure 5 and Figure 6).

Figure 5: Influence of the activation queue length on detection performance, indicating rapid convergence and limited sensitivity beyond moderate queue sizes.

Figure 6: Impact of the corpus dataset size on TANL’s OOD detection performance.

Early-stage errors, sample order, and the number of test samples are also investigated. The method exhibits robustness to initial misclassifications, rapid convergence with limited data, and maintains accuracy under temporal shifts, as demonstrated by further experimental figures.

Theoretical Justification

TANL’s method is grounded in a formal analysis of multi-label classification detection error. The theoretical results elucidate how the inclusion of negative labels with high OOD activation and low ID activation (i.e., $p_1 - p_2 < 0$) directly translates into reduced false-positive rates. This optimality criterion is operationalized in TANL via its test-time adaptation mechanics.

Practical and Theoretical Implications

TANL offers substantial impact for deployment-safe AI systems, requiring no retraining, and incurring minimal test-time overhead. The method’s capacity to robustly identify OOD samples through dynamically mined, adaptation-aligned negative labels fundamentally enhances the practical utility and generality of VLMs for real-world tasks. Its scalability across architectures (e.g., ResNet-50 to ViT-L/14) and settings—ranging from natural scenes to medical imaging—demonstrates domain-agnostic applicability.

Theoretically, the work clarifies the importance of distribution-adaptive negative anchors and the utility of temporal test-time statistics for online OOD adaptation, bridging gaps in the prior literature on negative label mining.

Limitations and Potential Extensions

A primary limitation is dependence on the corpus dataset’s coverage; if OOD semantics are not well-represented in the corpus or accessible to the frozen text encoder, detection improvements may be limited (particularly in domain-specialized tasks, such as medical OOD detection). Extensions may involve constructing domain-specific corpora and leveraging specialized VLMs.

Conclusion

TANL introduces a paradigm in OOD detection with VLMs: negative labels are not statically selected but are dynamically activated, mined, and scored at test time, aligning closely with evolving data distributions. This dynamic adaptation yields high test efficiency, theoretical justification, scalability, and substantial empirical gains across broad OOD detection scenarios. TANL raises important points for future research in both adaptive OOD detection and practical VLM deployment.

Figure 7: Early error rate analysis demonstrates TANL’s robustness to initial erroneous cache insertions.