TTL: Test-time Textual Learning for OOD Detection with Pretrained Vision-Language Models

Abstract: Vision-LLMs (VLMs) such as CLIP exhibit strong Out-of-distribution (OOD) detection capabilities by aligning visual and textual representations. Recent CLIP-based test-time adaptation methods further improve detection performance by incorporating external OOD labels. However, such labels are finite and fixed, while the real OOD semantic space is inherently open-ended. Consequently, fixed labels fail to represent the diverse and evolving OOD semantics encountered in test streams. To address this limitation, we introduce Test-time Textual Learning (TTL), a framework that dynamically learns OOD textual semantics from unlabeled test streams, without relying on external OOD labels. TTL updates learnable prompts using pseudo-labeled test samples to capture emerging OOD knowledge. To suppress noise introduced by pseudo-labels, we introduce an OOD knowledge purification strategy that selects reliable OOD samples for adaptation while suppressing noise. In addition, TTL maintains an OOD Textual Knowledge Bank that stores high-quality textual features, providing stable score calibration across batches. Extensive experiments on two standard benchmarks with nine OOD datasets demonstrate that TTL consistently achieves state-of-the-art performance, highlighting the value of textual adaptation for robust test-time OOD detection. Our code is available at https://github.com/figec/TTL.

• The paper introduces TTL that continuously adapts textual prompts during testing to address limitations of fixed label sets in OOD detection.
• It employs an OOD Knowledge Purification mechanism to mitigate pseudo-label noise and ensure robust, calibrated textual feature alignment.
• The framework achieves state-of-the-art results on major benchmarks, significantly improving AUROC and reducing FPR95 metrics.

Test-time Textual Learning for OOD Detection with Pretrained Vision-LLMs

Introduction

This paper introduces Test-time Textual Learning (TTL), a vision-language adaptation framework for out-of-distribution (OOD) detection using pretrained models, with a focus on CLIP. TTL addresses the inadequacy of existing approaches that employ a fixed, finite set of OOD textual labels for visual-textual alignment, which fails to accommodate the open-ended diversity of OOD semantics encountered in real-world deployment. TTL proposes continuous adaptation of textual prompts using unlabeled test streams, dynamically acquiring emergent OOD knowledge on-the-fly, and mitigating the reliance on any predefined OOD categories.

A key innovation is the design of a learnable OOD prompt for each ID class, whose prefix is updated online. This adaptation is guided by pseudo-labeled test samples and is regularized with an OOD Knowledge Purification (OKP) mechanism to reduce the impact of noisy pseudo-labels. Furthermore, the learned OOD textual features are organized into an OOD Textual Knowledge Bank (OKB), providing stable calibration and accumulation of robust OOD semantics across batches.

Figure 1: Existing methods (a) update visual features within a static text space, while TTL (b) learns new text space during test time, obtaining improved alignment between adaptively updated text and image features (c).

Limitations of Prior OOD Adaptation Paradigms

Traditional OOD detection for computer vision operates by identifying unfamiliar samples through single-modality discriminative cues—logits, features, or feature statistics. Vision-LLMs (VLMs), specifically CLIP, introduced cross-modal alignment as a powerful OOD detection tool. However, attempts at adapting CLIP to the OOD problem have remained tied to a limited, static text space: external OOD labels or textual proxies collected through manual curation or large-scale corpora.

Approaches such as AdaNeg, while leveraging both vision and language modalities, still depend on external finite textual resources. Their inability to encompass unforeseen OOD semantics constrains adaptability and limits discriminative capacity, leading to feature drift and accumulation of failure cases at test time. TTL directly addresses these constraints by decoupling OOD textual adaptation from reliance on any fixed label set.

Figure 2: TTL dynamically updates prompts at test time using pseudo OOD labels, purifies them based on confidence thresholds, and accumulates a knowledge bank for robust and stable score calibration.

TTL Framework

Dynamic OOD Prompt Learning

TTL parameterizes a set of learnable OOD prompts (one per ID class) whose prefixes are optimized at test time. Each OOD prompt is initialized with the same template as its corresponding ID prompt to exploit pretrained model priors and semantic locality. The adaptation process modifies only the prefix embedding of the OOD prompt, freezing the class name and the text and image encoders.

Pseudo-labels for OOD detection are produced using the base OOD detector, and batch-wise prompt updates are performed to maximize the semantic similarity between OOD-labeled image features and OOD prompt features. The online optimization is regularized by an OOD-focused minority-balanced loss $\mathcal{L}_{\text{OMB}}$, which balances the skewed ID/OOD proportions common in dynamic test streams.

OOD Knowledge Purification

Pseudo-label noise introduces bias, particularly as misclassified ID (boundary) samples are repeatedly reinforced. TTL introduces an explicit purification mechanism: given a batch of pseudo-OOD samples, samples are partitioned into high-confidence and low-confidence subsets using an adaptive threshold. The loss $\mathcal{L}_{\text{OKP}}$ is defined to explicitly maximize the gap in predicted OOD probability between these two groups, ensuring that the learnable prompts are updated toward clear OOD evidence and away from ID boundary contamination. The total learning objective is $$\mathcal{L} = \mathcal{L}_{\text{OMB}} + \alpha \mathcal{L}_{\text{OKP}}$$, with $\alpha$ modulating the tradeoff.

OOD Textual Knowledge Bank and Score Calibration

To mitigate catastrophic forgetting and batch-level semantic drift, TTL maintains an OOD Textual Knowledge Bank (OKB). Each time the OOD prompts are updated, their text embeddings are scored according to their minimum similarity to any ID prompt (favoring prompts maximally unlike the ID space), and only the top-$K$ OOD prompt features are retained. This memory bank provides a stable, diverse reference across test-time batches.

During inference, the OKB is used to calibrate the base OOD detector’s score: the OOD score for each test sample is augmented with the maximum similarity between its image feature and the stored OOD textual features, scaled by a tunable coefficient. This fusion mechanism shifts ambiguous samples more decisively away from the ID/OOD decision boundary.

Figure 3: TTL calibration produces a larger separation between ID and OOD score distributions, facilitating robust thresholding for OOD detection.

Experimental Results

TTL achieves state-of-the-art OOD detection across two major benchmarks (ImageNet-1K and CIFAR-100) and nine OOD datasets. Without any external OOD labels, TTL attains superior average results to both test-time adaptation (TTA) and training-based methods. On ImageNet-1K, TTL yields an AUROC of 97.29% and FPR95 of 12.46%, outperforming the prior best TTA method (AdaNeg) by 7% in FPR95, even when AdaNeg leverages external OOD textual resources.

TTL further demonstrates strong cross-dataset generalization, substantial improvements when integrated with varying base OOD detectors, and marked gains on challenging OOD splits such as Places365 and Texture. The ablation studies substantiate the contributions of $\mathcal{L}_{\text{OMB}}$, $\mathcal{L}_{\text{OKP}}$, and OKB individually and in concert.

Figure 4: TTL consistently boosts performance when applied atop multiple base OOD detectors, highlighting detector-agnostic adaptability.

Sensitivity Analysis and Interpretability

TTL exhibits robustness to hyperparameter settings (batch size, knowledge bank capacity, loss weights, fusion coefficients), with consistently superior performance across reasonable variations.

Prompt initialization with manual templates (“a photo of a [class]”) enables the most effective test-time adaptation; in contrast, random prompt initialization impairs alignment with model priors. TTL is relatively insensitive to OKB update strategy, but semantic scoring for OOD prompt retention maximizes effectiveness.

Figure 5: TTL’s performance is stable across wide ranges of hyperparameters, consistently outperforming AdaNeg’s dashed reference lines.

Figure 6: t-SNE embedding shows TTL’s learned OOD textual features cluster more closely with true OOD samples, confirming semantic targeting.

Implications and Theoretical Impact

TTL demonstrates that OOD detection using VLMs benefits substantially from online adaptation of the textual modality, not merely the visual or multimodal alignment. By removing dependence on any fixed OOD label set, TTL better matches the open-world, streaming nature and unpredictable semantic drift of real deployment. The memory bank mechanism serves as a minimal but stable means for continual adaptation, robust to batch order and label noise.

The explicit purification loss and the empirical gains observed with weak base detectors suggest the approach is resilient to pseudo-label noise, with TTL often acting as a robust amplifier of even weak separation signals.

Beyond direct OOD detection application, TTL’s approach to unsupervised, batch-wise semantic prompt learning opens the way for continual test-time adaptation in VLMs across broader image-text understanding domains. One emergent research direction is the unified framework for continual, open-set adaptation in both vision and language via efficient, low-footprint prompt updates.

Conclusion

TTL constitutes a substantive advance in test-time OOD detection by shifting adaptation focus onto the textual representation space and enabling dynamic, unsupervised acquisition of OOD semantics. The proposed knowledge purification and memory mechanisms eliminate catastrophic forgetting and ensure robust calibration across distribution shifts. TTL’s strong empirical performance—surpassing methods dependent on external resources—validates the utility of adaptive textual learning for real-world OOD robustness in VLMs.