GP-Adapter: Gaussian Process CLIP-Adapter for Few-Shot Out-of-Distribution Detection

Abstract: We propose GP-Adapter, a training-free framework that augments CLIP (Contrastive Language-Image Pre-training) with Gaussian Process (GP) uncertainty modeling for few-shot classification and out-of-distribution (OOD) detection. While CLIP achieves strong zero-shot recognition, it yields deterministic similarity scores and offers limited uncertainty information, which is critical under distribution shift and data scarcity. GP-Adapter constructs modality-specific, class-wise one-class GPs on top of frozen CLIP embeddings using an RBF kernel for image features and a linear kernel for text prompts and fuses their predictive statistics to produce a variance-aware confidence score for OOD detection. The method requires no fine-tuning of the CLIP backbone and relies only on a small $K$-shot cache and lightweight hyperparameter selection, with memory cost scaling as $O(CK^2)$ for $C$ classes and $K$ shots. Experiments on ImageNet and multiple OOD benchmarks show that GP-Adapter provides competitive few-shot performance and consistently improves OOD detection when combined with prompt-learning baselines, highlighting the complementarity between GP-based uncertainty modeling and prompt learning. Overall, our results suggest that integrating probabilistic inference with large pre-trained vision-LLMs can improve reliability in low-data and distribution-shifted settings. Code is available at https://github.com/tms-byte/GP-Adapter

• The paper introduces a training-free method by attaching one-class Gaussian Process heads to CLIP embeddings for few-shot OOD detection.
• The approach fuses predictive means and variances via modality-specific kernels, significantly enhancing variance-aware confidence scoring.
• Extensive experiments on ImageNet and OOD datasets demonstrate that GP-Adapter markedly improves AUROC and lowers FPR95 compared to baseline methods.

GP-Adapter: Gaussian Process CLIP-Adapter for Few-Shot Out-of-Distribution Detection

Introduction and Motivation

The increasing deployment of deep learning systems in open-world and safety-critical settings necessitates robust out-of-distribution (OOD) detection, especially under data scarcity conditions typical in medical imaging, industrial inspection, and related domains. While vision-LLMs (VLMs) such as CLIP enable strong zero-shot recognition by aligning images and text in a shared embedding space, their reliance on deterministic similarity scores limits their OOD uncertainty modeling capabilities. Existing prompt- and cache-based CLIP adaptation methods either require potentially costly fine-tuning or struggle to offer principled, interpretable uncertainty estimation—an essential property for reliable OOD detection under distributional shift.

To address these deficiencies, the paper introduces GP-Adapter, a training-free framework that attaches modality-specific, class-wise one-class Gaussian Processes (GPs) atop frozen CLIP embeddings for few-shot OOD detection and classification. The resulting method delivers analytically derived class probabilities and predictive variances, leveraging probabilistic inference to improve uncertainty quantification and OOD detection efficacy while keeping computational overhead and data requirements minimal.

Technical Contributions and Architecture

GP-Adapter instantiates two main technical ideas:

1. Class-wise One-Class GP Modeling: For each class, two separate GPs are constructed for the image and text modalities using a small $K$-shot labeled cache. The paper uses an RBF kernel for image embeddings—exploiting their non-linear geometry—and a linear kernel for text embeddings, which aligns with CLIP’s cosine similarity-based scoring.
2. Variance-Aware Confidence Scoring: At inference, both GPs output predictive mean and variance per class, fused by a weighted combination. The key OOD confidence metric is a "variance-aware" version of Maximum Softmax Probability (MSP), which modulates the traditional MSP by the max–min range of predictive variances across classes for a query. This exploits the property that OOD samples tend to result in more uniform, elevated variance, while in-distribution (ID) samples yield larger cross-class variance ranges.

   Figure 1: Overview of GP-Adapter with one-class Gaussian Processes, fusing modality-specific GP predictive means and variances for variance-aware OOD detection.

The pipeline is strictly training-free: all GP parameters are determined analytically or through lightweight grid search over hyperparameters, obviating any need for gradient-based tuning of either the CLIP backbone or the GP heads. The memory and compute footprint scales as $O(CK^2)$, where $C$ is the number of classes and $K$ the number of support samples per class.

Analysis of Modality-Specific Kernel Design

CLIP's shared embedding space aligns image and text embeddings semantically, yet residual modality gap and geometric discrepancies persist. The paper provides a $t$-SNE analysis that visually demonstrates this misalignment, advising against uniform kernelization across the modalities.

Figure 2: t-SNE visualization of CLIP embeddings demonstrates distinct clustering and systematic offsets between image and text modalities.

Accordingly, the image modality employs an RBF kernel to capture locality and nonlinearity in visual features, while the linear kernel for text prompts maintains compatibility with CLIP’s cosine-based matching. Empirically, modality-specific kernels improve stability in similarity estimation and, by extension, reliable uncertainty modeling.

Variance-Aware OOD Scoring

Unlike prior MSP or energy-based scoring approaches, GP-Adapter’s variance-aware MSP calibrates class probability confidence using the class-wise variance range. The ablation analysis reveals a marked improvement in separating ID and OOD queries by incorporating the proposed variance-aware term compared to only using the variance of the most probable class.

Figure 3: Range of GP predictive variances across classes enhances discrimination between ID and OOD samples.

Figure 4: Ablation shows that variance-aware MSP calibration consistently improves AUROC, especially when combined with strong prompt learning.

This insight underscores that the distribution of predictive variance—rather than just its marginal magnitude—serves as a robust uncertainty signal for OOD detection in few-shot regimes.

Strong Numerical Results

Across extensive experiments on ImageNet-1k and several standardized OOD datasets (iNaturalist, SUN, Places, Texture), GP-Adapter demonstrates statistically significant improvements over strong few-shot prompt-learning and zero-shot OOD baselines. When composed with prompt learning (CoOp/LoCoOp), further synergistic gains are observed for both OOD detection (measured by AUROC and FPR95) and ID Top-1 classification accuracy.

Key numerical results include:

• GP-Adapter+LoCoOp achieves the lowest average FPR95 and highest average AUROC in the few-shot (16-shot, default ViT-B/16 backbone) regime, outperforming all prompt-learning and zero-shot competitors.
• The hybrid approach consistently delivers enhanced OOD detection for both full-scale (ImageNet-1k) and reduced-scale (ImageNet-100) ID datasets, and with both transformer (ViT-B/16) and convolutional (ResNet-50) CLIP backbones.

Figure 5: GP-Adapter’s OOD detection performance becomes consistently superior to MCM as the number of shots per class increases into the 8–16 range.

These results are robust to hyperparameter selection (upper bound $\tau$ on log marginal likelihood and modality mixture $\alpha$), as well as the backbone architecture and dataset scale. The variance-aware MSP calibration is particularly effective when combined with prompt-learned adapters, sharply reducing overconfident misprediction on OOD queries.

Practical and Theoretical Implications

Practical, GP-Adapter offers a lightweight, plug-and-play uncertainty estimation and OOD detection mechanism for CLIP-based systems, requiring neither OOD training data nor gradient-based adaptation. This design is attractive for deployment in settings with tight data or resource constraints, enabling more reliable, uncertainty-aware predictions in open-world applications.

Theoretically, the work demonstrates the compatibility and complementarity between large-scale multimodal embedding models and Bayesian nonparametrics. It validates the intuition that GP predictive variance—appropriately conditioned and combined across modalities—exposes critical uncertainty structure not directly available in CLIP’s raw logits. This paves the way for further research integrating probabilistic inference with large foundation models for reliability, interpretability, and trustworthiness in AI.

Future Directions may include:

• Extension to larger support sets via scalable or sparse GP approximations.
• Application to domains beyond classification, such as retrieval and structured prediction.
• Exploration of joint multimodal GP modeling or transductive OOD detection.
• Adaptation to online or streaming data regimes.

Conclusion

GP-Adapter advances few-shot OOD detection by grafting class-wise Gaussian Process heads onto frozen CLIP representations, yielding closed-form predictive means and variances for principled, variance-calibrated OOD confidence. The approach remains gradient-free, fast, and memory-efficient, and consistently enhances both in-distribution and OOD performance, especially when combined with prompt learning. This establishes a new baseline for modular, Bayesian uncertainty modeling on top of large vision-LLMs and points to promising future directions for integrating probabilistic and foundation model paradigms in robust, trustworthy AI systems.