Dual Distribution Estimation for Zero-shot Noisy Test-Time Adaptation with VLMs

Abstract: While test-time adaptation (TTA) empowers vision-LLMs to adapt without costly retraining, it remains highly vulnerable to out-of-distribution (OOD) outliers prevalent in real-world applications. This discrepancy motivates Noisy TTA (NTTA), an online task to filter noisy OOD samples on the fly while maximizing in-distribution (ID) classification accuracy. Existing zero-shot NTTA approaches typically rely on test-time discriminative training, leading to overconfident misclassifications and significantly degraded inference efficiency. To address these limitations, we propose a novel framework named Dual Distribution Estimation (DDE), shifting the zero-shot NTTA paradigm from instance-level learning to training-free Gaussian distribution modeling. DDE incorporates two novel modules: Positive Feature Distribution Estimation (PFDE) and Negative Label Distribution Estimation (NLDE). PFDE explicitly models class-wise inclusion and exclusion Gaussian distributions to formulate a calibrated contrastive score, robustly enhancing ID accuracy. In parallel, NLDE improves OOD identification by explicitly modeling the negative label distribution to mine highly discriminative labels, effectively mitigating spurious correlations. Extensive experiments show that on the large-scale ImageNet benchmark, DDE achieves an improvement of 3.70\% in harmonic mean accuracy and reduces the FPR95 for OOD detection by 6.20\%, while ensuring highly scalable and efficient online inference. Furthermore, DDE is zero-shot and training-free, demonstrating remarkable robustness in data-scarce scenarios. Codes are available at https://github.com/ZhuWenjie98/DDE.

• The paper introduces dual Gaussian modeling via PFDE and NLDE to refine classification boundaries and filter out noise without retraining.
• It achieves significant gains, including a 3.70% increase in harmonic mean accuracy and reduced FPR95 on ImageNet, outperforming existing methods.
• Its training-free, adaptive pipeline robustly handles online classification and out-of-distribution detection across diverse, noisy datasets.

Dual Distribution Estimation for Zero-shot Noisy Test-Time Adaptation with VLMs

Problem Formulation and Motivation

Zero-shot Noisy Test-Time Adaptation (NTTA) operates in strict online settings, where vision-LLMs (VLMs) must both classify in-distribution (ID) samples and filter out noisy out-of-distribution (OOD) outliers without access to retraining or supervision. Traditional TTA methods either assume closed-world operation or rely on costly online discriminative training, which impedes deployment efficiency and exposes the model to performance collapse in noisy, data-scarce environments.

Recent approaches, such as AdaND, fail to capture the true distribution of test image features, depend heavily on CLIP’s text-image alignment, and require extensive training samples for reliable noise detection. These deficiencies motivate a paradigm shift: from instance-level discriminative learning to distribution-level modeling that leverages historical test data statistics in a training-free, zero-shot manner.

Figure 1: Visualized t-SNE comparison exposing CLIP classifier's misalignment with textual prototypes, and the more precise class distribution captured by DDE through dual Gaussian modeling.

Dual Distribution Estimation Framework

The proposed Dual Distribution Estimation (DDE) framework decomposes the NTTA problem into two complementary modules:

1. Positive Feature Distribution Estimation (PFDE): Explicitly models class-wise inclusion and exclusion Gaussian distributions using positive image features. PFDE refines test-time classification boundaries by calibrating the contrastive score—the difference between inclusion and exclusion probabilities—via Gaussian Discriminant Analysis (GDA) and Bayesian posterior inference.
2. Negative Label Distribution Estimation (NLDE): Enhances OOD detection by mining discriminative negative labels from a large corpus, quantifying their similarity discrepancy between positive and negative sample caches, and filtering labels with spurious correlations.

The DDE pipeline dynamically caches positive/negative samples, estimates dual Gaussian statistics, mines highly discriminative negative labels, and fuses GDA and CLIP classifiers under an adaptive thresholding regime, achieving robust ID classification and accurate OOD noise filtration even under severe distribution shifts.

Figure 2: Schematic of the DDE workflow: Positive/negative sample caching, dual Gaussian modeling, discriminative negative label estimation, and adaptive threshold-based classification / OOD detection.

Methodological Details

PFDE utilizes class-conditional GDA to maintain dual Gaussian parameters (means and covariances) for each ID class via test-stream EM updates. The model distinguishes inclusion samples (top-1 prediction as class $k$) and exclusion samples (top-2 prediction as class $k$, but not top-1) to capture ambiguous boundary cases. Logit fusion of CLIP and GDA classifier outputs is adaptively weighted by data accumulation, ensuring calibrated inference across varying sample availability.

NLDE scores negative labels for OOD discriminability based on their mean similarity gap between negative and positive sample caches, selecting the top-$\hat{M}$ negatives to enhance detection. Final OOD scores are computed with this refined negative label pool, and adaptive thresholding dynamically separates ID and OOD samples by minimizing intra-class variance of OOD scores.

Empirical Results

DDE demonstrates substantial gains over existing NTTA and OOD detection baselines across diverse datasets:

• On ImageNet, DDE achieves a 3.70% increase in $Acc_H$ harmonic mean accuracy and reduces FPR95 for OOD detection by 6.20% relative to AdaND, maintaining high $Acc_S$ and $Acc_N$ across all OOD evaluation sets.
• On challenging fine-grained datasets (CUB-200-2011, Stanford Cars, Food-101, Oxford-IIIT Pet), DDE consistently outperforms the strongest baselines, with significant improvements in both clean and noisy sample detection.
• For Near- and Far-OOD detection benchmarks (SSB-Hard, NINCO, iNaturalist, Textures, OpenImage-O), DDE provides absolute AUROC improvements up to +7.82% and lowest FPR95 values, indicating robust semantic separation via discriminative negative mining.

Figure 3: Model robustness to positive/negative cache noise ratio; harmonic mean accuracy remains stable as the noise rate increases, validating effective online noise filtration.

DDE provides superior scalability and efficiency: it is fully training-free, introduces negligible memory overhead (default $Q=1000$), and achieves the fastest inference time (1.84 min) among competitive TTA solutions, with no learnable parameters and low GPU memory consumption. Detailed ablation confirms the synergistic effect of PFDE’s exclusion distribution and NLDE’s discriminative negative mining for maximal robustness, especially in data-scarce and highly noisy settings.

Hyperparameter Stability and Practical Considerations

Comprehensive analysis shows DDE’s insensitivity to prompt templates, exclusion $k$ strategies, positive/negative selection thresholds, cache size $Q$, scaling factor $\rho$, exclusion weight $k$0, and discriminative negative label count $k$1. DDE consistently delivers high accuracy across backbone architectures and adaptation stages, with the adaptive thresholding mechanism avoiding brittle manual hyperparameter tuning.

Figure 4: Comparative effect of prompt template selection on final scores, confirming DDE’s robustness to prompt engineering.

Theoretical and Practical Implications

The DDE paradigm reconceptualizes test-time adaptation for VLMs by introducing explicit distribution-level modeling, achieving calibrated classification and noise detection without retraining. It demonstrates that class-wise statistical modeling and intelligent negative label mining can fundamentally improve adaptability and robustness of zero-shot models under realistic distribution shift and label noise.

This approach paves the way for future NTTA solutions with even richer statistical modeling, scalable distribution estimation on larger and more diverse datasets, and fine-grained semantic control for multimodal models. DDE’s methodological innovations are directly applicable to deployment scenarios in open-world vision, medical imaging, and industrial monitoring where noisy or corrupted test data are ubiquitous.

Conclusion

DDE establishes a new standard for zero-shot NTTA by leveraging dual distribution estimation—class-wise positive feature modeling via inclusion/exclusion Gaussian distributions, and discriminative negative label mining—under a fully training-free, efficient online pipeline. It achieves strong empirical gains in both ID classification and OOD detection, demonstrating exceptional robustness to distribution shifts and sample noise, with broad applicability and scalability for real-world VLM deployment scenarios (2606.25758).