Unified Entropy Optimization for Open-Set Test-Time Adaptation (2404.06065v1)
Abstract: Test-time adaptation (TTA) aims at adapting a model pre-trained on the labeled source domain to the unlabeled target domain. Existing methods usually focus on improving TTA performance under covariate shifts, while neglecting semantic shifts. In this paper, we delve into a realistic open-set TTA setting where the target domain may contain samples from unknown classes. Many state-of-the-art closed-set TTA methods perform poorly when applied to open-set scenarios, which can be attributed to the inaccurate estimation of data distribution and model confidence. To address these issues, we propose a simple but effective framework called unified entropy optimization (UniEnt), which is capable of simultaneously adapting to covariate-shifted in-distribution (csID) data and detecting covariate-shifted out-of-distribution (csOOD) data. Specifically, UniEnt first mines pseudo-csID and pseudo-csOOD samples from test data, followed by entropy minimization on the pseudo-csID data and entropy maximization on the pseudo-csOOD data. Furthermore, we introduce UniEnt+ to alleviate the noise caused by hard data partition leveraging sample-level confidence. Extensive experiments on CIFAR benchmarks and Tiny-ImageNet-C show the superiority of our framework. The code is available at https://github.com/gaozhengqing/UniEnt
- Feed two birds with one scone: Exploiting wild data for both out-of-distribution generalization and detection. In ICML, 2023.
- Parameter-free online test-time adaptation. In CVPR, 2022.
- Contrastive test-time adaptation. In CVPR, 2022.
- Learning open set network with discriminative reciprocal points. In ECCV, 2020.
- Adversarial reciprocal points learning for open set recognition. IEEE TPAMI, 2021.
- Improving test-time adaptation via shift-agnostic weight regularization and nearest source prototypes. In ECCV, 2022.
- Robustbench: a standardized adversarial robustness benchmark. In NeurIPS Datasets and Benchmarks Track, 2021.
- Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
- Reducing network agnostophobia. In NeurIPS, 2018.
- An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
- Domain-adversarial training of neural networks. JMLR, 2016.
- Note: Robust continual test-time adaptation against temporal correlation. In NeurIPS, 2022.
- Deep residual learning for image recognition. In CVPR, 2016.
- Benchmarking neural network robustness to common corruptions and perturbations. In ICLR, 2019.
- A baseline for detecting misclassified and out-of-distribution examples in neural networks. In ICLR, 2017.
- Deep anomaly detection with outlier exposure. In ICLR, 2019.
- Augmix: A simple data processing method to improve robustness and uncertainty. In ICLR, 2020.
- Natural adversarial examples. In CVPR, 2021.
- Scaling out-of-distribution detection for real-world settings. In ICML, 2022.
- Generalized odin: Detecting out-of-distribution image without learning from out-of-distribution data. In CVPR, 2020.
- Class-specific semantic reconstruction for open set recognition. IEEE TPAMI, 2022.
- Test-time classifier adjustment module for model-agnostic domain generalization. In NeurIPS, 2021.
- Training ood detectors in their natural habitats. In ICML, 2022.
- Sita: Single image test-time adaptation. arXiv preprint arXiv:2112.02355, 2021.
- Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
- Tiny imagenet visual recognition challenge. 2015.
- Towards open-set test-time adaptation utilizing the wisdom of crowds in entropy minimization. In ICCV, 2023.
- On the robustness of open-world test-time training: Self-training with dynamic prototype expansion. In ICCV, 2023.
- Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In ICML, 2020.
- Enhancing the reliability of out-of-distribution image detection in neural networks. In ICLR, 2018.
- Ttn: A domain-shift aware batch normalization in test-time adaptation. In ICLR, 2023.
- Energy-based out-of-distribution detection. In NeurIPS, 2020.
- Evaluating prediction-time batch normalization for robustness under covariate shift. arXiv preprint arXiv:2006.10963, 2020.
- Reading digits in natural images with unsupervised feature learning. 2011.
- Efficient test-time model adaptation without forgetting. In ICML, 2022.
- Towards stable test-time adaptation in dynamic wild world. In ICLR, 2023.
- Rdumb: A simple approach that questions our progress in continual test-time adaptation. arXiv:2306.05401, 2023.
- Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747, 2016.
- Improving robustness against common corruptions by covariate shift adaptation. In NeurIPS, 2020.
- Deeper insights into vits robustness towards common corruptions. arXiv:2204.12143, 2022.
- Training data-efficient image transformers & distillation through attention. In ICML, 2021.
- Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 2008.
- Open-set recognition: A good closed-set classifier is all you need. In ICLR, 2022.
- Tent: Fully test-time adaptation by entropy minimization. In ICLR, 2021.
- Generalizing to unseen domains: A survey on domain generalization. TKDE, 2022a.
- Continual test-time domain adaptation. In CVPR, 2022b.
- A fourier-based framework for domain generalization. In CVPR, 2021.
- Semantically coherent out-of-distribution detection. In ICCV, 2021.
- Auto: Adaptive outlier optimization for online test-time ood detection. arXiv preprint arXiv:2303.12267, 2023.
- Fda: Fourier domain adaptation for semantic segmentation. In CVPR, 2020.
- Test-time batch statistics calibration for covariate shift. arXiv preprint arXiv:2110.04065, 2021.
- Unsupervised out-of-distribution detection by maximum classifier discrepancy. In ICCV, 2019.
- Robust test-time adaptation in dynamic scenarios. In CVPR, 2023.
- Wide residual networks. In BMVC, 2016.
- Mixture outlier exposure: Towards out-of-distribution detection in fine-grained environments. In WACV, 2023a.
- Openood v1. 5: Enhanced benchmark for out-of-distribution detection. arXiv preprint arXiv:2306.09301, 2023b.
- Memo: Test time robustness via adaptation and augmentation. In NeurIPS, 2022.
- Domain generalization with mixstyle. In ICLR, 2021.
- Domain generalization: A survey. IEEE TPAMI, 2022.
- Ods: Test-time adaptation in the presence of open-world data shift. In ICML, 2023.