Unraveling Batch Normalization for Realistic Test-Time Adaptation (2312.09486v3)
Abstract: While recent test-time adaptations exhibit efficacy by adjusting batch normalization to narrow domain disparities, their effectiveness diminishes with realistic mini-batches due to inaccurate target estimation. As previous attempts merely introduce source statistics to mitigate this issue, the fundamental problem of inaccurate target estimation still persists, leaving the intrinsic test-time domain shifts unresolved. This paper delves into the problem of mini-batch degradation. By unraveling batch normalization, we discover that the inexact target statistics largely stem from the substantially reduced class diversity in batch. Drawing upon this insight, we introduce a straightforward tool, Test-time Exponential Moving Average (TEMA), to bridge the class diversity gap between training and testing batches. Importantly, our TEMA adaptively extends the scope of typical methods beyond the current batch to incorporate a diverse set of class information, which in turn boosts an accurate target estimation. Built upon this foundation, we further design a novel layer-wise rectification strategy to consistently promote test-time performance. Our proposed method enjoys a unique advantage as it requires neither training nor tuning parameters, offering a truly hassle-free solution. It significantly enhances model robustness against shifted domains and maintains resilience in diverse real-world scenarios with various batch sizes, achieving state-of-the-art performance on several major benchmarks. Code is available at \url{https://github.com/kiwi12138/RealisticTTA}.
- Parameter-free online test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8344–8353.
- Learning to transfer examples for partial domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2985–2994.
- Contrastive test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 295–305.
- Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4): 834–848.
- RobustBench: a standardized adversarial robustness benchmark. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
- Robust mean teacher for continual and gradual test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7704–7714.
- Test-time training with masked autoencoders. Advances in Neural Information Processing Systems, 35: 29374–29385.
- Unsupervised domain adaptation by backpropagation. In International Conference on Machine Learning, 1180–1189. PMLR.
- Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 770–778.
- Benchmarking neural network robustness to common corruptions and surface variations. arXiv preprint arXiv:1807.01697.
- AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty. In International Conference on Learning Representations.
- Cycada: Cycle-consistent adversarial domain adaptation. In International Conference on Machine Learning, 1989–1998. Pmlr.
- Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, 448–456. PMLR.
- Test-time classifier adjustment module for model-agnostic domain generalization. Advances in Neural Information Processing Systems, 34: 2427–2440.
- Sita: Single image test-time adaptation. arXiv preprint arXiv:2112.02355.
- ImageNet classification with deep convolutional neural networks. Communications of the ACM, 60(6): 84–90.
- Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning, 6028–6039. PMLR.
- TTN: A Domain-Shift Aware Batch Normalization in Test-Time Adaptation. In International Conference on Learning Representations.
- TTT++: When does self-supervised test-time training fail or thrive? Advances in Neural Information Processing Systems, 34: 21808–21820.
- Transfer feature learning with joint distribution adaptation. In Proceedings of the IEEE International Conference on Computer Vision, 2200–2207.
- The norm must go on: Dynamic unsupervised domain adaptation by normalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 14765–14775.
- Evaluating prediction-time batch normalization for robustness under covariate shift. arXiv preprint arXiv:2006.10963.
- Efficient test-time model adaptation without forgetting. In International Conference on Machine Learning, 16888–16905. PMLR.
- Towards Stable Test-time Adaptation in Dynamic Wild World. In International Conference on Learning Representations.
- Improving robustness against common corruptions by covariate shift adaptation. Advances in Neural Information Processing Systems, 33: 11539–11551.
- Test-time training with self-supervision for generalization under distribution shifts. In International Conference on Machine Learning, 9229–9248. PMLR.
- Tent: Fully Test-Time Adaptation by Entropy Minimization. In International Conference on Learning Representations.
- Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7201–7211.
- Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1492–1500.
- Test-time batch statistics calibration for covariate shift. arXiv preprint arXiv:2110.04065.
- Universal domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2720–2729.
- Robust test-time adaptation in dynamic scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 15922–15932.
- Wide residual networks. arXiv preprint arXiv:1605.07146.
- Memo: Test time robustness via adaptation and augmentation. Advances in Neural Information Processing Systems, 35: 38629–38642.
- DELTA: Degradation-free Fully Test-time Adaptation. In International Conference on Learning Representations.