Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
134 tokens/sec
GPT-4o
10 tokens/sec
Gemini 2.5 Pro Pro
47 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

RDumb: A simple approach that questions our progress in continual test-time adaptation (2306.05401v3)

Published 8 Jun 2023 in cs.LG and cs.CV

Abstract: Test-Time Adaptation (TTA) allows to update pre-trained models to changing data distributions at deployment time. While early work tested these algorithms for individual fixed distribution shifts, recent work proposed and applied methods for continual adaptation over long timescales. To examine the reported progress in the field, we propose the Continually Changing Corruptions (CCC) benchmark to measure asymptotic performance of TTA techniques. We find that eventually all but one state-of-the-art methods collapse and perform worse than a non-adapting model, including models specifically proposed to be robust to performance collapse. In addition, we introduce a simple baseline, "RDumb", that periodically resets the model to its pretrained state. RDumb performs better or on par with the previously proposed state-of-the-art in all considered benchmarks. Our results show that previous TTA approaches are neither effective at regularizing adaptation to avoid collapse nor able to outperform a simplistic resetting strategy.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (54)
  1. Gradual domain adaptation in the wild: When intermediate distributions are absent. arXiv preprint arXiv:2106.06080.
  2. Adapting to continuously shifting domains. Workshop Track - ICLR 2018.
  3. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.
  4. Challenges in task incremental learning for assistive robotics. IEEE Access, 8:3434–3441.
  5. Domain-adversarial training of neural networks. The journal of machine learning research, 17(1):2096–2030.
  6. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv:1811.12231.
  7. Note: Robust continual test-time adaptation against temporal correlation. In Advances in Neural Information Processing Systems.
  8. Test-time adaptation via conjugate pseudo-labels. arXiv preprint arXiv:2207.09640.
  9. Soda10m: A large-scale 2d self/semi-supervised object detection dataset for autonomous driving. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
  10. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770–778. IEEE Computer Society.
  11. The many faces of robustness: A critical analysis of out-of-distribution generalization. ArXiv preprint, abs/2006.16241.
  12. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261.
  13. Benchmarking neural network robustness to common corruptions and perturbations. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
  14. Augmix: A simple data processing method to improve robustness and uncertainty. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
  15. Continuous manifold based adaptation for evolving visual domains. In Computer Vision and Pattern Recognition (CVPR).
  16. 3d common corruptions and data augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18963–18974.
  17. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
  18. Learning multiple layers of features from tiny images.
  19. Imagenet classification with deep convolutional neural networks. In Bartlett, P. L., Pereira, F. C. N., Burges, C. J. C., Bottou, L., and Weinberger, K. Q., editors, Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of a meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States, pages 1106–1114.
  20. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist, 2.
  21. Lee, D.-H. (2013). Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In ICML Workshop : Challenges in Representation Learning (WREPL).
  22. Source data-absent unsupervised domain adaptation through hypothesis transfer and labeling transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence.
  23. Swin transformer v2: Scaling up capacity and resolution. In International Conference on Computer Vision and Pattern Recognition (CVPR).
  24. Open compound domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12406–12415.
  25. Core50: a new dataset and benchmark for continuous object recognition. In Levine, S., Vanhoucke, V., and Goldberg, K., editors, Proceedings of the 1st Annual Conference on Robot Learning, volume 78 of Proceedings of Machine Learning Research, pages 17–26. PMLR.
  26. Exploring the limits of weakly supervised pretraining. In Proceedings of the European Conference on Computer Vision (ECCV).
  27. Few-shot adversarial domain adaptation. Advances in neural information processing systems, 30.
  28. Test-time adaptation to distribution shift by confidence maximization and input transformation. arXiv preprint arXiv:2106.14999.
  29. Evaluating prediction-time batch normalization for robustness under covariate shift. ArXiv preprint, abs/2006.10963.
  30. Efficient test-time model adaptation without forgetting. arXiv preprint arXiv:2204.02610.
  31. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32.
  32. Gdumb: A simple approach that questions our progress in continual learning. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 524–540. Springer.
  33. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748–8763. PMLR.
  34. Ratcliff, R. (1990). Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, 97(2):285.
  35. If your data distribution shifts, use self-learning. Transactions of Machine Learning Research.
  36. Increasing the robustness of dnns against image corruptions by playing the game of noise. ArXiv preprint, abs/2001.06057.
  37. Imagenet large scale visual recognition challenge. International journal of computer vision (IJCV).
  38. Improving robustness against common corruptions by covariate shift adaptation. In Advances in neural information processing systems.
  39. Are we ready for service robots? the OpenLORIS-Scene datasets for lifelong SLAM. In 2020 International Conference on Robotics and Automation (ICRA), pages 3139–3145.
  40. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  41. Unsupervised domain adaptation through self-supervision. ArXiv preprint, abs/1909.11825.
  42. Test-time training for out-of-distribution generalization. ArXiv preprint, abs/1909.13231.
  43. Maxvit: Multi-axis vision transformer. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXIV, pages 459–479. Springer.
  44. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7167–7176.
  45. Three scenarios for continual learning. arXiv preprint arXiv:1904.07734.
  46. Fully test-time adaptation by entropy minimization. ArXiv preprint, abs/2006.10726.
  47. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006.10726.
  48. Continual test-time domain adaptation. arXiv preprint arXiv:2203.13591.
  49. Incremental adversarial domain adaptation for continually changing environments. In 2018 IEEE International conference on robotics and automation (ICRA), pages 4489–4495. IEEE.
  50. Self-training with noisy student improves imagenet classification. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 10684–10695. IEEE.
  51. Aggregated residual transformations for deep neural networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 5987–5995. IEEE Computer Society.
  52. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  53. Prototypical cross-domain self-supervised learning for few-shot unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13834–13844.
  54. Memo: Test time robustness via adaptation and augmentation. Advances in Neural Information Processing Systems, 35:38629–38642.
Citations (22)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - oripress/CCC: Code for Continuously Changing Corruptions (CCC) benchmark + evaluation (30 stars)