Papers
Topics
Authors
Recent
Assistant
AI Research Assistant
Well-researched responses based on relevant abstracts and paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses.
Gemini 2.5 Flash
Gemini 2.5 Flash 119 tok/s
Gemini 2.5 Pro 51 tok/s Pro
GPT-5 Medium 27 tok/s Pro
GPT-5 High 17 tok/s Pro
GPT-4o 60 tok/s Pro
Kimi K2 196 tok/s Pro
GPT OSS 120B 423 tok/s Pro
Claude Sonnet 4.5 37 tok/s Pro
2000 character limit reached

Entropy is not Enough for Test-Time Adaptation: From the Perspective of Disentangled Factors (2403.07366v1)

Published 12 Mar 2024 in cs.CV and cs.LG

Abstract: Test-time adaptation (TTA) fine-tunes pre-trained deep neural networks for unseen test data. The primary challenge of TTA is limited access to the entire test dataset during online updates, causing error accumulation. To mitigate it, TTA methods have utilized the model output's entropy as a confidence metric that aims to determine which samples have a lower likelihood of causing error. Through experimental studies, however, we observed the unreliability of entropy as a confidence metric for TTA under biased scenarios and theoretically revealed that it stems from the neglect of the influence of latent disentangled factors of data on predictions. Building upon these findings, we introduce a novel TTA method named Destroy Your Object (DeYO), which leverages a newly proposed confidence metric named Pseudo-Label Probability Difference (PLPD). PLPD quantifies the influence of the shape of an object on prediction by measuring the difference between predictions before and after applying an object-destructive transformation. DeYO consists of sample selection and sample weighting, which employ entropy and PLPD concurrently. For robust adaptation, DeYO prioritizes samples that dominantly incorporate shape information when making predictions. Our extensive experiments demonstrate the consistent superiority of DeYO over baseline methods across various scenarios, including biased and wild. Project page is publicly available at https://whitesnowdrop.github.io/DeYO/.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (71)
  1. Pseudo-labeling and confirmation bias in deep semi-supervised learning. In 2020 International Joint Conference on Neural Networks (IJCNN), pp.  1–8. IEEE, 2020.
  2. Invariant risk minimization. arXiv preprint arXiv:1907.02893, 2019.
  3. Self-supervised test-time adaptation on video data. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp.  3439–3448, 2022.
  4. Natural and adversarial error detection using invariance to image transformations. arXiv preprint arXiv:1902.00236, 2019.
  5. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems, 32, 2019.
  6. MT3: meta test-time training for self-supervised test-time adaption. In International Conference on Artificial Intelligence and Statistics, pp.  3080–3090, 2022.
  7. Visda-2021 competition: Universal domain adaptation to improve performance on out-of-distribution data. In NeurIPS 2021 Competitions and Demonstrations Track, pp.  66–79, 2022.
  8. Recognition in terra incognita. In Proceedings of the European conference on computer vision (ECCV), pp.  456–473, 2018.
  9. Mixmatch: A holistic approach to semi-supervised learning. Advances in neural information processing systems, 32, 2019.
  10. Generalizing from several related classification tasks to a new unlabeled sample. Advances in neural information processing systems, 24, 2011.
  11. Isolating sources of disentanglement in variational autoencoders. Advances in neural information processing systems, 31, 2018.
  12. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp.  1597–1607. PMLR, 2020a.
  13. Self-training avoids using spurious features under domain shift. Advances in Neural Information Processing Systems, 33:21061–21071, 2020b.
  14. Improving test-time adaptation via shift-agnostic weight regularization and nearest source prototypes. In European Conference on Computer Vision, pp.  440–458. Springer, 2022.
  15. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp.  702–703, 2020.
  16. Revisiting the evaluation of uncertainty estimation and its application to explore model complexity-uncertainty trade-off. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp.  4–5, 2020.
  17. On the transfer of disentangled representations in realistic settings. In International Conference on Learning Representations, 2021.
  18. Unsupervised domain adaptation by backpropagation. In International conference on machine learning, pp.  1180–1189. PMLR, 2015.
  19. Bias-reduced uncertainty estimation for deep neural classifiers. In International Conference on Learning Representations, 2019a.
  20. Bias-reduced uncertainty estimation for deep neural classifiers. In International Conference on Learning Representations, 2019b.
  21. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2019.
  22. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673, 2020.
  23. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations, 2018.
  24. Semi-supervised learning by entropy minimization. Advances in neural information processing systems, 17, 2004.
  25. In search of lost domain generalization. In International Conference on Learning Representations, 2021.
  26. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  770–778, 2016.
  27. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019.
  28. Augmix: A simple data processing method to improve robustness and uncertainty. In International Conference on Learning Representations, 2020.
  29. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  8340–8349, 2021.
  30. beta-vae: Learning basic visual concepts with a constrained variational framework. In International conference on learning representations, 2017.
  31. Generating instance-level prompts for rehearsal-free continual learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  11847–11857, 2023a.
  32. New insights for the stability-plasticity dilemma in online continual learning. In The Eleventh International Conference on Learning Representations, 2023b. URL https://openreview.net/forum?id=fxC7kJYwA_a.
  33. Contrastive adaptation network for unsupervised domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  4893–4902, 2019.
  34. Disentangling by factorising. In International Conference on Machine Learning, pp.  2649–2658. PMLR, 2018.
  35. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
  36. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
  37. Confidence score for source-free unsupervised domain adaptation. In International Conference on Machine Learning, pp.  12365–12377. PMLR, 2022.
  38. Learning debiased representation via disentangled feature augmentation. Advances in Neural Information Processing Systems, 34:25123–25133, 2021.
  39. Revisiting the importance of amplifying bias for debiasing. In Proceedings of the AAAI Conference on Artificial Intelligence, 2023.
  40. Rethinking distributional matching based domain adaptation. arXiv preprint arXiv:2006.13352, 2020.
  41. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International conference on machine learning, pp.  6028–6039. PMLR, 2020.
  42. Ttn: A domain-shift aware batch normalization in test-time adaptation. In The Eleventh International Conference on Learning Representations, 2023.
  43. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pp.  6781–6792. PMLR, 2021a.
  44. Ttt++: When does self-supervised test-time training fail or thrive? Advances in Neural Information Processing Systems, 34:21808–21820, 2021b.
  45. Transfer feature learning with joint distribution adaptation. In Proceedings of the IEEE international conference on computer vision, pp.  2200–2207, 2013.
  46. Efficient test-time model adaptation without forgetting. In International conference on machine learning, pp.  16888–16905. PMLR, 2022.
  47. Towards stable test-time adaptation in dynamic wild world. In Internetional Conference on Learning Representations, 2023.
  48. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10):1345–1359, 2009.
  49. Joint contrastive learning for unsupervised domain adaptation. arXiv preprint arXiv:2006.10297, 2020.
  50. Sentry: Selective entropy optimization via committee consistency for unsupervised domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  8558–8567, 2021.
  51. Distributionally robust neural networks. In International Conference on Learning Representations, 2020.
  52. Universal domain adaptation through self supervision. Advances in neural information processing systems, 33:16282–16292, 2020.
  53. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. Advances in neural information processing systems, 29, 2016.
  54. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pp.  618–626, 2017.
  55. A dirt-t approach to unsupervised domain adaptation. arXiv preprint arXiv:1802.08735, 2018.
  56. Salient imagenet: How to discover spurious features in deep learning? In International Conference on Learning Representations, 2022.
  57. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural information processing systems, 33:596–608, 2020.
  58. Ecotta: Memory-efficient continual test-time adaptation via self-distilled regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  11920–11929, 2023.
  59. Test-time training with self-supervision for generalization under distribution shifts. In International conference on machine learning, pp.  9229–9248. PMLR, 2020.
  60. Class-imbalanced domain adaptation: an empirical odyssey. In Computer Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16, pp.  585–602. Springer, 2020.
  61. The Caltech-UCSD Birds-200-2011 dataset. Technical report, California Institute of Technology, 2011.
  62. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations, 2021a. URL https://openreview.net/forum?id=uXl3bZLkr3c.
  63. Proselflc: Progressive self label correction for training robust deep neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  752–761, 2021b.
  64. Semi-supervised semantic segmentation using unreliable pseudo-labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  4248–4257, 2022.
  65. Ross Wightman. Pytorch image models. https://github.com/rwightman/pytorch-image-models, 2019.
  66. A fine-grained analysis on distribution shift. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=Dl4LetuLdyK.
  67. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  10687–10698, 2020.
  68. Memo: Test time robustness via adaptation and augmentation. Advances in Neural Information Processing Systems, 35:38629–38642, 2022.
  69. On pitfalls of test-time adaptation. In International Conference on Machine Learning (ICML), 2023.
  70. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6):1452–1464, 2017.
  71. Examining and combating spurious features under distribution shift. In International Conference on Machine Learning, pp.  12857–12867. PMLR, 2021.
Citations (17)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
Lightbulb Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up