Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Preserving Silent Features for Domain Generalization (2401.03170v1)

Published 6 Jan 2024 in cs.LG and cs.CV

Abstract: Domain generalization (DG) aims to improve the generalization ability of the model trained on several known training domains over unseen test domains. Previous work has shown that self-supervised contrastive pre-training improves the robustness of the model on downstream tasks. However, in this paper, we find that self-supervised models do not exhibit better generalization performance than supervised models pre-trained on the same dataset in the DG setting. We argue that this is owing to the fact that the richer intra-class discriminative features extracted by self-supervised contrastive learning, which we term silent features, are suppressed during supervised fine-tuning. These silent features are likely to contain features that are more generalizable on the test domain. In this work, we model and analyze this feature suppression phenomenon and theoretically prove that preserving silent features can achieve lower expected test domain risk under certain conditions. In light of this, we propose a simple yet effective method termed STEP (Silent Feature Preservation) to improve the generalization performance of the self-supervised contrastive learning pre-trained model by alleviating the suppression of silent features during the supervised fine-tuning process. Experimental results show that STEP exhibits state-of-the-art performance on standard DG benchmarks with significant distribution shifts.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (73)
  1. Invariant risk minimization. arXiv preprint arXiv:1907.02893, 2019.
  2. Ensemble of averages: Improving model selection and boosting performance in domain generalization. In NeurIPS, 2022.
  3. Metareg: Towards domain generalization using meta-regularization. In NeurIPS, 2018.
  4. Recognition in terra incognita. In ECCV, 2018.
  5. Domain generalization by marginal transfer learning. Journal of Machine Learning Research, 22(1):46–100, 2021.
  6. Exploiting domain-specific features to enhance domain generalization. In NeurIPS, pages 21189–21201, 2021.
  7. Domain generalization by solving jigsaw puzzles. In CVPR, 2019.
  8. Unsupervised learning of visual features by contrasting cluster assignments. In NeurIPS, pages 9912–9924, 2020.
  9. Swad: Domain generalization by seeking flat minima. In NeurIPS, pages 22405–22418, 2021.
  10. Domain generalization by mutual-information regularization with pre-trained models. In ECCV, pages 440–457, 2022.
  11. Learning to balance specificity and invariance for in and out of domain generalization. In ECCV, pages 301–318, 2020.
  12. A simple framework for contrastive learning of visual representations. In ICML, pages 1597–1607, 2020.
  13. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020.
  14. Exploring simple siamese representation learning. In CVPR, pages 15750–15758, 2021.
  15. Iterative feature matching: Toward provable domain generalization with logarithmic environments. In NeurIPS, 2022.
  16. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248–255, 2009.
  17. Deep domain generalization with structured low-rank constraint. IEEE TIP, pages 304–313, 2018.
  18. Adaptive methods for real-world domain generalization. In CVPR, pages 14340–14349, 2021.
  19. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. In ICCV, 2013.
  20. Domain-adversarial training of neural networks. Journal of Machine Learning Research, 17(59):1–35, 2016.
  21. Bootstrap your own latent - a new approach to self-supervised learning. In NeurIPS, pages 21271–21284, 2020.
  22. In search of lost domain generalization. In ICLR, 2021.
  23. Provable guarantees for self-supervised deep learning with spectral contrastive loss. In NeurIPS, pages 5000–5011, 2021.
  24. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
  25. Deep residual learning for image recognition. In CVPR, 2016.
  26. The many faces of robustness: A critical analysis of out-of-distribution generalization. In ICCV, pages 8340–8349, 2021.
  27. Using self-supervised learning can improve model robustness and uncertainty. In NeurIPS, 2019.
  28. Flat minima. Neural computation, 9(1):1–42, 1997.
  29. Self-challenging improves cross-domain generalization. In ECCV, pages 124–140, 2020.
  30. Test-time classifier adjustment module for model-agnostic domain generalization. In NeurIPS, pages 2427–2440, 2021.
  31. On large-batch training for deep learning: Generalization gap and sharp minima. In ICLR, 2017.
  32. Selfreg: Self-supervised contrastive regularization for domain generalization. In ICCV, pages 9619–9628, 2021.
  33. Adam: A method for stochastic optimization. In ICLR, 2015.
  34. Out-of-distribution generalization via risk extrapolation (rex). In ICML, pages 5815–5826, 2021.
  35. Fine-tuning can distort pretrained features and underperform out-of-distribution. In ICLR, 2022.
  36. Learning to generalize: Meta-learning for domain generalization. In AAAI, 2018.
  37. Deeper, broader and artier domain generalization. In ICCV, 2017.
  38. Domain generalization with adversarial feature learning. In CVPR, pages 5400–5409, 2018.
  39. Domain generalization via conditional invariant representations. In AAAI, volume 32, 2018.
  40. Deep domain generalization via conditional invariant adversarial networks. In ECCV, pages 624–639, 2018.
  41. Causality inspired representation learning for domain generalization. In CVPR, pages 8046–8056, 2022.
  42. Domain generalization using causal matching. In ICML, pages 7313–7324, 2021.
  43. Domain generalization via gradient surgery. In ICCV, pages 6630–6638, 2021.
  44. Foundations of machine learning. MIT press, 2018.
  45. Domain generalization via invariant feature representation. In ICML, pages 10–18, 2013.
  46. Reducing domain gap by reducing style bias. In CVPR, pages 8690–8699, 2021.
  47. Moment matching for multi-source domain adaptation. In ICCV, 2019.
  48. Understanding and mitigating the tradeoff between robustness and accuracy. In ICML, pages 7909–7919, 2020.
  49. The risks of invariant risk minimization. In ICLR, 2021.
  50. ImageNet Large Scale Visual Recognition Challenge. IJCV, 115(3):211–252, 2015.
  51. Distributionally robust neural networks. In ICLR, 2020.
  52. Deep coral: Correlation alignment for deep domain adaptation. In ECCV, pages 443–450, 2016.
  53. Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 1999.
  54. Deep hashing network for unsupervised domain adaptation. In CVPR, 2017.
  55. Provable domain generalization via invariant-feature subspace recovery. In ICML, pages 23018–23033, 2022.
  56. Continual test-time domain adaptation. In CVPR, pages 7201–7211, 2022.
  57. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In ICML, pages 9929–9939, 2020.
  58. Heterogeneous domain generalization via domain mixup. In ICASSP, pages 3622–3626, 2020.
  59. Revisiting locally supervised learning: an alternative to end-to-end training. In ICLR, 2021.
  60. Vector-decomposed disentanglement for domain-invariant object detection. In ICCV, pages 9342–9351, 2021.
  61. Unsupervised feature learning via non-parametric instance discrimination. In CVPR, 2018.
  62. In-n-out: Pre-training and self-training using auxiliary information for out-of-distribution robustness. In ICLR, 2021.
  63. A fourier-based framework for domain generalization. In CVPR, pages 14383–14392, 2021.
  64. Improve unsupervised domain adaptation with mixup training. arXiv preprint arXiv:2001.00677, 2020.
  65. Pcl: Proxy-based contrastive learning for domain generalization. In CVPR, pages 7097–7107, 2022.
  66. Ood-bench: Quantifying and understanding two dimensions of out-of-distribution generalization. In CVPR, pages 7947–7958, 2022.
  67. Barlow twins: Self-supervised learning via redundancy reduction. In ICML, pages 12310–12320, 2021.
  68. mixup: Beyond empirical risk minimization. In ICLR, 2018.
  69. Adaptive risk minimization: Learning to adapt to domain shift. In NeurIPS, pages 23664–23678, 2021.
  70. Deep domain-adversarial image generation for domain generalisation. In AAAI, pages 13025–13032, 2020.
  71. Learning to generate novel domains for domain generalization. In ECCV, pages 561–578, 2020.
  72. Domain adaptive ensemble learning. IEEE TIP, 30:8008–8018, 2021.
  73. Domain generalization with mixstyle. In ICLR, 2021.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now