Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
126 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Unknown Domain Inconsistency Minimization for Domain Generalization (2403.07329v1)

Published 12 Mar 2024 in cs.LG

Abstract: The objective of domain generalization (DG) is to enhance the transferability of the model learned from a source domain to unobserved domains. To prevent overfitting to a specific domain, Sharpness-Aware Minimization (SAM) reduces source domain's loss sharpness. Although SAM variants have delivered significant improvements in DG, we highlight that there's still potential for improvement in generalizing to unknown domains through the exploration on data space. This paper introduces an objective rooted in both parameter and data perturbed regions for domain generalization, coined Unknown Domain Inconsistency Minimization (UDIM). UDIM reduces the loss landscape inconsistency between source domain and unknown domains. As unknown domains are inaccessible, these domains are empirically crafted by perturbing instances from the source domain dataset. In particular, by aligning the loss landscape acquired in the source domain to the loss landscape of perturbed domains, we expect to achieve generalization grounded on these flat minima for the unknown domains. Theoretically, we validate that merging SAM optimization with the UDIM objective establishes an upper bound for the true objective of the DG task. In an empirical aspect, UDIM consistently outperforms SAM variants across multiple DG benchmark datasets. Notably, UDIM shows statistically significant improvements in scenarios with more restrictive domain information, underscoring UDIM's generalization capability in unseen domains. Our code is available at \url{https://github.com/SJShin-AI/UDIM}.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (60)
  1. Invariant risk minimization. arXiv preprint arXiv:1907.02893, 2019.
  2. Domain generalization by marginal transfer learning. The Journal of Machine Learning Research, 22(1):46–100, 2021.
  3. Improved test-time adaptation for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  24172–24182, 2023a.
  4. Domain generalization via rationale invariance. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  1751–1760, 2023b.
  5. Smoothness and stability in gans. In International Conference on Learning Representations, 2019.
  6. Gabriela Csurka. Domain adaptation for visual applications: A comprehensive survey. arXiv preprint arXiv:1702.05374, 2017.
  7. BackPACK: Packing more into backprop. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=BJlrF24twB.
  8. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.  248–255. Ieee, 2009.
  9. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint arXiv:1703.11008, 2017.
  10. Dan Feldman. Introduction to core-sets: an updated survey. arXiv preprint arXiv:2011.09384, 2020.
  11. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations, 2020.
  12. Domain-adversarial training of neural networks. The journal of machine learning research, 17(1):2096–2030, 2016.
  13. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
  14. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  770–778, 2016.
  15. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261, 2019.
  16. Self-challenging improves cross-domain generalization. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pp.  124–140. Springer, 2020.
  17. Unknown-aware domain adversarial learning for open-set domain adaptation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=IwC_x50fvU.
  18. Fisher sam: Information geometry and sharpness aware minimisation. In International Conference on Machine Learning, pp.  11148–11161. PMLR, 2022.
  19. SAAL: Sharpness-aware active learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp.  16424–16440. PMLR, 23–29 Jul 2023. URL https://proceedings.mlr.press/v202/kim23c.html.
  20. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  21. Learning multiple layers of features from tiny images. 2009.
  22. Out-of-distribution generalization via risk extrapolation (rex). In International Conference on Machine Learning, pp.  5815–5826. PMLR, 2021.
  23. Asam: Adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks. In International Conference on Machine Learning, pp.  5905–5914. PMLR, 2021.
  24. Adaptive estimation of a quadratic functional by model selection. Annals of statistics, pp.  1302–1338, 2000.
  25. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pp.  5542–5550, 2017.
  26. Learning to generalize: Meta-learning for domain generalization. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018a.
  27. Domain generalization with adversarial feature learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  5400–5409, 2018b.
  28. Progressive domain expansion network for single domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  224–233, 2021a.
  29. A simple feature augmentation for domain generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  8886–8895, 2021b.
  30. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European conference on computer vision (ECCV), pp.  624–639, 2018c.
  31. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017.
  32. David A McAllester. Pac-bayesian model averaging. In Proceedings of the twelfth annual conference on Computational learning theory, pp.  164–170, 1999.
  33. Reducing domain gap by reducing style bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  8690–8699, 2021.
  34. Learning explanations that are hard to vary. In International Conference on Learning Representations, 2020a.
  35. Learning explanations that are hard to vary. arXiv preprint arXiv:2009.00329, 2020b.
  36. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE International Conference on Computer Vision, pp.  1406–1415, 2019.
  37. Learning to learn single domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  12556–12565, 2020a.
  38. Learning to learn single domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  12556–12565, 2020b.
  39. Fishr: Invariant gradient variances for out-of-distribution generalization. In International Conference on Machine Learning, pp.  18347–18377. PMLR, 2022.
  40. A closer look at smoothness in domain adversarial training. In International Conference on Machine Learning, pp.  18378–18399. PMLR, 2022.
  41. Distributionally robust neural networks. In International Conference on Learning Representations, 2019.
  42. Nicol N Schraudolph. Fast curvature matrix-vector products for second-order gradient descent. Neural computation, 14(7):1723–1738, 2002.
  43. Loss-curvature matching for dataset selection and condensation. In International Conference on Artificial Intelligence and Statistics, pp.  8606–8628. PMLR, 2023.
  44. On the benefits of representation regularization in invariance based domain generalization. Machine Learning, 111(3):895–915, 2022.
  45. Deep coral: Correlation alignment for deep domain adaptation. In Computer Vision–ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III 14, pp.  443–450. Springer, 2016.
  46. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  5018–5027, 2017.
  47. On calibration and out-of-domain generalization. Advances in neural information processing systems, 34:2215–2227, 2021.
  48. Generalizing to unseen domains: A survey on domain generalization. IEEE Transactions on Knowledge and Data Engineering, 2022.
  49. Sharpness-aware gradient matching for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  3769–3778, 2023.
  50. Learning to diversify for single domain generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  834–843, 2021a.
  51. Learning to diversify for single domain generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  834–843, 2021b.
  52. Improve unsupervised domain adaptation with mixup training. arXiv preprint arXiv:2001.00677, 2020.
  53. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pp.  6023–6032, 2019.
  54. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
  55. Adaptive risk minimization: Learning to adapt to domain shift. Advances in Neural Information Processing Systems, 34:23664–23678, 2021.
  56. Flatness-aware minimization for domain generalization. arXiv preprint arXiv:2307.11108, 2023a.
  57. Gradient norm aware minimization seeks first-order flatness and improves generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  20247–20257, 2023b.
  58. Adversarial style augmentation for domain generalized urban-scene segmentation. Advances in Neural Information Processing Systems, 35:338–350, 2022.
  59. Domain generalization with mixstyle. arXiv preprint arXiv:2104.02008, 2021.
  60. Domain generalization: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets