Generalizing to Unseen Domains: A Survey on Domain Generalization (2103.03097v7)

Abstract: Machine learning systems generally assume that the training and testing distributions are the same. To this end, a key requirement is to develop models that can generalize to unseen distributions. Domain generalization (DG), i.e., out-of-distribution generalization, has attracted increasing interests in recent years. Domain generalization deals with a challenging setting where one or several different but related domain(s) are given, and the goal is to learn a model that can generalize to an unseen test domain. Great progress has been made in the area of domain generalization for years. This paper presents the first review of recent advances in this area. First, we provide a formal definition of domain generalization and discuss several related fields. We then thoroughly review the theories related to domain generalization and carefully analyze the theory behind generalization. We categorize recent algorithms into three classes: data manipulation, representation learning, and learning strategy, and present several popular algorithms in detail for each category. Third, we introduce the commonly used datasets, applications, and our open-sourced codebase for fair evaluation. Finally, we summarize existing literature and present some potential research topics for the future.

• The paper’s main contribution is its formal definition of domain generalization and theoretical analysis, differentiating it from domain adaptation with concrete error bounds.
• It categorizes DG methods into data manipulation, representation learning, and learning strategy, supported by practical benchmarks and numerical results.
• The survey highlights DG applications in computer vision, medical analysis, and more, while outlining promising future research directions in test-time adaptation and continuous generalization.

Overview of "Generalizing to Unseen Domains: A Survey on Domain Generalization"

The paper, "Generalizing to Unseen Domains: A Survey on Domain Generalization," authored by Jindong Wang et al., provides a comprehensive survey of domain generalization (DG) techniques designed to perform well on unseen test domains, differing from the training distributions. This survey spans the theoretical foundations, algorithmic strategies, datasets, applications, challenges, and potential future research directions for DG.

Key Contributions

1. Definition and Theoretical Foundations:
   • The authors formally define DG, emphasizing its distinction from related fields like domain adaptation, transfer learning, and zero-shot learning.
   • Theoretical analyses cover both domain adaptation and domain generalization, introducing crucial concepts such as the $H\DeltaH$-divergence and providing error bounds for DG.
2. Methodological Categorization:
   • The paper categorizes DG approaches into three primary strategies: data manipulation, representation learning, and learning strategy.
   • Detailed exploration of each category includes methods like domain randomization, domain-invariant representation learning, and gradient operation-based DG.
3. Datasets and Benchmarks:
   • A summary of popular datasets like PACS, VLCS, Office-Home, and the comprehensive WILDS benchmark suite is provided.
   • Evaluation strategies and a novel codebase, DeepDG, are introduced for consistent benchmarking, alongside insightful experimental results.
4. Application Spectrum:
   • The survey details diverse DG applications in computer vision (e.g., image classification, semantic segmentation), medical analysis, reinforcement learning, and natural language processing.

Strong Numerical Results and Bold Claims

• The paper's systematic categorization of DG methodologies, supported by rigorous theoretical analysis and practical insights, provides a solid foundation for understanding DG's efficacy.
• Experimentation with the DeepDG codebase revealed nuanced performance insights, such as RSC outperforming baseline methods on PACS and Office-Home datasets, yet highlighted the marginal improvements by some methods, including DANN and ANDMask.

Implications and Speculation on Future AI Developments

Practical Implications:

• DG methodologies are critical in applications where training and test environments cannot be accurately predicted or sampled during training, such as autonomous driving, medical diagnosis, and adaptive robotics.
• The survey underscores the importance of robust evaluation protocols to ensure that DG methods genuinely generalize to unseen domains and are not overfitting to specific dataset idiosyncrasies.

Theoretical Implications:

• The adaptation of theoretical insights from domain adaptation to DG provides a deeper understanding of how invariant risk minimization and representation learning can be beneficial.
• Causal inference and its role in DG are highlighted, proposing that causal representations lead to better generalization across domains.

Future Directions:

• Future research may delve into the continuous domain generalization and adaptation to novel categories, enhancing generalization capabilities as domains evolve or as new classes emerge.
• Incorporating large-scale pre-training and self-supervised learning techniques could further improve DG effectiveness, leveraging large datasets to build more robust models.
• Test-time generalization offers exciting potential where models adapt during inference to unseen domains without requiring extensive retraining, ensuring continuous adaptability.

Summary

This survey encapsulates the state-of-the-art in domain generalization, offering a detailed exposition of theories, methods, and applications while underscoring the need for robust evaluation frameworks. Future research in continuous domain generalization, novel category generalization, and test-time adaptation holds promise for advancing the field further. The contribution of this work lies in providing a comprehensive foundation upon which subsequent research can build, addressing the persistent challenge of unseen domain generalization in machine learning.