Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Open Domain Generalization with a Single Network by Regularization Exploiting Pre-trained Features (2312.05141v1)

Published 8 Dec 2023 in cs.CV

Abstract: Open Domain Generalization (ODG) is a challenging task as it not only deals with distribution shifts but also category shifts between the source and target datasets. To handle this task, the model has to learn a generalizable representation that can be applied to unseen domains while also identify unknown classes that were not present during training. Previous work has used multiple source-specific networks, which involve a high computation cost. Therefore, this paper proposes a method that can handle ODG using only a single network. The proposed method utilizes a head that is pre-trained by linear-probing and employs two regularization terms, each targeting the regularization of feature extractor and the classification head, respectively. The two regularization terms fully utilize the pre-trained features and collaborate to modify the head of the model without excessively altering the feature extractor. This ensures a smoother softmax output and prevents the model from being biased towards the source domains. The proposed method shows improved adaptability to unseen domains and increased capability to detect unseen classes as well. Extensive experiments show that our method achieves competitive performance in several benchmarks. We also justify our method with careful analysis of the effect on the logits, features, and the head.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (44)
  1. 2008. Spearman Rank Correlation Coefficient, 502–505. New York, NY: Springer New York. ISBN 978-0-387-32833-1.
  2. Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing Systems, 34: 22405–22418.
  3. Domain Generalization by Mutual-Information Regularization with Pre-trained Models. arXiv preprint arXiv:2203.10789.
  4. An analysis of single-layer networks in unsupervised feature learning. In International Conference on Artificial Intelligence and Statistics (AISTATS), 215–223.
  5. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, 248–255. Ieee.
  6. Domain generalization via model-agnostic learning of semantic features. Advances in Neural Information Processing Systems, 32.
  7. Learning to Detect Open Classes for Universal Domain Adaptation. In European Conference on Computer Vision (ECCV).
  8. Scatter component analysis: A unified framework for domain adaptation and domain generalization. IEEE transactions on pattern analysis and machine intelligence, 39(7): 1414–1430.
  9. Domain generalization for object recognition with multi-task autoencoders. In Proceedings of the IEEE international conference on computer vision, 2551–2559.
  10. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 770–778.
  11. Universal Language Model Fine-tuning for Text Classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 328–339.
  12. Self-Challenging Improves Cross-Domain Generalization. In European Conference on Computer Vision (ECCV).
  13. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407.
  14. Partial transfusion: on the expressive influence of trainable batch norm parameters for transfer learning. In Medical Imaging with Deep Learning, 338–353. PMLR.
  15. Selfreg: Self-supervised contrastive regularization for domain generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 9619–9628.
  16. Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution. In International Conference on Learning Representations.
  17. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1): 1334–1373.
  18. Learning to generalize: Meta-learning for domain generalization. In Proceedings of the AAAI conference on artificial intelligence, volume 32.
  19. Deeper, broader and artier domain generalization. In IEEE International Conference on Computer Vision (ICCV), 5543–5551. IEEE.
  20. Domain generalization with adversarial feature learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, 5400–5409.
  21. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV), 624–639.
  22. Feature-Critic Networks for Heterogeneous Domain Generalization. In International Conference on Machine Learning (ICML), volume 97, 3915–3924.
  23. Enhancing the reliability of out-of-distribution image detection in neural networks. arXiv preprint arXiv:1706.02690.
  24. Towards Recognizing Unseen Categories in Unseen Domains. In European Conference on Computer Vision (ECCV).
  25. Domain generalization via invariant feature representation. In International Conference on Machine Learning, 10–18. PMLR.
  26. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32, 8024–8035. Curran Associates, Inc.
  27. Moment matching for multi-source domain adaptation. In IEEE International Conference on Computer Vision (ICCV), 1406–1415.
  28. VisDA: The Visual Domain Adaptation Challenge.
  29. Efficient domain generalization via common-specific low-rank decomposition. In International Conference on Machine Learning, 7728–7738. PMLR.
  30. Adapting Visual Category Models to New Domains. In European Conference on Computer Vision (ECCV).
  31. Semi-supervised domain adaptation via minimax entropy. In Proceedings of the IEEE/CVF international conference on computer vision, 8050–8058.
  32. Generalizing Across Domains via Cross-Gradient Training. In International Conference on Learning Representations.
  33. Open domain generalization with domain-augmented meta-learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9624–9633.
  34. Open-Set Recognition: A Good Closed-Set Classifier is All You Need. In International Conference on Learning Representations.
  35. Deep Hashing Network for Unsupervised Domain Adaptation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  36. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems (NeurIPS), 10506–10518.
  37. Composed fine-tuning: Freezing pre-trained denoising autoencoders for improved generalization. In International Conference on Machine Learning, 11424–11435. PMLR.
  38. Universal domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2720–2729.
  39. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 18123–18133.
  40. Domain generalization via entropy regularization. Advances in Neural Information Processing Systems, 33: 16096–16107.
  41. Learning placeholders for open-set recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 4401–4410.
  42. Learning to generate novel domains for domain generalization. In European conference on computer vision, 561–578. Springer.
  43. Domain Generalization with MixStyle. In International Conference on Learning Representations.
  44. CrossMatch: Cross-Classifier Consistency Regularization for Open-Set Single Domain Generalization. In International Conference on Learning Representations.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Inseop Chung (13 papers)
  2. KiYoon Yoo (13 papers)
  3. Nojun Kwak (116 papers)

Summary

Open Domain Generalization with a Single Network by Regularization Exploiting Pre-trained Features

This paper presents a novel approach to Open Domain Generalization (ODG), tackling the problems of distribution shifts and category shifts between source and target datasets, with the proposal to utilize a single neural network. Unlike preceding methods that employ multiple source-specific networks with significant computational costs, this paper proposes a more resource-efficient architecture. It integrates two regularization terms to leverage pre-trained features, enabling better generalization across unseen domains and identifying unknown classes.

Key Methodological Innovations

The paper introduces several key methodological contributions:

  1. Network Architecture:
    • Utilizes a single network comprising a feature extractor and a classifier head.
    • Leverages a pre-trained head by linear-probing, avoiding a complete overhaul of the feature extractor weights and thereby preserving beneficial pre-trained features.
  2. Regularization Terms:
    • Feature Regularization: Clusters source domain features by minimizing the distance to pre-established prototypes, computed across pre-trained features.
    • Head Regularization: Maximizes the entropy of the softmax output over pre-trained feature inputs, pushing the network to generalize beyond the source domains.

These regularization techniques prevent the overfitting of the network to source domains, leading to a smoother softmax output that resists biases towards known domains.

Empirical Results

The proposed method undergoes extensive testing on benchmark datasets such as PACS, Office-Home, and a Multi-Dataset scenario. Key results include:

  • Performance: The method achieves competitive performance, often surpassing existing ensemble methods like DAML in terms of accuracy and harmonic mean scores.
  • Efficiency: With only one network, the approach reduces both inference cost and computational demand significantly compared to multi-network solutions.

Analysis and Theoretical Insights

  • The paper provides thorough quantitative analyses on several aspects, including domain gap and intra-class variance, illustrating the efficacy of their approach in narrowing the domain gap and maintaining smaller intra-class variance.
  • It reports higher adaptability of the model via the proposed regularization, evidenced by smoother class probabilities and reduced bias towards source domain classes.

Implications and Future Directions

This research contributes to the ongoing discourse in domain generalization by demonstrating that sophisticated usage of pre-trained features can significantly enhance performance even with the constraint of a single network. The simplification in network architecture can have profound implications for deploying domain generalization systems in environments with limited resources such as mobile devices or IoT systems.

Theoretical implications point toward the enduring value of pre-trained models and the potential to utilize them beyond the current transfer learning paradigms. Future research could explore expanding this regularization framework to other architectures or tasks, potentially broadening the toolkit for tackling both seen and unseen classes more efficiently and effectively.

In conclusion, the paper provides a meaningful step forward in the domain generalization field, combining the efficiency of single-network solutions with the robustness required to handle open domain scenarios. This methodology sets a precedent for further reducing computational loads while enhancing system adaptability to uncharted domains and data distributions.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown