Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
86 tokens/sec
GPT-4o
11 tokens/sec
Gemini 2.5 Pro Pro
53 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
3 tokens/sec
DeepSeek R1 via Azure Pro
33 tokens/sec
2000 character limit reached

Selection, Ensemble, and Adaptation: Advancing Multi-Source-Free Domain Adaptation via Architecture Zoo (2403.01582v2)

Published 3 Mar 2024 in cs.LG

Abstract: Conventional Multi-Source Free Domain Adaptation (MSFDA) assumes that each source domain provides a single source model, and all source models adopt a uniform architecture. This paper introduces Zoo-MSFDA, a more general setting that allows each source domain to offer a zoo of multiple source models with different architectures. While it enriches the source knowledge, Zoo-MSFDA risks being dominated by suboptimal/harmful models. To address this issue, we theoretically analyze the model selection problem in Zoo-MSFDA, and introduce two principles: transferability principle and diversity principle. Recognizing the challenge of measuring transferability, we subsequently propose a novel Source-Free Unsupervised Transferability Estimation (SUTE). It enables assessing and comparing transferability across multiple source models with different architectures under domain shift, without requiring target labels and source data. Based on above, we introduce a Selection, Ensemble, and Adaptation (SEA) framework to address Zoo-MSFDA, which consists of: 1) source models selection based on the proposed principles and SUTE; 2) ensemble construction based on SUTE-estimated transferability; 3) target-domain adaptation of the ensemble model. Evaluations demonstrate that our SEA framework, with the introduced Zoo-MSFDA setting, significantly improves adaptation performance (e.g., 13.5% on DomainNet). Additionally, our SUTE achieves state-of-the-art performance in transferability estimation.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (57)
  1. S. M. Ahmed, D. S. Raychaudhuri, S. Paul, S. Oymak, and A. K. Roy-Chowdhury, “Unsupervised multi-source domain adaptation without access to source data,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 10 103–10 112.
  2. J. Dong, Z. Fang, A. Liu, G. Sun, and T. Liu, “Confident anchor-induced multi-source free domain adaptation,” Advances in Neural Information Processing Systems, vol. 34, pp. 2848–2860, 2021.
  3. H. Feng, Z. You, M. Chen, T. Zhang, M. Zhu, F. Wu, C. Wu, and W. Chen, “Kd3a: Unsupervised multi-source decentralized domain adaptation via knowledge distillation.” in ICML, 2021, pp. 3274–3283.
  4. R. Gontijo-Lopes, Y. Dauphin, and E. D. Cubuk, “No one representation to rule them all: Overlapping features of training methods,” arXiv preprint arXiv:2110.12899, 2021.
  5. Y. Chen, T. Hu, F. Zhou, Z. Li, and Z.-M. Ma, “Explore and exploit the diverse knowledge in model zoo for domain generalization,” in International Conference on Machine Learning.   PMLR, 2023, pp. 4623–4640.
  6. A. T. Tran, C. V. Nguyen, and T. Hassner, “Transferability and hardness of supervised classification tasks,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 1395–1405.
  7. C. Nguyen, T. Hassner, M. Seeger, and C. Archambeau, “Leep: A new measure to evaluate transferability of learned representations,” in International Conference on Machine Learning.   PMLR, 2020, pp. 7294–7305.
  8. K. You, Y. Liu, J. Wang, and M. Long, “Logme: Practical assessment of pre-trained models for transfer learning,” in International Conference on Machine Learning.   PMLR, 2021, pp. 12 133–12 143.
  9. R. Peng, H. Zou, H. Wang, Y. Zeng, Z. Huang, and J. Zhao, “Energy-based automated model evaluation,” arXiv preprint arXiv:2401.12689, 2024.
  10. Y. Yu, Z. Yang, A. Wei, Y. Ma, and J. Steinhardt, “Predicting out-of-distribution error with the projection norm,” in International Conference on Machine Learning.   PMLR, 2022, pp. 25 721–25 746.
  11. Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky, “Domain-adversarial training of neural networks,” The journal of machine learning research, vol. 17, no. 1, pp. 2096–2030, 2016.
  12. Z. Han, H. Sun, and Y. Yin, “Learning transferable parameters for unsupervised domain adaptation,” IEEE Transactions on Image Processing, 2022.
  13. S. Ben-David, J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, and J. W. Vaughan, “A theory of learning from different domains,” Machine learning, vol. 79, no. 1, pp. 151–175, 2010.
  14. A. Gretton, K. Borgwardt, M. Rasch, B. Schölkopf, and A. Smola, “A kernel method for the two-sample-problem,” Advances in neural information processing systems, vol. 19, pp. 513–520, 2006.
  15. M. Long, Z. Cao, J. Wang, and M. I. Jordan, “Conditional adversarial domain adaptation,” arXiv preprint arXiv:1705.10667, 2017.
  16. Z. Deng, K. Zhou, D. Li, J. He, Y.-Z. Song, and T. Xiang, “Dynamic instance domain adaptation,” arXiv preprint arXiv:2203.05028, 2022.
  17. S. Chhabra, P. B. Dutta, B. Li, and H. Venkateswara, “Glocal alignment for unsupervised domain adaptation,” in Multimedia Understanding with Less Labeling on Multimedia Understanding with Less Labeling, 2021, pp. 45–51.
  18. S. Chhabra, H. Venkateswara, and B. Li, “Iterative image translation for unsupervised domain adaptation,” in Multimedia Understanding with Less Labeling on Multimedia Understanding with Less Labeling, 2021, pp. 37–44.
  19. W. Deng, Q. Liao, L. Zhao, D. Guo, G. Kuang, D. Hu, and L. Liu, “Joint clustering and discriminative feature alignment for unsupervised domain adaptation,” IEEE Transactions on Image Processing, vol. 30, pp. 7842–7855, 2021.
  20. H. Xu, M. Yang, L. Deng, Y. Qian, and C. Wang, “Neutral cross-entropy loss based unsupervised domain adaptation for semantic segmentation,” IEEE Transactions on Image Processing, vol. 30, pp. 4516–4525, 2021.
  21. J. Liang, D. Hu, and J. Feng, “Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation,” in International Conference on Machine Learning.   PMLR, 2020, pp. 6028–6039.
  22. M. Ye, J. Zhang, J. Ouyang, and D. Yuan, “Source data-free unsupervised domain adaptation for semantic segmentation,” in Proceedings of the 29th ACM International Conference on Multimedia, 2021, pp. 2233–2242.
  23. B. Yang, H.-W. Yeh, T. Harada, and P. C. Yuen, “Model-induced generalization error bound for information-theoretic representation learning in source-data-free unsupervised domain adaptation,” IEEE Transactions on Image Processing, vol. 31, pp. 419–432, 2021.
  24. Y. Du, H. Yang, M. Chen, J. Jiang, H. Luo, and C. Wang, “Generation, augmentation, and alignment: A pseudo-source domain based method for source-free domain adaptation,” arXiv preprint arXiv:2109.04015, 2021.
  25. N. Ding, Y. Xu, Y. Tang, C. Xu, Y. Wang, and D. Tao, “Source-free domain adaptation via distribution estimation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 7212–7222.
  26. Q. Tian, C. Ma, F.-Y. Zhang, S. Peng, and H. Xue, “Source-free unsupervised domain adaptation with sample transport learning,” Journal of Computer Science and Technology, vol. 36, no. 3, pp. 606–616, 2021.
  27. V. K. Kurmi, V. K. Subramanian, and V. P. Namboodiri, “Domain impression: A source data free domain adaptation method,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2021, pp. 615–625.
  28. R. Li, Q. Jiao, W. Cao, H.-S. Wong, and S. Wu, “Model adaptation: Unsupervised domain adaptation without source data,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 9641–9650.
  29. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” Advances in neural information processing systems, vol. 27, 2014.
  30. M. Arjovsky and L. Bottou, “Towards principled methods for training generative adversarial networks,” arXiv preprint arXiv:1701.04862, 2017.
  31. J. Liang, D. Hu, Y. Wang, R. He, and J. Feng, “Source data-absent unsupervised domain adaptation through hypothesis transfer and labeling transfer,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
  32. Z. Zhang, W. Chen, H. Cheng, Z. Li, S. Li, L. Lin, and G. Li, “Divide and contrast: Source-free domain adaptation via adaptive contrastive learning,” arXiv preprint arXiv:2211.06612, 2022.
  33. B. Chidlovskii, S. Clinchant, and G. Csurka, “Domain adaptation in the absence of source domain data,” in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016, pp. 451–460.
  34. P. Morerio, R. Volpi, R. Ragonesi, and V. Murino, “Generative pseudo-label refinement for unsupervised domain adaptation,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2020, pp. 3130–3139.
  35. T. van Laarhoven and E. Marchiori, “Unsupervised domain adaptation with random walks on target labelings,” arXiv preprint arXiv:1706.05335, 2017.
  36. S. Yang, Y. Wang, J. van de Weijer, L. Herranz, and S. Jui, “Exploiting the intrinsic neighborhood structure for source-free domain adaptation,” arXiv preprint arXiv:2110.04202, 2021.
  37. J. Guo, D. J. Shah, and R. Barzilay, “Multi-source domain adaptation with mixture of experts,” arXiv preprint arXiv:1809.02256, 2018.
  38. L. Yang, Y. Balaji, S.-N. Lim, and A. Shrivastava, “Curriculum manager for source selection in multi-source domain adaptation,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIV 16.   Springer, 2020, pp. 608–624.
  39. J. Hoffman, M. Mohri, and N. Zhang, “Algorithms and theory for multiple-source adaptation,” Advances in Neural Information Processing Systems, vol. 31, 2018.
  40. X. Peng, Q. Bai, X. Xia, Z. Huang, K. Saenko, and B. Wang, “Moment matching for multi-source domain adaptation,” in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 1406–1415.
  41. H. Zhao, S. Zhang, G. Wu, J. M. Moura, J. P. Costeira, and G. J. Gordon, “Adversarial multiple source domain adaptation,” Advances in neural information processing systems, vol. 31, 2018.
  42. S. Zhao, B. Li, X. Yue, Y. Gu, P. Xu, R. Hu, H. Chai, and K. Keutzer, “Multi-source domain adaptation for semantic segmentation,” Advances in Neural Information Processing Systems, vol. 32, 2019.
  43. C. Lin, S. Zhao, L. Meng, and T.-S. Chua, “Multi-source domain adaptation for visual sentiment classification,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 03, 2020, pp. 2661–2668.
  44. S. Zhao, G. Wang, S. Zhang, Y. Gu, Y. Li, Z. Song, P. Xu, R. Hu, H. Chai, and K. Keutzer, “Multi-source distilling domain adaptation,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 07, 2020, pp. 12 975–12 983.
  45. X. Chen, S. Wang, M. Long, and J. Wang, “Transferability vs. discriminability: Batch spectral penalization for adversarial domain adaptation,” in International conference on machine learning.   PMLR, 2019, pp. 1081–1090.
  46. J. N. Kundu, N. Venkat, R. V. Babu et al., “Universal source-free domain adaptation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 4544–4553.
  47. V. Vapnik, “Principles of risk minimization for learning theory,” Advances in neural information processing systems, vol. 4, 1991.
  48. M. Long, Y. Cao, J. Wang, and M. Jordan, “Learning transferable features with deep adaptation networks,” in International conference on machine learning.   PMLR, 2015, pp. 97–105.
  49. H. Bahng, S. Chun, S. Yun, J. Choo, and S. J. Oh, “Learning de-biased representations with biased representations,” in International Conference on Machine Learning.   PMLR, 2020, pp. 528–539.
  50. A. Kankainen, “Consistent testing of total independence based on the empirical characteristic function, vol. 29,” Jyväskylä: University of Jyväskylä, 1995.
  51. A. Gretton, K. Fukumizu, C. Teo, L. Song, B. Schölkopf, and A. Smola, “A kernel statistical test of independence,” Advances in neural information processing systems, vol. 20, 2007.
  52. M.-C. Dinu, M. Holzleitner, M. Beck, H. D. Nguyen, A. Huber, H. Eghbal-zadeh, B. A. Moser, S. Pereverzyev, S. Hochreiter, and W. Zellinger, “Addressing parameter choice issues in unsupervised domain adaptation by aggregation,” arXiv preprint arXiv:2305.01281, 2023.
  53. S. Yang, Y. Wang, K. Wang, S. Jui et al., “Attracting and dispersing: A simple approach for source-free domain adaptation,” in Advances in Neural Information Processing Systems, 2022.
  54. H. Venkateswara, J. Eusebio, S. Chakraborty, and S. Panchanathan, “Deep hashing network for unsupervised domain adaptation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 5018–5027.
  55. H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” arXiv preprint arXiv:1710.09412, 2017.
  56. L.-K. Huang, J. Huang, Y. Rong, Q. Yang, and Y. Wei, “Frustratingly easy transferability estimation,” in International Conference on Machine Learning.   PMLR, 2022, pp. 9201–9225.
  57. C. Spearman, “The proof and measurement of association between two things,” The American journal of psychology, vol. 100, no. 3/4, pp. 441–471, 1987.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now