Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Source-Free Domain Adaptation with Frozen Multimodal Foundation Model (2311.16510v3)

Published 27 Nov 2023 in cs.CV

Abstract: Source-Free Domain Adaptation (SFDA) aims to adapt a source model for a target domain, with only access to unlabeled target training data and the source model pre-trained on a supervised source domain. Relying on pseudo labeling and/or auxiliary supervision, conventional methods are inevitably error-prone. To mitigate this limitation, in this work we for the first time explore the potentials of off-the-shelf vision-language (ViL) multimodal models (e.g.,CLIP) with rich whilst heterogeneous knowledge. We find that directly applying the ViL model to the target domain in a zero-shot fashion is unsatisfactory, as it is not specialized for this particular task but largely generic. To make it task specific, we propose a novel Distilling multimodal Foundation model(DIFO)approach. Specifically, DIFO alternates between two steps during adaptation: (i) Customizing the ViL model by maximizing the mutual information with the target model in a prompt learning manner, (ii) Distilling the knowledge of this customized ViL model to the target model. For more fine-grained and reliable distillation, we further introduce two effective regularization terms, namely most-likely category encouragement and predictive consistency. Extensive experiments show that DIFO significantly outperforms the state-of-the-art alternatives. Code is here

Definition Search Book Streamline Icon: https://streamlinehq.com
References (54)
  1. Robust cross-modal representation learning with progressive self-distillation. In CVPR, 2022.
  2. Contrastive test-time adaptation. In CVPR, 2022a.
  3. Self-supervised noisy label learning for source-free unsupervised domain adaptation. In IROS, 2022b.
  4. Promptstyler: Prompt-driven style generation for source-free domain generalization. In ICCV, 2023.
  5. Source-free domain adaptation via distribution estimation. In CVPR, 2022.
  6. Generation, augmentation, and alignment: A pseudo-source domain based method for source-free domain adaptation. arXiv:2109.04015, 2021.
  7. Interpreting kullback–leibler divergence with the neyman–pearson lemma. Journal of Multivariate Analysis, 97(9):2034–2040, 2006.
  8. Unsupervised domain adaptation by backpropagation. In ICML, 2015.
  9. Domain adaptation via prompt learning. IEEE Transactions on Neural Networks and Learning Systems, 2023.
  10. A survey on vision transformer. IEEE transactions on pattern analysis and machine intelligence, 45(1):87–110, 2022.
  11. Deep residual learning for image recognition. In CVPR, 2016.
  12. Model adaptation: Historical contrastive learning for unsupervised domain adaptation without source data. In NeurIPS, 2021.
  13. Invariant information clustering for unsupervised image classification and segmentation. In CVPR, 2019.
  14. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, 2021.
  15. Contrastive adaptation network for unsupervised domain adaptation. In CVPR, 2019.
  16. Balancing discriminability and transferability for source-free domain adaptation. In ICML, 2022.
  17. Domain impression: A source data free domain adaptation method. In WACV, 2021.
  18. Padclip: Pseudo-labeling with adaptive debiasing in clip for unsupervised domain adaptation. In ICCV, 2023.
  19. Hypothesis disparity regularized mutual information maximization. In AAAI, 2021.
  20. Confidence score for source-free unsupervised domain adaptation. In ICML, 2022.
  21. Divergence-agnostic unsupervised domain adaptation by adversarial attacks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11):8196–8211, 2021.
  22. Model adaptation: Unsupervised domain adaptation without ource data. In CVPR, 2020.
  23. Open-vocabulary semantic segmentation with mask-adapted clip. In CVPR, 2023.
  24. Distant supervised centroid shift: A simple and efficient approach to visual domain adaptation. In CVPR, 2019.
  25. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In ICML, 2020.
  26. Guiding pseudo-labels with uncertainty estimation for source-free unsupervised domain adaptation. In CVPR, 2023.
  27. Conditional adversarial domain adaptation. In NeurIPS, 2018.
  28. When does label smoothing help? In NeurIPS, 2019.
  29. Visda: The visual domain adaptation challenge. arXiv:1710.06924, 2017.
  30. Moment matching for multi-source domain adaptation. In ICCV, 2019.
  31. Learning transferable visual models from natural language supervision. In ICML, 2021.
  32. Uncertainty-guided source-free domain adaptation. In ECCV, 2022.
  33. Adapting visual category models to new domains. In ECCV, 2010.
  34. Semi-supervised domain adaptation via minimax entropy. In ICCV, 2019.
  35. Grad-cam: Visual explanations from deep networks via gradient-based localization. In ICCV, 2017.
  36. Ad-clip: Adapting domains in prompt space using clip. In ICCV Workshop, 2023.
  37. Adaptive pedestrian detection by predicting classifier. Neural Computing and Applications, 31:1189–1200, 2019.
  38. Model adaptation through hypothesis transfer with gradual knowledge distillation. In IROS, 2021.
  39. Nearest neighborhood-based deep clustering for source data-absent unsupervised domain adaptation. arXiv:2107.12585, 2021.
  40. Semantic consistency learning on manifold for source data-free unsupervised domain adaptation. Neural Networks, 152, 2022.
  41. Source-free domain adaptation via target prediction distribution searching. International Journal of Computer Vision, pages 1–19, 2023.
  42. A prototype-oriented framework for unsupervised domain adaptation. In NeurIPS, 2021.
  43. Vdm-da: Virtual domain modeling for source data-free domain adaptation. IEEE Transactions on Circuits and Systems for Video Technology, 32(6):3749–3760, 2021a.
  44. Vdm-da: Virtual domain modeling for source data-free domain adaptation. IEEE Transactions on Circuits and Systems for Video Technology, 32(6):3749–3760, 2021b.
  45. Deep hashing network for unsupervised domain adaptation. In CVPR, 2017.
  46. Exploring domain-invariant parameters for source free domain adaptation. In CVPR, 2022.
  47. Adaptive adversarial network for source-free domain adaptation. In CVPR, 2021.
  48. Larger norm more transferable: An adaptive feature norm approach for unsupervised domain adaptation. In CVPR, 2019.
  49. Exploiting the intrinsic neighborhood structure for source-free domain adaptation. In NeurIPS, 2021.
  50. Attracting and dispersing: A simple approach for source-free domain adaptation. In NeurIPS, 2022.
  51. When source-free domain adaptation meets learning with noisy labels. In ICLR, 2023.
  52. Class relationship embedded learning for source-free unsupervised domain adaptation. In CVPR, 2023.
  53. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337–2348, 2022.
  54. ZongxianLee. A pytorch implementation of maximum mean discrepancies (MMD) loss. https://github.com/ZongxianLee/MMD_Loss.Pytorch, 2019.
Citations (10)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now