Papers
Topics
Authors
Recent
Search
2000 character limit reached

DPStyler: Dynamic PromptStyler for Source-Free Domain Generalization

Published 25 Mar 2024 in cs.CV | (2403.16697v2)

Abstract: Source-Free Domain Generalization (SFDG) aims to develop a model that works for unseen target domains without relying on any source domain. Research in SFDG primarily bulids upon the existing knowledge of large-scale vision-LLMs and utilizes the pre-trained model's joint vision-language space to simulate style transfer across domains, thus eliminating the dependency on source domain images. However, how to efficiently simulate rich and diverse styles using text prompts, and how to extract domain-invariant information useful for classification from features that contain both semantic and style information after the encoder, are directions that merit improvement. In this paper, we introduce Dynamic PromptStyler (DPStyler), comprising Style Generation and Style Removal modules to address these issues. The Style Generation module refreshes all styles at every training epoch, while the Style Removal module eliminates variations in the encoder's output features caused by input styles. Moreover, since the Style Generation module, responsible for generating style word vectors using random sampling or style mixing, makes the model sensitive to input text prompts, we introduce a model ensemble method to mitigate this sensitivity. Extensive experiments demonstrate that our framework outperforms state-of-the-art methods on benchmark datasets.

Authors (4)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (71)
  1. M. B. Sariyildiz, Y. Kalantidis, D. Larlus, and K. Alahari, “Concept generalization in visual representation learning,” in ICCV, 2021.
  2. Y. Wang, S. Tang, F. Zhu, L. Bai, R. Zhao, D. Qi, and W. Ouyang, “Revisiting the transferability of supervised pretraining: an mlp perspective,” in CVPR, 2022.
  3. Y. Bengio, T. Deleu, N. Rahaman, N. R. Ke, S. Lachapelle, O. Bilaniuk, A. Goyal, and C. Pal, “A meta-transfer objective for learning to disentangle causal mechanisms,” in ICLR, 2019.
  4. B. Recht, R. Roelofs, L. Schmidt, and V. Shankar, “Do imagenet classifiers generalize to imagenet?” in ICML, 2019.
  5. D. Hendrycks and T. Dietterich, “Benchmarking neural network robustness to common corruptions and perturbations,” in ICLR, 2018.
  6. Y. Ye, Y. He, T. Pan, J. Li, and H. T. Shen, “Alleviating domain shift via discriminative learning for generalized zero-shot learning,” TMM, 2021.
  7. S. Ben-David, J. Blitzer, K. Crammer, and F. Pereira, “Analysis of representations for domain adaptation,” in NeurIPS, 2006.
  8. J. Hoffman, E. Tzeng, T. Park, J.-Y. Zhu, P. Isola, K. Saenko, A. Efros, and T. Darrell, “Cycada: Cycle-consistent adversarial domain adaptation,” in ICML, 2018.
  9. S. Lee, T. Son, and S. Kwak, “Fifo: Learning fog-invariant features for foggy scene segmentation,” in CVPR, 2022.
  10. Y. Lee, A. S. Chen, F. Tajwar, A. Kumar, H. Yao, P. Liang, and C. Finn, “Surgical fine-tuning improves adaptation to distribution shifts,” in ICLR, 2022.
  11. K. Saito, D. Kim, S. Sclaroff, T. Darrell, and K. Saenko, “Semi-supervised domain adaptation via minimax entropy,” in ICCV, 2019.
  12. B. Sun, J. Feng, and K. Saenko, “Return of frustratingly easy domain adaptation,” in AAAI, 2016.
  13. E. Tzeng, J. Hoffman, K. Saenko, and T. Darrell, “Adversarial discriminative domain adaptation,” in CVPR, 2017.
  14. H. Zhao, R. T. Des Combes, K. Zhang, and G. Gordon, “On learning invariant representations for domain adaptation,” in ICML, 2019.
  15. R. Wang, Z. Wu, Z. Weng, J. Chen, G.-J. Qi, and Y.-G. Jiang, “Cross-domain contrastive learning for unsupervised domain adaptation,” TMM, 2022.
  16. Y. Lu, D. Li, W. Wang, Z. Lai, J. Zhou, and X. Li, “Discriminative invariant alignment for unsupervised domain adaptation,” TMM, 2021.
  17. J. Huang, D. Guan, A. Xiao, and S. Lu, “Fsdr: Frequency space domain randomization for domain generalization,” in CVPR, 2021.
  18. X. Yue, Y. Zhang, S. Zhao, A. Sangiovanni-Vincentelli, K. Keutzer, and B. Gong, “Domain randomization and pyramid consistency: Simulation-to-real generalization without accessing target domain data,” in ICCV, 2019.
  19. S. Seo, Y. Suh, D. Kim, G. Kim, J. Han, and B. Han, “Learning to optimize domain specific normalization for domain generalization,” in ECCV, 2020.
  20. J. Kang, S. Lee, N. Kim, and S. Kwak, “Style neophile: Constantly seeking novel styles for domain generalization,” in CVPR, 2022.
  21. H. Yang, S. Tang, M. Chen, Y. Wang, F. Zhu, L. Bai, R. Zhao, and W. Ouyang, “Domain invariant masked autoencoders for self-supervised learning from multi-domains,” in ECCV, 2022.
  22. H. Yang, X. Li, S. Tang, F. Zhu, Y. Wang, M. Chen, L. Bai, R. Zhao, and W. Ouyang, “Cycle-consistent masked autoencoder for unsupervised domain generalization,” in ICLR, 2022.
  23. C. Yang, M. B. Westover, and J. Sun, “Manydg: Many-domain generalization for healthcare applications,” in ICLR, 2022.
  24. Z. Niu, J. Yuan, X. Ma, Y. Xu, J. Liu, Y.-W. Chen, R. Tong, and L. Lin, “Knowledge distillation-based domain-invariant representation learning for domain generalization,” TMM, 2023.
  25. X. Jin, C. Lan, W. Zeng, and Z. Chen, “Style normalization and restitution for domain generalization and adaptation,” TMM, 2021.
  26. J. Cho, G. Nam, S. Kim, H. Yang, and S. Kwak, “Promptstyler: Prompt-driven style generation for source-free domain generalization,” in ICCV, 2023.
  27. A. Frikha, H. Chen, D. Krompaß, T. Runkler, and V. Tresp, “Towards data-free domain generalization,” in ACML, 2023.
  28. H. Niu, H. Li, F. Zhao, and B. Li, “Domain-unified prompt representations for source-free domain generalization,” arXiv preprint arXiv:2209.14926, 2022.
  29. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in ICML, 2021.
  30. J. Wang, C. Lan, C. Liu, Y. Ouyang, T. Qin, W. Lu, Y. Chen, W. Zeng, and P. Yu, “Generalizing to unseen domains: A survey on domain generalization,” TKDE, 2022.
  31. F. M. Carlucci, A. D’Innocente, S. Bucci, B. Caputo, and T. Tommasi, “Domain generalization by solving jigsaw puzzles,” in CVPR, 2019.
  32. J. Cha, K. Lee, S. Park, and S. Chun, “Domain generalization by mutual-information regularization with pre-trained models,” in ECCV, 2022.
  33. I. Gulrajani and D. Lopez-Paz, “In search of lost domain generalization,” in ICLR, 2020.
  34. D. Kim, Y. Yoo, S. Park, J. Kim, and J. Lee, “Selfreg: Self-supervised contrastive regularization for domain generalization,” in ICCV, 2021.
  35. K. Zhou, Y. Yang, Y. Qiao, and T. Xiang, “Domain generalization with mixstyle,” in ICLR, 2020.
  36. K. Zhou, Y. Yang, T. Hospedales, and T. Xiang, “Deep domain-adversarial image generation for domain generalisation,” in AAAI, 2020.
  37. H. Nam, H. Lee, J. Park, W. Yoon, and D. Yoo, “Reducing domain gap by reducing style bias,” in CVPR, 2021.
  38. R. Volpi, H. Namkoong, O. Sener, J. C. Duchi, V. Murino, and S. Savarese, “Generalizing to unseen domains via adversarial data augmentation,” in NeurIPS, 2018.
  39. K. Zhou, Y. Yang, T. Hospedales, and T. Xiang, “Learning to generate novel domains for domain generalization,” in ECCV, 2020.
  40. H. Li, S. J. Pan, S. Wang, and A. C. Kot, “Domain generalization with adversarial feature learning,” in CVPR, 2018.
  41. R. Shao, X. Lan, J. Li, and P. C. Yuen, “Multi-adversarial discriminative deep domain generalization for face presentation attack detection,” in CVPR, 2019.
  42. X. Yao, Y. Bai, X. Zhang, Y. Zhang, Q. Sun, R. Chen, R. Li, and B. Yu, “Pcl: Proxy-based contrastive learning for domain generalization,” in CVPR, 2022.
  43. D. Li, J. Zhang, Y. Yang, C. Liu, Y.-Z. Song, and T. M. Hospedales, “Episodic training for domain generalization,” in CVPR, 2019.
  44. X. Fan, Q. Wang, J. Ke, F. Yang, B. Gong, and M. Zhou, “Adversarially adaptive normalization for single domain generalization,” in CVPR, 2021.
  45. L. Li, K. Gao, J. Cao, Z. Huang, Y. Weng, X. Mi, Z. Yu, X. Li, and B. Xia, “Progressive domain expansion network for single domain generalization,” in CVPR, 2021.
  46. F. Qiao, L. Zhao, and X. Peng, “Learning to learn single domain generalization,” in CVPR, 2020.
  47. Z. Wang, Y. Luo, R. Qiu, Z. Huang, and M. Baktashmotlagh, “Learning to diversify for single domain generalization,” in ICCV, 2021.
  48. J. Li, D. Li, C. Xiong, and S. Hoi, “Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” in ICML, 2022.
  49. J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” arXiv preprint arXiv:2301.12597, 2023.
  50. C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. Le, Y.-H. Sung, Z. Li, and T. Duerig, “Scaling up visual and vision-language representation learning with noisy text supervision,” in ICML, 2021.
  51. J. Yang, J. Duan, S. Tran, Y. Xu, S. Chanda, L. Chen, B. Zeng, T. Chilimbi, and J. Huang, “Vision-language pre-training with triple contrastive learning,” in CVPR, 2022.
  52. L. Dunlap, C. Mohri, D. Guillory, H. Zhang, T. Darrell, J. E. Gonzalez, A. Raghunathan, and A. Rohrbach, “Using language to extend to unseen domains,” in ICLR, 2022.
  53. R. Gal, O. Patashnik, H. Maron, A. H. Bermano, G. Chechik, and D. Cohen-Or, “Stylegan-nada: Clip-guided domain adaptation of image generators,” TOG, 2022.
  54. O. Patashnik, Z. Wu, E. Shechtman, D. Cohen-Or, and D. Lischinski, “Styleclip: Text-driven manipulation of stylegan imagery,” in ICCV, 2021.
  55. G. Kwon and J. C. Ye, “Clipstyler: Image style transfer with a single text condition,” in CVPR, 2022.
  56. R. Gal, Y. Alaluf, Y. Atzmon, O. Patashnik, A. H. Bermano, G. Chechik, and D. Cohen-or, “An image is worth one word: Personalizing text-to-image generation using textual inversion,” in ICLR, 2022.
  57. J. Deng, J. Guo, N. Xue, and S. Zafeiriou, “Arcface: Additive angular margin loss for deep face recognition,” in CVPR, 2019.
  58. D. Li, Y. Yang, Y. Song, and T. M. Hospedales, “Deeper, broader and artier domain generalization,” in ICCV, 2017.
  59. C. Fang, Y. Xu, and D. N. Rockmore, “Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias,” in ICCV, 2013.
  60. H. Venkateswara, J. Eusebio, S. Chakraborty, and S. Panchanathan, “Deep hashing network for unsupervised domain adaptation,” in CVPR, 2017.
  61. X. Peng, Q. Bai, X. Xia, Z. Huang, K. Saenko, and B. Wang, “Moment matching for multi-source domain adaptation,” in ICCV, 2019.
  62. I. Gulrajani and D. Lopez-Paz, “In search of lost domain generalization,” in ICLR, 2021.
  63. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in NeurIPS, 2017.
  64. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in CVPR, 2016.
  65. Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky, “Domain-adversarial training of neural networks,” JMLR, 2016.
  66. Z. Huang, H. Wang, E. P. Xing, and D. Huang, “Self-challenging improves cross-domain generalization,” in ECCV, 2020.
  67. D. Li, Y. Yang, Y.-Z. Song, and T. Hospedales, “Learning to generalize: Meta-learning for domain generalization,” in AAAI, 2018.
  68. S. Min, N. Park, S. Kim, S. Park, and J. Kim, “Grounding visual representations with texts for domain generalization,” in ECCV, 2022.
  69. Y. Ruan, Y. Dubois, and C. J. Maddison, “Optimal representations for covariate shift,” in ICLR, 2021.
  70. L. van der Maaten and G. Hinton, “Viualizing data using t-sne,” JMLR, 2008.
  71. K. Zhou, J. Yang, C. C. Loy, and Z. Liu, “Conditional prompt learning for vision-language models,” in CVPR, 2022.
Citations (2)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up