Papers
Topics
Authors
Recent
Search
2000 character limit reached

Unified Language-driven Zero-shot Domain Adaptation

Published 10 Apr 2024 in cs.CV | (2404.07155v1)

Abstract: This paper introduces Unified Language-driven Zero-shot Domain Adaptation (ULDA), a novel task setting that enables a single model to adapt to diverse target domains without explicit domain-ID knowledge. We identify the constraints in the existing language-driven zero-shot domain adaptation task, particularly the requirement for domain IDs and domain-specific models, which may restrict flexibility and scalability. To overcome these issues, we propose a new framework for ULDA, consisting of Hierarchical Context Alignment (HCA), Domain Consistent Representation Learning (DCRL), and Text-Driven Rectifier (TDR). These components work synergistically to align simulated features with target text across multiple visual levels, retain semantic correlations between different regional representations, and rectify biases between simulated and real target visual features, respectively. Our extensive empirical evaluations demonstrate that this framework achieves competitive performance in both settings, surpassing even the model that requires domain-ID, showcasing its superiority and generalization ability. The proposed method is not only effective but also maintains practicality and efficiency, as it does not introduce additional computational costs during inference. Our project page is https://senqiaoyang.com/project/ULDA .

Definition Search Book Streamline Icon: https://streamlinehq.com
References (85)
  1. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716–23736, 2022.
  2. One-shot unsupervised domain adaptation with personalized diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 698–708, 2023.
  3. Léon Bottou. Large-scale machine learning with stochastic gradient descent. In COMPSTAT, 2010.
  4. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4):834–848, 2017.
  5. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV, 2018.
  6. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
  7. Generalized parametric contrastive learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–12, 2023.
  8. Source-free domain adaptation via distribution estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7212–7222, 2022.
  9. Pøda: Prompt-driven zero-shot domain adaptation. In ICCV, 2023.
  10. Towards robust object detection invariant to real-world domain shifts. In ICLR, 2023.
  11. Stylegan-nada: Clip-guided domain adaptation of image generators. ACM Transactions on Graphics (TOG), 41(4):1–13, 2022.
  12. Domain-adversarial training of neural networks. JMLR, 2016.
  13. Domain adaptation via prompt learning. arXiv preprint arXiv:2202.06687, 2022.
  14. Cycada: Cycle-consistent adversarial domain adaptation. In ICML, 2018.
  15. Daformer: Improving network architectures and training strategies for domain-adaptive semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9924–9935, 2022a.
  16. HRDA: Context-aware high-resolution domain-adaptive semantic segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), 2022b.
  17. Mic: Masked image consistency for context-enhanced domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11721–11732, 2023.
  18. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, 2021.
  19. Guided point contrastive learning for semi-supervised point cloud semantic segmentation. In ICCV, 2021.
  20. Development of autonomous car—part i: Distributed system architecture and development process. IEEE Transactions on Industrial Electronics, 61(12):7131–7140, 2014.
  21. Cluster-to-adapt: Few shot domain adaptation for semantic segmentation across disjoint labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4121–4131, 2022.
  22. Clipstyler: Image style transfer with a single text condition. arXiv preprint arXiv:2112.00374, 2021.
  23. Clipstyler: Image style transfer with a single text condition. In CVPR, 2022.
  24. One-shot adaptation of gan in just one clip. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
  25. Semi-supervised semantic segmentation with directional context-aware consistency. In CVPR, 2021.
  26. Decouplenet: Decoupled network for domain adaptive semantic segmentation. In ECCV, 2022.
  27. Lisa: Reasoning segmentation via large language model. arXiv preprint arXiv:2308.00692, 2023.
  28. Zero-shot day-night domain adaptation with a physics prior. In ICCV, 2021.
  29. Content-consistent matching for domain adaptive semantic segmentation. In ECCV, 2020.
  30. Align before fuse: Vision and language representation learning with momentum distillation. NeurIPS, 2021.
  31. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023a.
  32. Manipllm: Embodied multimodal large language model for object-centric robotic manipulation. arXiv preprint arXiv:2312.16217, 2023b.
  33. Bidirectional learning for domain adaptation of semantic segmentation. In CVPR, 2019.
  34. Llama-vid: An image is worth 2 tokens in large language models. arXiv preprint arXiv:2311.17043, 2023c.
  35. Full-scale selective transformer for semantic segmentation. In Proceedings of the Asian Conference on Computer Vision, pages 2663–2679, 2022a.
  36. Prototype-guided continual adaptation for class-incremental unsupervised domain adaptation. In European Conference on Computer Vision, pages 351–368. Springer, 2022b.
  37. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
  38. Spatial pruned sparse convolution for efficient 3d object detection. In NeurIPS, 2022a.
  39. Unsupervised spike depth estimation via cross-modality cross-domain knowledge transfer. arXiv preprint arXiv:2208.12527, 2022b.
  40. Adaptive distribution masked autoencoders for continual test-time adaptation. arXiv preprint arXiv:2312.12480, 2023a.
  41. Vida: Homeostatic visual domain adapter for continual test time adaptation. arXiv preprint arXiv:2306.04344, 2023b.
  42. Learning transferable features with deep adaptation networks. In International conference on machine learning, pages 97–105. PMLR, 2015.
  43. Deep transfer learning with joint adaptation networks. In International conference on machine learning, pages 2208–2217. PMLR, 2017.
  44. Pfenet++: Boosting few-shot semantic segmentation with the noise-filtered context-aware prior mask. TPAMI, 2024.
  45. Adversarial style mining for one-shot unsupervised domain adaptation. In NeurIPS, 2020.
  46. Distribution-aware continual test time adaptation for semantic segmentation. arXiv preprint arXiv:2309.13604, 2023.
  47. Online domain adaptation for semantic segmentation in ever-changing conditions. In European Conference on Computer Vision, pages 128–146. Springer, 2022.
  48. Hierarchical dense correlation distillation for few-shot segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23641–23651, 2023.
  49. Oa-cnns: Omni-adaptive sparse cnns for 3d semantic segmentation. arXiv preprint arXiv:2403.14418, 2024.
  50. Learning transferable visual models from natural language supervision. In ICML, 2021.
  51. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821–8831. PMLR, 2021.
  52. Playing for data: Ground truth from computer games. In ECCV, 2016.
  53. Acdc: The adverse conditions dataset with correspondences for semantic driving scene understanding. In ICCV, 2021.
  54. Deep coral: Correlation alignment for deep domain adaptation. In ECCV, 2016.
  55. Semantic diffusion network for semantic segmentation. Advances in Neural Information Processing Systems, 35:8702–8716, 2022.
  56. Generalized few-shot semantic segmentation. In CVPR, 2022a.
  57. Prior guided feature enrichment network for few-shot segmentation. TPAMI, 2022b.
  58. Adaptive perspective distillation for semantic segmentation. TPAMI, 2023a.
  59. Learning context-aware classifier for semantic segmentation. In AAAI, 2023b.
  60. Dacs: Domain adaptation via cross-domain mixed sampling. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1379–1389, 2021.
  61. Learning to adapt structured output space for semantic segmentation. In CVPR, 2018.
  62. Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation. In CVPR, 2019.
  63. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006.10726, 2020.
  64. Exploring domain-invariant parameters for source free domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7151–7160, 2022a.
  65. Generalizing to unseen domains: A survey on domain generalization. IEEE Transactions on Knowledge and Data Engineering, 2022b.
  66. Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7201–7211, 2022c.
  67. Handwriting recognition under natural writing habits based on a low-cost inertial sensor. IEEE Sensors Journal, 2023.
  68. Deep domain adaptation by geodesic distance minimization. In ICCV Workshops, 2017.
  69. Fully transformer networks for semantic image segmentation. arXiv preprint arXiv:2106.04108, 2021a.
  70. Dannet: A one-stage domain adaptation network for unsupervised nighttime semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15769–15778, 2021b.
  71. Style mixing and patchwise prototypical matching for one-shot unsupervised domain adaptive semantic segmentation. In AAAI, 2022.
  72. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 34:12077–12090, 2021.
  73. Opencda: an open cooperative driving automation framework integrated with co-simulation. In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pages 1155–1162. IEEE, 2021.
  74. Lidar-llm: Exploring the potential of large language models for 3d lidar understanding. arXiv preprint arXiv:2312.14074, 2023a.
  75. An improved baseline for reasoning segmentation with large language model. arXiv preprint arXiv:2312.17240, 2023b.
  76. Exploring sparse visual prompt for cross-domain semantic segmentation. arXiv preprint arXiv:2303.09792, 2023c.
  77. Exploring sparse visual prompt for domain adaptive dense prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 16334–16342, 2024.
  78. Fda: Fourier domain adaptation for semantic segmentation. In CVPR, 2020.
  79. Boosting novel category discovery over domains with soft contrastive learning and all in one classifier. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11858–11867, 2023.
  80. Deep learning for medical image segmentation: Tricks, challenges and future directions. arXiv preprint arXiv:2209.10307.
  81. Seine: Structure encoding and interaction network for nuclei instance segmentation. arXiv preprint arXiv:2401.09773, 2024a.
  82. Boundary-aware contrastive learning for semi-supervised nuclei instance segmentation. arXiv preprint arXiv:2402.04756, 2024b.
  83. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2881–2890, 2017.
  84. Regionblip: A unified multi-modal pre-training framework for holistic and regional comprehension. arXiv preprint arXiv:2308.02299, 2023.
  85. Confidence regularized self-training. In ICCV, 2019.
Citations (3)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up