What is the Added Value of UDA in the VFM Era? (2504.18190v1)

Abstract: Unsupervised Domain Adaptation (UDA) can improve a perception model's generalization to an unlabeled target domain starting from a labeled source domain. UDA using Vision Foundation Models (VFMs) with synthetic source data can achieve generalization performance comparable to fully-supervised learning with real target data. However, because VFMs have strong generalization from their pre-training, more straightforward, source-only fine-tuning can also perform well on the target. As data scenarios used in academic research are not necessarily representative for real-world applications, it is currently unclear (a) how UDA behaves with more representative and diverse data and (b) if source-only fine-tuning of VFMs can perform equally well in these scenarios. Our research aims to close these gaps and, similar to previous studies, we focus on semantic segmentation as a representative perception task. We assess UDA for synth-to-real and real-to-real use cases with different source and target data combinations. We also investigate the effect of using a small amount of labeled target data in UDA. We clarify that while these scenarios are more realistic, they are not necessarily more challenging. Our results show that, when using stronger synthetic source data, UDA's improvement over source-only fine-tuning of VFMs reduces from +8 mIoU to +2 mIoU, and when using more diverse real source data, UDA has no added value. However, UDA generalization is always higher in all synthetic data scenarios than source-only fine-tuning and, when including only 1/16 of Cityscapes labels, synthetic UDA obtains the same state-of-the-art segmentation quality of 85 mIoU as a fully-supervised model using all labels. Considering the mixed results, we discuss how UDA can best support robust autonomous driving at scale.

Evaluating the Value of Unsupervised Domain Adaptation in the Vision Foundation Models Era

The research article titled "What is the Added Value of UDA in the VFM Era?", authored by Brunó B. Englert, Tommie Kerssies, and Gijs Dubbelman, explores the effectiveness and relevance of Unsupervised Domain Adaptation (UDA) when utilizing Vision Foundation Models (VFMs) for semantic segmentation tasks. The paper seeks to assess the utility of UDA in scenarios that involve synthetic-to-real (synth-to-real) and real-to-real adaptations, comparing UDA against source-only fine-tuning of VFMs.

Background and Objectives

The paper highlights the capability of UDA to enhance generalization from a labeled source domain to an unlabeled target domain, especially when VFMs are employed with synthetic data as the source. VFMs, due to their pre-training on extensive datasets, exhibit promising generalization capabilities. Therefore, understanding whether UDA still holds added value over simpler methods like source-only fine-tuning in diverse and realistic data scenarios becomes crucial.

The primary objectives of the research are:

• To analyze UDA's behavior with diverse and representative data scenarios.
• To investigate if straightforward source-only fine-tuning of VFMs can achieve comparable results in such scenarios.

Methodology

The paper systematically evaluates UDA alongside source-only fine-tuning by experimenting with various data conditions:

• Synth-to-real adaptations: involving scaling and diversifying synthetic source data, scaling target data, and incorporating limited labeled target data.
• Real-to-real adaptations: focusing on the impact of diverse real source data and using small amounts of labeled target data.

The Cityscapes dataset serves as the primary target, while WildDash2 is used to evaluate generalization to unseen conditions.

Key Results

1. Synth-to-real scenarios: UDA displayed notable improvement over naive approaches. The added value of UDA diminished from +8 mIoU to +2 mIoU when synthetic data diversity increased. Interestingly, variations in target data had limited impact on UDA's performance.
2. Robustness to distribution changes: UDA was less sensitive to variations in the synthetic source data composition compared to source-only fine-tuning.
3. Real-to-real scenarios: When diverse real data was available, UDA offered minimal advantage over source-only fine-tuning.
4. Mixing few target labels: UDA using VFMs performed equally to fully-supervised models with all Cityscapes labels when provided only 1/16 of the target labels.

Discussion and Implications

The research presents a nuanced viewpoint on UDA's necessity within autonomous driving applications. While academically demonstrating that UDA using VFMs can closely approximate fully-supervised outcomes, the practical significance of its improvements over source-only fine-tuning remains modest. UDA might not justify its complexity in typical autonomous driving scenarios unless substantial domain gaps are present without labeled target data.

However, UDA may play a critical role in cases involving severe domain shifts or lack of sufficient source data diversity. This paper encourages future exploration into these niche applications to validate UDA's strategic importance in enhancing model robustness across broader and dynamic environments.

Conclusion

In conclusion, the paper challenges the assumption of UDA's indispensability for autonomous driving while acknowledging its potential value as a targeted strategy for overcoming significant domain shifts. It impels researchers and practitioners towards more representative data scenarios and effective adaptation techniques, aligning with VFMs' capacity to transform computer vision methodologies in real-world applications.