Stronger, Fewer, & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentation (2312.04265v5)

Abstract: In this paper, we first assess and harness various Vision Foundation Models (VFMs) in the context of Domain Generalized Semantic Segmentation (DGSS). Driven by the motivation that Leveraging Stronger pre-trained models and Fewer trainable parameters for Superior generalizability, we introduce a robust fine-tuning approach, namely Rein, to parameter-efficiently harness VFMs for DGSS. Built upon a set of trainable tokens, each linked to distinct instances, Rein precisely refines and forwards the feature maps from each layer to the next layer within the backbone. This process produces diverse refinements for different categories within a single image. With fewer trainable parameters, Rein efficiently fine-tunes VFMs for DGSS tasks, surprisingly surpassing full parameter fine-tuning. Extensive experiments across various settings demonstrate that Rein significantly outperforms state-of-the-art methods. Remarkably, with just an extra 1% of trainable parameters within the frozen backbone, Rein achieves a mIoU of 78.4% on the Cityscapes, without accessing any real urban-scene datasets.Code is available at https://github.com/w1oves/Rein.git.

• The paper introduces Rein, a parameter-efficient fine-tuning strategy that adapts vision foundation models to enhance segmentation across unseen domains.
• The study benchmarks several VFMs, including CLIP, MAE, SAM, EVA02, and DINOv2, demonstrating superior generalization with fewer trainable parameters.
• Experimental results on datasets like GTAV→Cityscapes, BDD100K, and Mapillary confirm notable mIoU gains and effective trade-offs between model complexity and performance.

Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentation

The paper presents a novel approach to tackling the challenges of Domain Generalized Semantic Segmentation (DGSS) by leveraging Vision Foundation Models (VFMs). Unlike prior models that often rely on outdated backbones such as ResNet or VGGNet, this work demonstrates the robust potential of VFMs in domain generalization tasks, showcasing notable improvements in model performance with fewer trainable parameters.

Overview and Methodological Contribution

Initially, the authors assess VFMs in DGSS settings to establish baselines. They explore VFMs, including CLIP, MAE, SAM, EVA02, and DINOv2, exhibiting their superior performance compared to existing DGSS methods. These VFMs are originally pretrained on diverse and large-scale datasets, contributing to their effectiveness in generalizing across unseen domains, a characteristic that is especially beneficial for tasks like semantic segmentation in urban scenes.

Central to the proposed approach is the "Rein" method, an innovative parameter-efficient fine-tuning strategy built upon lightweight learnable tokens. Rein modifies feature maps at each backbone layer, refining and forwarding these features to subsequent layers with precision. By employing fewer trainable parameters, Rein surpasses the performance achieved by full parameter fine-tuning across various datasets, demonstrating a remarkable trade-off between model complexity and generalization ability.

Experimental Results and Analysis

The paper provides extensive empirical validation across multiple datasets and settings. In a primary experimental setting (GTAV $\rightarrow$ Cityscapes + BDD100K + Mapillary), the Rein method achieves notable mIoU improvements, surpassing existing methods by significant margins with fewer parameter overheads. These findings illustrate that VFMs, when adapted using the Rein strategy, are not only capable of achieving high generalization but are also efficient in terms of parameter tuning.

Additionally, the paper evaluates the impact of different token lengths and ranks in the Rein method, offering insights into the optimal settings for balancing parameter count and performance. Across various VFMs, Rein exhibits consistent superiority, reinforcing its adaptability and efficacy in DGSS tasks.

Implications and Future Research

The implications of this paper are multifold. On a practical level, the paper demonstrates a promising direction for deploying VFMs in scenarios where data diversity is paramount, such as autonomous driving and real-time urban scene understanding. Theoretically, it contributes to the ongoing discourse on efficient model adaptation and generalization in machine learning.

Moving forward, the integration of Rein with other foundational tasks, like instance and panoptic segmentation, presents a rich avenue for future research. Further exploration could also investigate Rein's effectiveness under diverse and challenging conditions, such as adverse weather or nighttime scenarios. The paper sets a robust foundation for more versatile and scalable segmentation systems, potentially bridging gaps between synthetic training environments and real-world application domains.