RobustNet: Improving Domain Generalization in Urban-Scene Segmentation via Instance Selective Whitening

Abstract: Enhancing the generalization capability of deep neural networks to unseen domains is crucial for safety-critical applications in the real world such as autonomous driving. To address this issue, this paper proposes a novel instance selective whitening loss to improve the robustness of the segmentation networks for unseen domains. Our approach disentangles the domain-specific style and domain-invariant content encoded in higher-order statistics (i.e., feature covariance) of the feature representations and selectively removes only the style information causing domain shift. As shown in Fig. 1, our method provides reasonable predictions for (a) low-illuminated, (b) rainy, and (c) unseen structures. These types of images are not included in the training dataset, where the baseline shows a significant performance drop, contrary to ours. Being simple yet effective, our approach improves the robustness of various backbone networks without additional computational cost. We conduct extensive experiments in urban-scene segmentation and show the superiority of our approach to existing work. Our code is available at https://github.com/shachoi/RobustNet.

• The paper proposes an instance selective whitening loss to reduce domain shift and enhance segmentation robustness in urban scenes.
• It leverages higher-order statistics and k-means clustering to disentangle style from content without incurring additional computation.
• Empirical evaluations on benchmarks like Cityscapes and GTAV demonstrate significant mIoU improvements over existing methods.

An Expert Review of "RobustNet: Improving Domain Generalization in Urban-Scene Segmentation via Instance Selective Whitening"

The paper "RobustNet: Improving Domain Generalization in Urban-Scene Segmentation via Instance Selective Whitening" presents a novel approach to addressing the issue of domain generalization (DG) in urban-scene segmentation, a critical challenge for deploying deep learning models in real-world applications such as autonomous driving. This study introduces Instance Selective Whitening (ISW) as a method to enhance the generalization capability of segmentation networks without incurring additional computational costs.

Overview of the Proposed Method

The authors propose an instance selective whitening loss which targets the specific aspect of feature covariance to address domain shift. Typically, domain shift hinders the performance of deep neural networks (DNNs) trained on a source domain when exposed to unforeseen target domains. The proposed method leverages higher-order statistics, specifically feature covariances, to disentangle domain-specific style from domain-invariant content. By selectively removing the style information encoded in feature covariances, RobustNet aims to reduce the domain shift effect and thereby improve model robustness.

This is achieved by incorporating no explicit closed-form whitening transformation, which is computationally expensive, but instead utilizing a loss function that implicitly encourages this transformation during training. Photometric transformations (e.g., color jittering and Gaussian blur) are applied to differentiate style-sensitive covariances from content representations. The k-means clustering algorithm is utilized to categorize these covariances, allowing ISW to suppress only those covariances that contribute to domain shifts.

Empirical Evaluation and Results

The empirical validation of the approach is thorough, involving several datasets including Cityscapes, BDD-100K, Mapillary, GTAV, and SYNTHIA. RobustNet demonstrates superior performance over established benchmarks such as IBN-Net and IterNorm in urban-scene segmentation tasks on unseen domains, validating its efficacy in generalization. For instance, when trained on GTAV and evaluated on Cityscapes, BDD-100K, and other datasets, RobustNet consistently achieved higher mean Intersection over Union (mIoU) compared to other methods. This reinforces the validity of selectively whitening style-sensitive covariances in enhancing DG.

Practical and Theoretical Implications

Practically, the proposed method could significantly improve the safety and reliability of computer vision systems in real-world settings that present numerous domain shifts, such as changing illumination and adverse weather conditions. Theoretically, the study adds to the discourse on feature covariance's role in style representation in neural networks, offering a promising research direction for those exploring DG.

RobustNet’s effective instance-selective whitening based on discriminative learning without the need for additional computational resources highlights a compelling direction for further studies. Additionally, while the focus of this work is on semantic segmentation in DG contexts, the principles could potentially be applied to other tasks facing similar domain shift challenges.

Speculations on Future Developments

The introduction of selective whitening in domain generalization opens up multiple avenues for future research. Further studies could explore the integration of ISW with more advanced meta-learning frameworks, its applicability to other computer vision tasks beyond semantic segmentation, and enhancements through other forms of augmentation to better isolate domain-invariant features. Additionally, it would be intriguing to explore incorporating ISW in a domain adaptation framework where some target domain data is available, thus blending DG and DA methodologies.

In summary, the RobustNet approach to improving domain generalization through instance selective whitening represents a significant advancement in addressing domain shifts in urban-scene segmentation. Its application across various datasets has underscored its promise and potential implications for improving AI systems' robustness in real-world environments.