Refign: Align and Refine for Adaptation of Semantic Segmentation to Adverse Conditions (2207.06825v3)

Abstract: Due to the scarcity of dense pixel-level semantic annotations for images recorded in adverse visual conditions, there has been a keen interest in unsupervised domain adaptation (UDA) for the semantic segmentation of such images. UDA adapts models trained on normal conditions to the target adverse-condition domains. Meanwhile, multiple datasets with driving scenes provide corresponding images of the same scenes across multiple conditions, which can serve as a form of weak supervision for domain adaptation. We propose Refign, a generic extension to self-training-based UDA methods which leverages these cross-domain correspondences. Refign consists of two steps: (1) aligning the normal-condition image to the corresponding adverse-condition image using an uncertainty-aware dense matching network, and (2) refining the adverse prediction with the normal prediction using an adaptive label correction mechanism. We design custom modules to streamline both steps and set the new state of the art for domain-adaptive semantic segmentation on several adverse-condition benchmarks, including ACDC and Dark Zurich. The approach introduces no extra training parameters, minimal computational overhead -- during training only -- and can be used as a drop-in extension to improve any given self-training-based UDA method. Code is available at https://github.com/brdav/refign.

• The paper introduces Refign, an unsupervised domain adaptation method that uses cross-domain reference images through alignment and refinement steps to improve semantic segmentation in adverse conditions.
• Refign's two-step process involves aligning images from normal and adverse conditions using unsupervised dense matching with uncertainty estimation, followed by adaptive label correction guided by confidence maps.
• Refign achieves state-of-the-art results on benchmarks like ACDC (65.5% mIoU, 10.1% improvement over baseline) and Dark Zurich (56.2% mIoU), demonstrating significant performance gains and reduced annotation needs for real-world autonomous systems.

Semantic Segmentation Adaptation to Adverse Conditions with Refign

Semantic segmentation is a fundamental task in computer vision, particularly critical for autonomous driving systems that require robust scene understanding. However, models trained under normal visual conditions are notably susceptible to performance degradation when applied to adverse conditions such as fog, night, rain, and snow. This paper introduces Refign, an approach leveraging cross-domain reference images to enhance unsupervised domain adaptation (UDA) for semantic segmentation under such adverse conditions.

Refign Approach

Refign extends self-training-based UDA methods by integrating a two-step process: alignment and refinement. The process leverages cross-condition correspondences available in multiple driving datasets, enhancing the robustness of semantic segmentation models.

1. Alignment: The approach uses a dense matching network, trained using an unsupervised methodology, to align images recorded under normal conditions with their adverse counterparts. The network incorporates predictive uncertainty to assess the reliability of the dense correspondences, producing a confidence map for each alignment.
2. Refinement: Once aligned, the model employs an adaptive label correction mechanism. This mechanism combines the target domain's predictions with those of the aligned reference domain, guided by the confidence map to correct potential errors in the target domain predictions.

This mechanism is a non-parametric plug-in that gains computational efficiency by avoiding additional training parameters and only introducing minimal overhead during training.

Numerical Results and Benchmarks

Refign marks significant improvements over base UDA methods, achieving state-of-the-art results on benchmarks like ACDC and Dark Zurich. For instance, when combined with DAFormer, Refign achieves 65.5% mean Intersection over Union (mIoU) on ACDC, a 10.1% improvement over the baseline. On Dark Zurich-test, it outperforms existing methods with a mIoU of 56.2%, and notably generalizes well to unseen domains like Nighttime Driving and BDD100k-night.

Implications and Future Work

The implications of such advancements are multi-fold, affecting both practical applications in autonomous systems and theoretical foundations in domain adaptation. Practically, this work reduces the effort required for acquiring dense annotations in adverse conditions, thus lowering the entry barrier for deploying reliable autonomous driving systems in varied environmental settings. Theoretically, it demonstrates the potential and versatility of using cross-domain correspondences and adaptive label corrections for mitigating domain shifts.

Future research directions can explore the integration of Refign with a broader range of base models and in more diverse application contexts beyond automotive use cases. Further investigations into enhancing the depth and accuracy of uncertainty estimations might augment the efficacy of the alignment process, thus providing more reliable refinement outcomes.

Conclusion

The paper positions Refign as an efficient and effective methodology for adapting semantic segmentation models to adverse conditions. By utilizing both cross-domain correspondences and adaptive label correction, it sets a new standard in the field of domain adaptation, demonstrating significant performance improvements with minimal additional computational requirements. The advancement signals a progressive step toward achieving more resilient computer vision applications suitable for real-world deployment.