Learning to Adapt Structured Output Space for Semantic Segmentation (1802.10349v3)

Abstract: Convolutional neural network-based approaches for semantic segmentation rely on supervision with pixel-level ground truth, but may not generalize well to unseen image domains. As the labeling process is tedious and labor intensive, developing algorithms that can adapt source ground truth labels to the target domain is of great interest. In this paper, we propose an adversarial learning method for domain adaptation in the context of semantic segmentation. Considering semantic segmentations as structured outputs that contain spatial similarities between the source and target domains, we adopt adversarial learning in the output space. To further enhance the adapted model, we construct a multi-level adversarial network to effectively perform output space domain adaptation at different feature levels. Extensive experiments and ablation study are conducted under various domain adaptation settings, including synthetic-to-real and cross-city scenarios. We show that the proposed method performs favorably against the state-of-the-art methods in terms of accuracy and visual quality.

Learning to Adapt Structured Output Space for Semantic Segmentation

Semantic segmentation aims to assign a semantic label to each pixel in an image, facilitating its understanding and application in tasks such as autonomous driving and image editing. However, conventional convolutional neural network (CNN)-based methods often struggle to generalize well to unseen image domains due to the domain gap arising from variations in appearance, lighting, and other scene properties. This necessitates the development of domain adaptation techniques that can transfer knowledge from a labeled source domain to an unlabeled target domain.

Summary

The paper "Learning to Adapt Structured Output Space for Semantic Segmentation" by Yi-Hsuan Tsai et al. introduces a novel domain adaptation method based on adversarial learning in the output space for semantic segmentation. The core insight is that segmentation outputs contain rich structural information, such as spatial layout and local context, which remain consistent across domains despite variations in image appearance. Hence, the proposed method focuses on aligning the segmentation outputs of source and target domains rather than on feature-level adaptation.

Methodology

The proposed method integrates two main components:

1. Segmentation Network (Generator, G): This network predicts segmentation maps from input images.
2. Discriminator (D): A fully-convolutional network tasked with distinguishing whether segmentation outputs are from the source or the target domain.

The network is trained using two types of losses:

• Segmentation Loss: A cross-entropy loss applied to the predictions for source domain images.
• Adversarial Loss: A loss that encourages the generation of similar segmentation distributions for both source and target domains, achieved by training the discriminator to distinguish between them and then training the generator to fool the discriminator.

To improve adaptation further, the authors propose a multi-level adversarial learning scheme. It incorporates additional discriminators at different feature levels within the segmentation network, enabling better adaptation of both high- and low-level features.

Experimental Setup

The authors validate their approach using synthetic-to-real and cross-city adaptation scenarios:

• Synthetic-to-Real: Models are trained on synthetic datasets like GTA5 and SYNTHIA and tested on real-world datasets like Cityscapes.
• Cross-City: Models trained on images from one city are adapted to perform on another city, adjusting to subtle differences across urban environments.

Comprehensive experiments compare the performance of their method against state-of-the-art techniques and include ablation studies to evaluate the relative contributions of feature-level versus output space adaptation, as well as single-level versus multi-level adversarial learning.

Results

The proposed method demonstrates superior performance in terms of mean Intersection-over-Union (mIoU) compared to baseline models and contemporary state-of-the-art algorithms. For instance, the adaptation of GTA5 to Cityscapes using the single-level output space adaptation achieves a notable mIoU improvement over feature-level adaptation approaches. When employing multi-level adversarial learning, additional performance gains are observed, showcasing the efficacy of incorporating multiple adaptation points within the network.

Numerical Highlights

• GTA5 to Cityscapes (VGG-16 baseline): The proposed single-level adaptation method achieves an mIoU of 35.0%, outperforming methods such as "CyCADA (pixel)" which achieves 34.8%.
• GTA5 to Cityscapes (ResNet-101 baseline): Multi-level adversarial learning attains an mIoU of 42.4%, highlighting significant improvement over the baseline's 36.6%.

Implications and Future Work

The paper's contributions are significant for both practical and theoretical aspects of semantic segmentation:

• Practical Implication: The method reduces the labor-intensive process of annotating images in the target domain, showing that effective domain adaptation can be achieved by focusing on structured output alignment.
• Theoretical Implication: It provides a new perspective on adversarial learning, emphasizing the efficacy of output space adaptation for pixel-level prediction tasks.

Future developments could explore combining pixel-level transformation techniques, such as those used in CyCADA, with output space adaptation to enhance performance further. Additionally, the method's extension to other pixel-level tasks such as instance segmentation and optical flow estimation holds promising potential.

In conclusion, the paper offers a robust, adversarial learning-based domain adaptation framework for semantic segmentation, demonstrating significant improvements across varied benchmarks and paving the way for future advancements in unsupervised domain adaptation techniques.