ResUNet-a: a deep learning framework for semantic segmentation of remotely sensed data (1904.00592v3)

Abstract: Scene understanding of high resolution aerial images is of great importance for the task of automated monitoring in various remote sensing applications. Due to the large within-class and small between-class variance in pixel values of objects of interest, this remains a challenging task. In recent years, deep convolutional neural networks have started being used in remote sensing applications and demonstrate state of the art performance for pixel level classification of objects. \textcolor{black}{Here we propose a reliable framework for performant results for the task of semantic segmentation of monotemporal very high resolution aerial images. Our framework consists of a novel deep learning architecture, ResUNet-a, and a novel loss function based on the Dice loss. ResUNet-a uses a UNet encoder/decoder backbone, in combination with residual connections, atrous convolutions, pyramid scene parsing pooling and multi-tasking inference. ResUNet-a infers sequentially the boundary of the objects, the distance transform of the segmentation mask, the segmentation mask and a colored reconstruction of the input. Each of the tasks is conditioned on the inference of the previous ones, thus establishing a conditioned relationship between the various tasks, as this is described through the architecture's computation graph. We analyse the performance of several flavours of the Generalized Dice loss for semantic segmentation, and we introduce a novel variant loss function for semantic segmentation of objects that has excellent convergence properties and behaves well even under the presence of highly imbalanced classes.} The performance of our modeling framework is evaluated on the ISPRS 2D Potsdam dataset. Results show state-of-the-art performance with an average F1 score of 92.9\% over all classes for our best model.

• The paper presents a ResUNet-a architecture with residual connections, atrous convolutions, and pyramid pooling for enhanced segmentation of VHR images.
• The paper introduces a novel Tanimoto loss function that improves convergence and accuracy in class-imbalanced datasets.
• The framework employs multi-task learning to jointly predict segmentation, boundary, distance transform, and color reconstruction for robust performance.

Overview of ResUNet-a: A Deep Learning Framework for Semantic Segmentation of Remotely Sensed Data

The paper introduces ResUNet-a, a novel deep learning architecture specifically designed for semantic segmentation of very high-resolution (VHR) remotely sensed images, addressing critical challenges in automated scene understanding tasks integral to remote sensing applications. ResUNet-a incorporates several advanced features including residual connections, atrous convolutions, pyramid scene parsing pooling, and multi-tasking inference. The authors also propose a new Tanimoto-based loss function, providing improved performance even in class-imbalanced scenarios.

Key Contributions

1. Architecture: ResUNet-a builds on the UNet encoder/decoder backbone and integrates residual connections, which facilitate training deeper networks by mitigating vanishing/exploding gradient issues. Atrous convolutions are employed to capture multi-scale information, essential for accurate scene understanding. The use of pyramid scene parsing pooling layers further helps in aggregating contextual information, which enhances segmentation performance.
2. Loss Function: The paper proposes the Tanimoto loss with complement, a variant of the Dice loss designed to accelerate convergence and improve segmentation accuracy, particularly in class-imbalanced datasets. This loss function also demonstrates utility in continuous variable prediction domains.
3. Multi-Task Learning: ResUNet-a employs a multi-task learning framework where the network simultaneously predicts the segmentation mask, boundary, distance transform, and a colored reconstruction of the input. This integration aids the network in learning a more comprehensive understanding of the imagery, leading to improved segmentation outcomes.
4. Data Augmentation: The authors implement a robust data augmentation strategy that involves random rotations, scaling, and reflect padding. This approach is designed to provide variant perspectives of the imagery, thereby enhancing the model’s ability to generalize and recognize objects under different transformations.

Numerical Results and Performance

The performance of ResUNet-a is rigorously evaluated using the ISPRS 2D Potsdam dataset. The model achieves an average F1 score of 92.9%, with a remarkable balance across varying object classes, including built-up areas, vegetation, and vehicles. The paper particularly highlights significant improvements in classes historically challenging due to their spectral and shape similarities.

The introduction of the Tanimoto loss function provides tangible benefits, contributing to faster convergence rates and greater segmentation accuracy. Experimental comparisons delineate that the conditioned multi-task learning (CMTSK) integration further stabilizes training and reduces error variance.

Theoretical and Practical Implications

The theoretical advancements presented through ResUNet-a and the Tanimoto loss function can be extended to various other domains requiring precise object boundaries and segmentations, including medical imaging and automated driving. Practically, the use of ResUNet-a in remote sensing applications can significantly improve urban planning, infrastructure management, and environmental monitoring by providing accurate and reliable scene segmentation.

Future Directions

Future enhancements can explore the integration of additional data modalities, such as multi-temporal and hyperspectral imaging, to further augment the model’s segmentation capability. There is also potential in leveraging transfer learning techniques to initialize ResUNet-a with pre-trained weights, which could reduce training times and improve initial performance metrics.

Moreover, the scalability and adaptability of the architecture can be tested on larger and more diverse datasets, potentially fostering its application in global-scale remote sensing projects.

Conclusion

The ResUNet-a framework, with its novel architectural features and advanced loss function, sets a high benchmark in the field of semantic segmentation for remotely sensed data. It presents a significant step forward, not only in the domain of remote sensing but also in broadening the scope of deep learning in practical, high-resolution image analysis tasks.