SegNet: A Deep Convolutional Encoder-Decoder Architecture for Robust Semantic Pixel-Wise Labelling (1505.07293v1)

Abstract: We propose a novel deep architecture, SegNet, for semantic pixel wise image labelling. SegNet has several attractive properties; (i) it only requires forward evaluation of a fully learnt function to obtain smooth label predictions, (ii) with increasing depth, a larger context is considered for pixel labelling which improves accuracy, and (iii) it is easy to visualise the effect of feature activation(s) in the pixel label space at any depth. SegNet is composed of a stack of encoders followed by a corresponding decoder stack which feeds into a soft-max classification layer. The decoders help map low resolution feature maps at the output of the encoder stack to full input image size feature maps. This addresses an important drawback of recent deep learning approaches which have adopted networks designed for object categorization for pixel wise labelling. These methods lack a mechanism to map deep layer feature maps to input dimensions. They resort to ad hoc methods to upsample features, e.g. by replication. This results in noisy predictions and also restricts the number of pooling layers in order to avoid too much upsampling and thus reduces spatial context. SegNet overcomes these problems by learning to map encoder outputs to image pixel labels. We test the performance of SegNet on outdoor RGB scenes from CamVid, KITTI and indoor scenes from the NYU dataset. Our results show that SegNet achieves state-of-the-art performance even without use of additional cues such as depth, video frames or post-processing with CRF models.

• The paper introduces SegNet, which uses stored pooling indices in decoders for effective upsampling and smooth pixel labeling.
• It employs a deep convolutional encoder-decoder architecture trained with L-BFGS to minimize pixel-wise cross-entropy loss.
• Experiments on datasets like CamVid, KITTI, and NYU v2 show SegNet’s superior performance in capturing small and thin structural details.

SegNet: A Deep Convolutional Encoder-Decoder Architecture for Robust Semantic Pixel-Wise Labelling

Introduction

The paper proposes SegNet, a novel deep convolutional encoder-decoder architecture tailored for semantic pixel-wise image labeling. Unlike prior deep learning models optimized for object categorization, SegNet addresses the need for spatial context and resolution in pixel-wise labeling. Through a stack of encoders followed by decoders, the architecture efficiently maps low-resolution feature maps back to the original image dimensions for accurate and smooth pixel labeling.

Architecture Overview

SegNet's architecture comprises several key components:

1. Encoders: Each encoder consists of convolutional layers, ReLU activations, and non-overlapping max-pooling layers, which progressively downsample the input image. The pooling indices are stored for use by the corresponding decoders.
2. Decoders: The decoder layers upsample the feature maps using the saved pooling indices, followed by convolution operations to restore high-resolution feature maps. This process counteracts the loss of spatial resolution caused by the pooling in the encoders.
3. Soft-max Layer: The final layer applies a soft-max classifier on the upsampled feature maps to perform pixel-wise classification, resulting in a labeled output image.

By structurally linking each decoder to its respective encoder via stored pooling indices, SegNet allows for efficient and effective upsampling. This architectural design avoids the need for ad hoc upsampling methods such as replication, which can lead to blocky and noisy predictions.

Methodology

The training of SegNet employs L-BFGS optimization for its stability and efficient convergence properties. The modular training approach involves sequentially adding and training deeper encoder-decoder pairs while maintaining the same objective function, i.e., minimizing the pixel-wise cross-entropy loss. This method ensures that the feature activations learned are robust and suitable for pixel-wise semantic labeling.

Performance and Experiments

The paper benchmarks SegNet's performance against several well-established datasets and methods:

1. CamVid Dataset: On this challenging dataset for outdoor scenes, SegNet demonstrated superior performance in labeling small and thin structures such as pedestrians, cars, and poles. Its deep architecture allowed for increased spatial context, resulting in smooth and accurate predictions.
2. KITTI Dataset: SegNet's robustness was further validated on the KITTI dataset, emphasizing its capability in handling varying illumination conditions and different visual environments. The experiments highlighted the benefit of supervised pre-training on the CamVid dataset, enhancing SegNet's performance on the KITTI dataset with minimal additional computational overhead.
3. NYU v2 Dataset: In indoor RGBD scenes, SegNet outperformed multi-scale convnets on several classes, showcasing its ability to manage scale changes and complex indoor environments. The high-dimensional feature maps produced smoother and more accurate semantic segmentation results.

SegNet's performance was notably high in both qualitative and quantitative evaluations, often surpassing methods using additional cues such as motion and depth. Its architecture demonstrated the capacity to retain small and thin structures within images, marking its efficacy in semantic pixel-wise labeling tasks.

Implications and Future Work

The implications of SegNet's architecture are manifold. Practically, it provides a robust solution for applications needing accurate pixel-wise labeling, such as autonomous driving and scene understanding. Theoretically, it underscores the significance of preserving spatial resolution through strategic upsampling in deep networks.

Future developments may include refining the decoder mechanisms for even more accurate pixel-wise reconstructions, integrating more sophisticated post-processing techniques like CRFs, and leveraging unsupervised training approaches for broader applicability. Additionally, exploring how SegNet handles varying amounts of missing data can further bolster its real-world utility.

Conclusion

The paper presents a comprehensive solution to semantic pixel-wise labeling through the SegNet architecture. By learning to upsample features effectively, SegNet addresses common pitfalls in pixel-wise labeling tasks and achieves high accuracy across multiple datasets. Its ability to smoothly map low-resolution encoder representations to high-resolution pixel labels makes it a valuable contribution to the field of semantic segmentation.