Decoupled Deep Neural Network for Semi-supervised Semantic Segmentation (1506.04924v2)

Abstract: We propose a novel deep neural network architecture for semi-supervised semantic segmentation using heterogeneous annotations. Contrary to existing approaches posing semantic segmentation as a single task of region-based classification, our algorithm decouples classification and segmentation, and learns a separate network for each task. In this architecture, labels associated with an image are identified by classification network, and binary segmentation is subsequently performed for each identified label in segmentation network. The decoupled architecture enables us to learn classification and segmentation networks separately based on the training data with image-level and pixel-wise class labels, respectively. It facilitates to reduce search space for segmentation effectively by exploiting class-specific activation maps obtained from bridging layers. Our algorithm shows outstanding performance compared to other semi-supervised approaches even with much less training images with strong annotations in PASCAL VOC dataset.

• The paper introduces a decoupled architecture that trains separate networks for classification and segmentation to leverage heterogeneous annotations.
• It uses bridging layers to transfer class-specific information, effectively reducing the segmentation search space.
• Empirical results on the PASCAL VOC 2012 dataset demonstrate superior performance compared to existing semi-supervised methods.

Decoupled Deep Neural Network for Semi-Supervised Semantic Segmentation

The paper introduces a novel approach to semi-supervised semantic segmentation by utilizing a decoupled architecture of deep neural networks (DNNs). Unlike traditional methods that treat semantic segmentation as a single task, this approach distinctly separates the tasks of classification and segmentation, leveraging heterogeneous annotations for improved efficiency and performance.

Problem Context and Challenges

Semantic segmentation aims to assign semantic labels to pixels within an image, facing challenges like pose variation, occlusion, and limited annotated datasets. Traditional deep learning approaches necessitate extensive pixel-wise annotations, making it challenging to apply supervised methods broadly due to significant annotation costs. Semi- or weakly-supervised learning methods have been developed to alleviate annotation bottlenecks, employing various weak supervision strategies such as image-level or bounding box labels. However, these methods often suffer from convergence issues and require complex procedures to achieve reasonable segmentation performance.

Proposed Architecture

The authors propose a decoupled neural network architecture consisting of two separate networks for classification and segmentation, connected via bridging layers. The classification network predicts object labels within an image using image-level annotations, while the segmentation network subsequently performs figure-ground segmentation for each identified label using pixel-wise annotations. This decoupling reduces the segmentation search space significantly, enabling effective training with limited segmentation annotations. The bridging layers play a crucial role by transmitting class-specific information from the classification network to the segmentation network.

Architectural Components

• Classification Network: Based on VGG 16-layer architecture, it outputs class scores relevant to the image. It is trained using image-level annotations to predict a set of labels associated with input images.
• Segmentation Network: Employs deconvolution techniques to generate class-specific segmentation maps. It is trained with pixel-wise annotations, simplified by the class-specific activation maps provided by bridging layers.
• Bridging Layers: Facilitate the transfer of class-specific and spatial information from the classification to the segmentation network, thereby enhancing the latter's focus on relevant segments of the input image.

Training and Inference

The training process is straightforward, involving independent training of classification and segmentation networks. The network takes advantage of image-level annotations for initial training, followed by joint optimization of the bridging layers and segmentation network with available pixel-wise annotations. A data augmentation strategy, combinatorial cropping, is employed to generate additional training examples and mitigate the scarcity of strong annotations.

Experimental Results

Empirical evaluation on the PASCAL VOC 2012 dataset demonstrates significant performance improvements over existing semi-supervised methods. The proposed architecture shows superior results even with a minimal number of strongly annotated examples, thus validating the efficiency of the decoupled approach in reducing annotation burdens while maintaining high segmentation quality.

Implications and Future Directions

The decoupled architecture presented in this paper offers a promising direction for semantic segmentation by effectively leveraging heterogeneous annotation sources. While the approach shows substantial improvements in semi-supervised settings, further exploration in joint optimization strategies could potentially enhance performance when full supervision is available. The methodology holds potential for expanded application to other computer vision tasks and could benefit from exploring advanced bridging mechanisms or integrating with other weak-supervision strategies to further mitigate training data limitations.