FickleNet: Weakly and Semi-supervised Semantic Image Segmentation using Stochastic Inference (1902.10421v2)

Abstract: The main obstacle to weakly supervised semantic image segmentation is the difficulty of obtaining pixel-level information from coarse image-level annotations. Most methods based on image-level annotations use localization maps obtained from the classifier, but these only focus on the small discriminative parts of objects and do not capture precise boundaries. FickleNet explores diverse combinations of locations on feature maps created by generic deep neural networks. It selects hidden units randomly and then uses them to obtain activation scores for image classification. FickleNet implicitly learns the coherence of each location in the feature maps, resulting in a localization map which identifies both discriminative and other parts of objects. The ensemble effects are obtained from a single network by selecting random hidden unit pairs, which means that a variety of localization maps are generated from a single image. Our approach does not require any additional training steps and only adds a simple layer to a standard convolutional neural network; nevertheless it outperforms recent comparable techniques on the Pascal VOC 2012 benchmark in both weakly and semi-supervised settings.

• The paper presents a novel stochastic inference method using random feature selection in a modified VGG-16 to enhance object localization.
• It employs spatial dropout and map expansion techniques to generate diverse pseudo-labels from weak annotations.
• Experiments on PASCAL VOC 2012 show improved segmentation with mIoU scores of 61.2% (weakly supervised) and 65.8% (semi-supervised).

Overview of "FickleNet: Weakly and Semi-supervised Semantic Image Segmentation using Stochastic Inference"

The paper "FickleNet: Weakly and Semi-supervised Semantic Image Segmentation using Stochastic Inference" introduces a novel approach to tackle the challenges associated with semantic image segmentation using only image-level annotations. The principal hurdle addressed by this research is obtaining pixel-level segmentation data from weak annotations that merely confirm the presence of specific objects in an image without detailed location data. To overcome this, the authors present FickleNet, a framework leveraging stochastic inference to enhance localization maps necessary for semantically segmenting images.

Methodology

FickleNet operates in two primary phases: training a classification network with stochastic feature selection and inferring a set of localization maps that serve as pseudo-labels. It employs a traditional neural network framework, specifically utilizing a modified VGG-16 architecture, where feature maps undergo random selection of hidden units incorporating spatial dropout. This stochastic approach generates diverse receptive fields, some of which resemble dilated convolutional features, allowing the network to consider variable object parts and expand beyond the most obvious discriminative areas.

Key Innovations

• Stochastic Hidden Unit Selection: Traditional methods focus on static, discriminative regions, possibly missing comprehensive object coverage. FickleNet's random feature selection and spatial dropout allow exploration of various object parts, enabling greater generalization and object boundary refinement.
• Map Expansion Technique: To optimize computation without sacrificing feature diversity, the authors propose expanding feature maps to prevent overlap in sliding window positions. This approach significantly accelerates processing while conserving memory, accommodating the augmented size of the processed feature maps.
• Aggregation of Localization Maps: The technique aggregates multiple localization maps from the inferred stochastic inferences into a singular coherent map. By iteratively generating and merging these maps, broader and more inclusive object region identification is achieved.

Experimental Results

FickleNet was evaluated on the PASCAL VOC 2012 dataset under both weakly supervised and semi-supervised scenarios. It surpassed several contemporary methods by achieving a mean intersection-over-union (mIoU) score of 61.2% for weakly supervised segmentation. In semi-supervised settings, with access to a subset of fully annotated data, the framework obtained an mIoU of 65.8%, approaching the performance of models trained on full supervision.

Implications and Future Directions

FickleNet’s results signify substantial potential in reducing the reliance on expensive, labor-intensive pixel-level annotations for semantic segmentation tasks. The approach effectively bridges the gap between weakly and fully supervised segmentation by enhancing object coverage using stochastic processes. The efficient design of the system, notably the innovative map expansion, implies practical viability for large-scale implementations without prohibitive computational costs.

The paper’s promising outcomes encourage further exploration into stochastic methods in neural networks, which could catalyze advancements in weakly supervised learning domains. Subsequent research may refine these stochastic processes, optimize ensemble effects further, and possibly extend the methodology to other computer vision challenges, such as object detection and instance segmentation. These extensions could lead to improvements in training large-scale models with limited granular input, fostering more accessible AI model development pathways.

In summary, the work presents a substantial contribution in the field of computer vision, advancing methodologies that leverage limited annotations for precise semantic segmentation, with potential widespread impact in academia and industry applications.