Deeply supervised salient object detection with short connections (1611.04849v4)

Abstract: Recent progress on saliency detection is substantial, benefiting mostly from the explosive development of Convolutional Neural Networks (CNNs). Semantic segmentation and saliency detection algorithms developed lately have been mostly based on Fully Convolutional Neural Networks (FCNs). There is still a large room for improvement over the generic FCN models that do not explicitly deal with the scale-space problem. Holistically-Nested Edge Detector (HED) provides a skip-layer structure with deep supervision for edge and boundary detection, but the performance gain of HED on salience detection is not obvious. In this paper, we propose a new method for saliency detection by introducing short connections to the skip-layer structures within the HED architecture. Our framework provides rich multi-scale feature maps at each layer, a property that is critically needed to perform segment detection. Our method produces state-of-the-art results on 5 widely tested salient object detection benchmarks, with advantages in terms of efficiency (0.15 seconds per image), effectiveness, and simplicity over the existing algorithms.

• The paper introduces a deeply supervised network with short connections that blends multi-level features for more precise saliency maps.
• It employs additional convolutional layers and a fully connected CRF to significantly improve F-measure and MAE scores across five benchmarks.
• The method demonstrates practical efficiency by processing images in just 0.08 seconds, proving its robustness in complex visual environments.

Deeply Supervised Salient Object Detection with Short Connections

The paper tackles the problem of salient object detection (SOD) by leveraging Convolutional Neural Networks (CNNs), particularly focusing on enhancing Fully Convolutional Networks (FCNs) through deep supervision and short connections. Aimed at producing more accurate saliency maps, the proposed approach builds on the Holistically-Nested Edge Detector (HED) framework but introduces modifications to address its limitations in saliency detection.

Core Contributions and Architecture

The core contribution of this paper is the introduction of short connections between various layers within CNNs, enhancing multi-level and multi-scale feature representations. While HED uses skip-layer structures for edge detection, the gain for saliency detection is not substantial due to the different nature of these tasks. Saliency detection demands not only identifying boundaries but also distinguishing entire objects from complex backgrounds. To address these challenges, the authors propose a deeply supervised network with short connections that provide advanced feature representations at each layer, which are critically needed for effective saliency detection.

The network architecture is based on VGGNet and introduces six side outputs, each linked to a different convolutional layer of VGGNet. The innovative aspect is the addition of short connections, which facilitate the flow of high-level semantic information from deeper layers to shallower layers and vice versa, refining the predictions at each stage. Specifically, deeper side outputs can locate salient objects, while shallower ones capture fine-grained details, resulting in precise and coherent saliency maps.

Methodology and Experimental Results

The proposed network introduces two additional convolutional layers in each side output of the HED architecture, improving feature learning capabilities. Utilizing a class-balanced cross-entropy loss, the network benefits from advanced multi-scale and multi-level feature representations. Furthermore, a fully connected Conditional Random Field (CRF) is employed during the inference stage to enhance spatial coherence and correct potential prediction errors.

Experimental results on five widely used SOD benchmarks (MSRA-B, ECSSD, HKU-IS, PASCALS, and SOD) demonstrate the efficacy of the proposed approach. The network achieves significant improvements in terms of F-measure and mean absolute error (MAE) scores, surpassing existing methods like MDF, DS, DCL, ELD, MC, RFCN, and DHS. Notably, the network processes images rapidly, requiring only 0.08 seconds per image, and achieves state-of-the-art results with commendable efficiency and simplicity.

Ablation Studies and Robustness

Ablation experiments underline the importance of the short connections and additional convolutional layers in each side output. The paper explores various configurations, confirming that more channels or deeper convolutional layers contribute positively to the overall performance. Additionally, experiments with different upsampling operations and data augmentation techniques solidify the robustness of the approach.

The paper also explores the impact of different training datasets on performance, revealing that larger or more complex datasets do not always guarantee better results. Nonetheless, the proposed method achieves consistent performance gains across multiple datasets, proving its generalizability and robustness.

Implications and Future Developments

The implications of this research span both practical and theoretical domains. Practically, the improved SOD method can enhance various computer vision applications such as image segmentation, content-aware editing, and visual tracking. Theoretically, the paper reinforces the significance of multi-scale and multi-level feature integration, paving the way for more advanced architectures.

Future developments could focus on addressing the failure cases identified in the paper, possibly by integrating segment-level information or leveraging more powerful training data. Expanding the approach to incorporate more complex scenes and transparent objects could further solidify the network's applicability in real-world scenarios.

Conclusion

The proposed deeply supervised network with short connections exemplifies a significant advancement in salient object detection. By effectively combining multi-level and multi-scale features, the network achieves high precision in diverse and challenging scenarios, setting a new benchmark in the field. The paper provides a detailed analysis and a unified training set framework, promising a fair benchmarking environment for future research in salient object detection.