Tree Energy Loss: Towards Sparsely Annotated Semantic Segmentation (2203.10739v2)

Abstract: Sparsely annotated semantic segmentation (SASS) aims to train a segmentation network with coarse-grained (i.e., point-, scribble-, and block-wise) supervisions, where only a small proportion of pixels are labeled in each image. In this paper, we propose a novel tree energy loss for SASS by providing semantic guidance for unlabeled pixels. The tree energy loss represents images as minimum spanning trees to model both low-level and high-level pair-wise affinities. By sequentially applying these affinities to the network prediction, soft pseudo labels for unlabeled pixels are generated in a coarse-to-fine manner, achieving dynamic online self-training. The tree energy loss is effective and easy to be incorporated into existing frameworks by combining it with a traditional segmentation loss. Compared with previous SASS methods, our method requires no multistage training strategies, alternating optimization procedures, additional supervised data, or time-consuming post-processing while outperforming them in all SASS settings. Code is available at https://github.com/megvii-research/TreeEnergyLoss.

• The paper introduces Tree Energy Loss (TEL), which leverages minimum spanning trees to model pixel affinities for improved segmentation under sparse supervision.
• It generates hierarchical pseudo labels by integrating both low-level color and high-level semantic features, streamlining training with limited annotations.
• TEL outperforms traditional methods, achieving over 8% mIoU gains on datasets like PASCAL VOC 2012, Cityscapes, and ADE20K in sparse settings.

An Expert Overview of "Tree Energy Loss: Towards Sparsely Annotated Semantic Segmentation"

The paper "Tree Energy Loss: Towards Sparsely Annotated Semantic Segmentation" by Zhiyuan Liang et al. tackles the challenge of semantic segmentation in scenarios where annotations are sparse. The authors propose a novel loss function, Tree Energy Loss (TEL), to enhance the capability of segmentation networks trained with limited labeled data, such as point-, scribble-, and block-wise annotations. This approach is poised to address the labor-intensive nature of generating fully annotated datasets which are commonplace in conventional semantic segmentation tasks.

Technical Contribution: Tree Energy Loss

The key contribution of the paper is the formulation of Tree Energy Loss. TEL leverages the structure-preserving capabilities of minimum spanning trees (MSTs) to model the pairwise affinities between pixels. The process involves:

1. Affinity Modeling: Construction of MSTs using both low-level color information and high-level semantic features. This methodology ensures that pairwise affinities account for both local appearance and global semantic content.
2. Pseudo Label Generation: By applying the tree-based affinities to network predictions, the framework generates pseudo labels for unlabeled pixels. These labels are refined in a hierarchical manner, starting with low-level features and culminating at the high-level features, forming a robust self-training mechanism.
3. Integration with Standard Losses: TEL supplements the traditional segmentation loss, ideally coupling with existing frameworks without necessitating complex multi-stage training or post-processing steps. This simplicity in integration makes TEL a versatile choice for various network architectures.

Numerical Performance

Through comprehensive experiments on datasets such as PASCAL VOC 2012, Cityscapes, and ADE20K, the TEL method demonstrates superior performance in sparsely annotated settings. Notably, the approach outperforms existing methods while requiring significantly less annotation effort. The superiority of TEL is validated in point-, scribble-, and block-wise settings, showing enhancements over baselines by margins often exceeding 8% in mIoU scores.

Implications and Future Directions

The implications of TEL are notable both in theory and practice:

• Theoretical Implications: TEL’s usage of MSTs bridges low-level and high-level information seamlessly, offering a new perspective on how tree-based models can be utilized in deep learning. It also proposes a significant stride in weakly supervised learning paradigms, indicating potential applications in other domains with sparse supervision.
• Practical Applications: By reducing reliance on fully labeled data, TEL holds promise in domains where data labeling is expensive or infeasible, such as medical imaging or satellite imagery. The efficiency of TEL in sparse settings encourages further exploration into cost-effective labeling strategies.

Looking forward, research could focus on optimizing the TEL approach further by incorporating advanced self-training mechanisms or exploring its application with other tree structures and graph models. Additionally, exploring TEL's adaptability with evolving neural architectures could provide insights into scalable models for large-scale deployment in diverse conditions.

In conclusion, the paper presents a well-structured approach to a prevalent challenge in computer vision, efficiently utilizing sparse annotations to achieve high-quality segmentation. Its contributions are both solid in their scientific formulation and promising in real-world applications.