Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels (2203.03884v2)

Abstract: The crux of semi-supervised semantic segmentation is to assign adequate pseudo-labels to the pixels of unlabeled images. A common practice is to select the highly confident predictions as the pseudo ground-truth, but it leads to a problem that most pixels may be left unused due to their unreliability. We argue that every pixel matters to the model training, even its prediction is ambiguous. Intuitively, an unreliable prediction may get confused among the top classes (i.e., those with the highest probabilities), however, it should be confident about the pixel not belonging to the remaining classes. Hence, such a pixel can be convincingly treated as a negative sample to those most unlikely categories. Based on this insight, we develop an effective pipeline to make sufficient use of unlabeled data. Concretely, we separate reliable and unreliable pixels via the entropy of predictions, push each unreliable pixel to a category-wise queue that consists of negative samples, and manage to train the model with all candidate pixels. Considering the training evolution, where the prediction becomes more and more accurate, we adaptively adjust the threshold for the reliable-unreliable partition. Experimental results on various benchmarks and training settings demonstrate the superiority of our approach over the state-of-the-art alternatives.

• The paper introduces U2PL, a framework that repurposes unreliable pixel predictions as negative samples to boost segmentation performance.
• It employs entropy-based separation and a category-wise negative sample queue for dynamically balancing prediction reliability during training.
• Experimental results on PASCAL VOC 2012 and Cityscapes show significant improvements, especially when labeled data is extremely limited.

Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels

The paper "Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels" addresses the challenge of leveraging unlabeled data in semi-supervised semantic segmentation by proposing a novel approach that utilizes unreliable pseudo-labels. This work contributes a robust framework named U$^2$PL, which significantly enhances model training by effectively utilizing both reliable and unreliable pixel predictions.

Core Contributions

The central thesis of the paper challenges the conventional practice of using only highly confident predictions as pseudo-label ground-truths, where ambiguous predictions are discarded. The authors propose that leveraging these 'unreliable' predictions as negative samples can enhance the training process. This contribution is predicated on the insight that while unreliable predictions might be confused between top probable classes, they can still confidently indicate non-membership to other classes, thus serving as viable negative samples.

The U$^2$PL framework involves several key steps:

1. Entropy-Based Segmentation: Separation of pixel predictions into reliable and unreliable categories using entropy as a metric.
2. Negative Sample Queue: Utilization of a category-wise queue for storing features from unreliable predictions as negative examples, ensuring balanced representation across all classes.
3. Adaptive Threshold Adjustment: A dynamic adjustment method that tunes the threshold between reliable and unreliable predictions over the course of training, in alignment with model accuracy improvements.

Experimental Validation

The efficacy of U$^2$PL was validated on standard benchmarks such as PASCAL VOC 2012 and Cityscapes datasets. The results demonstrated marked enhancements over state-of-the-art semi-supervised methods. In the PASCAL VOC 2012 experiments, U$^2$PL achieved higher mIoU scores significantly across all tested labeled/unlabeled data configurations, particularly excelling under scenarios with extremely limited labeled data (e.g., 1/16 partition). This underscores the potential of U$^2$PL to make superior use of available data even when labeled samples are sparse.

Implications and Future Directions

The introduction of unreliable pseudo-labels into the semi-supervised learning paradigm presents substantial theoretical and practical implications. Theoretically, it encourages a shift in understanding label noise management, suggesting that leveraging rather than discarding ambiguous predictions can lead to improved model training. Practically, this approach can alleviate reliance on large-scale labeled data and reduce annotation costs, a considerable benefit for industries deploying semantic segmentation models.

Future research could explore broader applications of the U$^2$PL methodology beyond semantic segmentation, particularly in other domains suffering from similar data limitations. Additionally, expanding the scope of dynamic adjustment strategies and robustness against diverse noisy conditions could further enhance applicability and performance.

In conclusion, U$^2$PL's introduction of unreliable pseudo-labels coupled with adaptive learning offers a notable advancement for semi-supervised learning frameworks, yielding significant performance improvements and setting a foundation for future research in effectively leveraging all available data.