Semi-Supervised Semantic Segmentation via Adaptive Equalization Learning (2110.05474v1)

Abstract: Due to the limited and even imbalanced data, semi-supervised semantic segmentation tends to have poor performance on some certain categories, e.g., tailed categories in Cityscapes dataset which exhibits a long-tailed label distribution. Existing approaches almost all neglect this problem, and treat categories equally. Some popular approaches such as consistency regularization or pseudo-labeling may even harm the learning of under-performing categories, that the predictions or pseudo labels of these categories could be too inaccurate to guide the learning on the unlabeled data. In this paper, we look into this problem, and propose a novel framework for semi-supervised semantic segmentation, named adaptive equalization learning (AEL). AEL adaptively balances the training of well and badly performed categories, with a confidence bank to dynamically track category-wise performance during training. The confidence bank is leveraged as an indicator to tilt training towards under-performing categories, instantiated in three strategies: 1) adaptive Copy-Paste and CutMix data augmentation approaches which give more chance for under-performing categories to be copied or cut; 2) an adaptive data sampling approach to encourage pixels from under-performing category to be sampled; 3) a simple yet effective re-weighting method to alleviate the training noise raised by pseudo-labeling. Experimentally, AEL outperforms the state-of-the-art methods by a large margin on the Cityscapes and Pascal VOC benchmarks under various data partition protocols. Code is available at https://github.com/hzhupku/SemiSeg-AEL

Insights into Adaptive Equalization Learning for Semi-Supervised Semantic Segmentation

The paper "Semi-Supervised Semantic Segmentation via Adaptive Equalization Learning" introduces a framework aimed at enhancing the performance of semi-supervised semantic segmentation under conditions of data imbalance, particularly in datasets exhibiting a long-tailed label distribution like Cityscapes. The proposed Adaptive Equalization Learning (AEL) framework seeks to address the challenges posed by such imbalances by introducing strategies that focus on under-represented and under-performing categories, providing a comprehensive approach to improving segmentation results using both labeled and unlabeled data.

Core Contributions

The authors identify a critical issue in semi-supervised semantic segmentation, where datasets tend to be imbalanced regarding the distribution of categories. This imbalance is problematic in training scenarios that rely heavily on pseudo-labeling and consistency regularization, potentially leading to degradation in learning effectiveness for under-represented classes.

To remediate this, the paper makes several key contributions:

1. Confidence Bank: This component records category-wise performance dynamically during training and aids in identifying under-performing categories without explicit class frequency information. It informs the other components of AEL by providing a real-time performance metric for each category, improving the model's focus dynamically.
2. Adaptive Data Augmentation: Two novel data augmentation techniques—Adaptive CutMix and Adaptive Copy-Paste—are introduced. These methods intelligently increase the presence of under-performing categories during training. This aspect challenges previous approaches, demonstrating that well-planned augmentation strategies can ameliorate the segmentation bias common in imbalanced datasets.
3. Adaptive Equalization Sampling: This strategy selectively samples more pixels from the under-performing categories based on the confidence scores, ensuring these categories are better represented in the training phase.
4. Dynamic Re-Weighting: By applying this strategic re-weighting method to the learning process, noise from erroneous pseudo-labeling is alleviated, enabling smoother learning curves for less represented classes.

Experimental Evaluation

The experimental setup demonstrates the effectiveness of the proposed AEL framework on the Cityscapes and PASCAL VOC 2012 benchmarks, surpassing existing state-of-the-art results. For instance, on the Cityscapes dataset, AEL achieved mIoU improvements of up to 16.39 percentage points over supervised baselines under reduced data partitions. This significant performance lift highlights AEL's potential in enhancing learning for categories traditionally underserved due to data availability constraints.

Additionally, the experiments extend to a full dataset scenario using Cityscapes coarse data as unlabeled input, revealing that AEL effectively improves even when ample labeled data is present, suggesting robustness to various semi-supervised setups.

Implications and Future Research Directions

The implications of AEL are twofold. Practically, it enables more effective utilization of available annotated datasets while minimizing annotation efforts, critical in real-world applications where data labelling is costly. Theoretically, it pushes the boundary on approaches to handling class imbalance in semi-supervised setups, a persistent issue across various machine learning domains.

Future research can explore further refinements to the confidence bank mechanism to increase its adaptability and precision. Investigating how AEL interacts with different backbone architectures or larger, more diverse datasets such as ADE20K may provide additional insights into its generalizability. Moreover, exploring the integration of AEL techniques into fully supervised learning could yield innovative strategies to handle ever-present class imbalance issues.