CorrMatch: Label Propagation via Correlation Matching for Semi-Supervised Semantic Segmentation (2306.04300v3)

Abstract: This paper presents a simple but performant semi-supervised semantic segmentation approach, called CorrMatch. Previous approaches mostly employ complicated training strategies to leverage unlabeled data but overlook the role of correlation maps in modeling the relationships between pairs of locations. We observe that the correlation maps not only enable clustering pixels of the same category easily but also contain good shape information, which previous works have omitted. Motivated by these, we aim to improve the use efficiency of unlabeled data by designing two novel label propagation strategies. First, we propose to conduct pixel propagation by modeling the pairwise similarities of pixels to spread the high-confidence pixels and dig out more. Then, we perform region propagation to enhance the pseudo labels with accurate class-agnostic masks extracted from the correlation maps. CorrMatch achieves great performance on popular segmentation benchmarks. Taking the DeepLabV3+ with ResNet-101 backbone as our segmentation model, we receive a 76%+ mIoU score on the Pascal VOC 2012 dataset with only 92 annotated images. Code is available at https://github.com/BBBBchan/CorrMatch.

• The paper presents CorrMatch, a novel framework that uses correlation maps for effective pseudo label propagation, achieving competitive mIoU scores on benchmarks.
• It employs pixel and region propagation strategies to spread high-confidence predictions and refine shape details in semi-supervised semantic segmentation.
• The streamlined architecture reduces reliance on extensive labeled data, offering a cost-effective and practical solution for real-world segmentation tasks.

Overview of "CorrMatch: Label Propagation via Correlation Matching for Semi-Supervised Semantic Segmentation"

The paper "CorrMatch: Label Propagation via Correlation Matching for Semi-Supervised Semantic Segmentation" presents a novel approach for improving semi-supervised semantic segmentation, focusing on leveraging correlation maps for label propagation. This paper introduces CorrMatch, a simple yet effective framework intended to enhance the efficiency of utilizing unlabeled data in semantic segmentation tasks, aiming to balance the performance achieved with limited annotated data.

Core Contributions and Methodology

The paper makes a significant contribution by focusing on correlation maps as a mechanism for enhancing pseudo-label generation. CorrMatch capitalizes on the beneficial properties of correlation maps for clustering pixels of the same semantic category and retaining shape information, which prior methods have not fully exploited.

The two principal strategies employed by CorrMatch for label propagation are the pixel propagation and region propagation:

1. Pixel Propagation: This strategy utilizes the pairwise similarities encoded in correlation maps to spread high-confidence predictions across similar pixels. It enhances the segmentation predictions by incorporating widespread similarity information derived from the correlation data, thus enriching the pseudo labels with global semantic consistency.
2. Region Propagation: This strategy exploits local shape information by enhancing pseudo labels using class-agnostic masks extracted from correlation maps. By aligning the most salient class found within these shapes with high-confidence regions, the model achieves more accurate pseudo-label expansion.

The architecture of CorrMatch is notably streamlined compared to previous methods, as it doesn't rely on multiple networks, training stages, or strong augmentation data streams.

Experimental Results

The empirical evaluation demonstrates that CorrMatch surpasses state-of-the-art results on standard benchmarks like Pascal VOC 2012 and Cityscapes. Notably, using DeepLabV3+ with a ResNet-101 backbone, the approach achieves a mean Intersection-over-Union (mIoU) score exceeding 76% on the Pascal VOC 2012 dataset using only 92 labeled images. In various experimental settings, CorrMatch consistently outperforms other contemporary methods that incorporate more complex training strategies.

Significance and Implications

By simplifying the model architecture and enriching pseudo label quality through label propagation, CorrMatch makes a compelling case for an enhanced semi-supervised segmentation paradigm. Its reliance on correlation maps suggests a robust method for leveraging semantic similarity within datasets, paving the way for more efficient and generalizable segmentation frameworks.

The successful implementation of CorrMatch offers several practical implications:

• Efficiency in Labeling: Reducing the dependency on large amounts of labeled data significantly lowers the cost and effort involved in preparing segmentation datasets.
• Real-World Applicability: As semi-supervised learning aligns well with the conditions of real-world scenarios where labeled data is often scarce, CorrMatch is particularly relevant for deployment in settings with limited label availability.

Future Prospects

Future research could explore the application of CorrMatch strategies to other related tasks in computer vision, such as instance segmentation or panoptic segmentation. Additionally, further investigation could be conducted into optimizing correlation map representations or integrating other forms of contextual information to potentially increase the resilience and accuracy of segmentation models in diverse environments.

In summary, CorrMatch stands out as a promising technique in the semi-supervised learning landscape by effectively leveraging correlation maps for enhancing semantic segmentation tasks. It strikes a balance between simplicity and performance, opening several avenues for ongoing and future exploration in AI-driven image processing tasks.