Token Contrast for Weakly-Supervised Semantic Segmentation (2303.01267v1)

Abstract: Weakly-Supervised Semantic Segmentation (WSSS) using image-level labels typically utilizes Class Activation Map (CAM) to generate the pseudo labels. Limited by the local structure perception of CNN, CAM usually cannot identify the integral object regions. Though the recent Vision Transformer (ViT) can remedy this flaw, we observe it also brings the over-smoothing issue, \ie, the final patch tokens incline to be uniform. In this work, we propose Token Contrast (ToCo) to address this issue and further explore the virtue of ViT for WSSS. Firstly, motivated by the observation that intermediate layers in ViT can still retain semantic diversity, we designed a Patch Token Contrast module (PTC). PTC supervises the final patch tokens with the pseudo token relations derived from intermediate layers, allowing them to align the semantic regions and thus yield more accurate CAM. Secondly, to further differentiate the low-confidence regions in CAM, we devised a Class Token Contrast module (CTC) inspired by the fact that class tokens in ViT can capture high-level semantics. CTC facilitates the representation consistency between uncertain local regions and global objects by contrasting their class tokens. Experiments on the PASCAL VOC and MS COCO datasets show the proposed ToCo can remarkably surpass other single-stage competitors and achieve comparable performance with state-of-the-art multi-stage methods. Code is available at https://github.com/rulixiang/ToCo.

• The paper introduces Token Contrast, a novel strategy that alleviates over-smoothing in Vision Transformers via patch and class token contrasts.
• It refines semantic segmentation by supervising patch tokens with intermediate layer features to produce more precise CAMs, achieving 71.1% mIoU on VOC.
• The method offers a cost-effective alternative to pixel-level annotations and sets a new benchmark for weakly-supervised segmentation.

Analyzing "Token Contrast for Weakly-Supervised Semantic Segmentation"

The paper, "Token Contrast for Weakly-Supervised Semantic Segmentation," introduces an innovative approach to address a prevalent challenge in Weakly-Supervised Semantic Segmentation (WSSS): the over-smoothing issue inherent in Vision Transformers (ViT). This discussion provides an overview of the method proposed by Ru et al. and evaluates its implications and impact on the field.

In the context of WSSS, leveraging image-level labels is a cost-effective alternative to pixel-wise annotations. Typically, Class Activation Maps (CAM) and their variants are deployed to generate pseudo labels, which essentially guide these models. However, the limited scope of Convolutional Neural Networks (CNNs) often results in a narrow focus on only the most distinctive features, neglecting more extensive object areas. Recent works have engaged ViTs to overcome this pitfall due to their capacity to model global interactions via self-attention mechanisms, but have encountered challenges like the over-smoothing of patch tokens. The paper at hand proposes the Token Contrast (ToCo) methodology to mitigate this issue.

ToCo integrates two key components: the Patch Token Contrast (PTC) and the Class Token Contrast (CTC). The PTC aims to supervise the final layer of patch tokens using knowledge derived from intermediate layers of the ViT, where semantic diversity is still retained. This supervision helps to produce more precise CAM and mitigate token uniformity. In contrast, the CTC exploits the semantic aggregation capability of class tokens to enhance the representation fidelity of uncertain image regions and achieve more accurate CAMs. Specifically, CTC contrasts the class tokens of local crops from these uncertain regions against those of global images to reinforce representation consistency.

The robustness of this approach is evidenced through extensive experimentation, predominantly on the commonly used PASCAL VOC and MS COCO datasets. With remarkable improvements over single-stage methods, ToCo aligns more closely with state-of-the-art multi-stage methods, achieving 71.1% mIoU on the VOC validation set. It also demonstrated significant superiority over ViT-PCM, despite using a similar backbone, underscoring the efficacy of addressing over-smoothing through token contrast.

The implications of this research are manifold. Practically, it offers a refined tool for more efficient and cost-effective semantic segmentation in applications where pixel-level annotation may be infeasible. Theoretically, it broadens the understanding of the interplay between various layers within ViT architectures and their potential exploitation in segmentation tasks. Furthermore, it marks an advancement from CNN-based segmentation models to leveraging transformers, encouraging future exploration into addressing the constraints observed in such transformer models.

Looking ahead, this paper opens pathways to further investigate the potential modifications and enhancements to the current ViT architectures beyond the domain of WSSS. With the indication that intermediate ViT layers preserve necessary semantic features for effective segmentation, a deeper exploration into ViT architectural nuances could yield even more sophisticated models adept at broader visual recognition tasks.

In summary, the paper "Token Contrast for Weakly-Supervised Semantic Segmentation" lays down a crucial step forward in addressing inherent challenges in utilizing ViT for semantic segmentation, setting a benchmark for future explorations in WSSS methodologies.