WeakTr: Exploring Plain Vision Transformer for Weakly-supervised Semantic Segmentation (2304.01184v2)

Abstract: This paper explores the properties of the plain Vision Transformer (ViT) for Weakly-supervised Semantic Segmentation (WSSS). The class activation map (CAM) is of critical importance for understanding a classification network and launching WSSS. We observe that different attention heads of ViT focus on different image areas. Thus a novel weight-based method is proposed to end-to-end estimate the importance of attention heads, while the self-attention maps are adaptively fused for high-quality CAM results that tend to have more complete objects. Besides, we propose a ViT-based gradient clipping decoder for online retraining with the CAM results to complete the WSSS task. We name this plain Transformer-based Weakly-supervised learning framework WeakTr. It achieves the state-of-the-art WSSS performance on standard benchmarks, i.e., 78.4% mIoU on the val set of PASCAL VOC 2012 and 50.3% mIoU on the val set of COCO 2014. Code is available at https://github.com/hustvl/WeakTr.

• The paper introduces a weight-based mechanism to assess ViT attention heads for enhanced class activation maps.
• It proposes a novel ViT-based gradient clipping decoder that refines segmentation maps, achieving 78.4% mIoU on PASCAL VOC and 50.3% on COCO.
• The findings demonstrate the potential of using plain ViTs over CNNs, setting a new state-of-the-art for weakly supervised semantic segmentation.

An Examination of WeakTr: Advancements in Vision Transformers for Weakly-supervised Semantic Segmentation

The paper "WeakTr: Exploring Plain Vision Transformer for Weakly-supervised Semantic Segmentation" by Lianghui Zhu et al. presents a robust paper into applying Vision Transformers (ViTs) to the task of Weakly-supervised Semantic Segmentation (WSSS). Unlike prior approaches that often depend heavily on convolutional neural networks, this research explores employing plain ViTs to enhance the quality of class activation maps (CAMs), which are pivotal for WSSS.

The central innovation introduced is a weight-based mechanism for evaluating and optimizing the contribution of various attention heads within the ViT architecture. By determining the significance of each head, the paper proposes a method to amalgamate self-attention maps dynamically. This results in CAMs that encapsulate object boundaries more completely, which is a notable improvement over conventional mean-sum approaches for attention aggregation.

The method culminates in the proposal of WeakTr, a novel framework integrating a ViT-based gradient clipping decoder. This element of the system facilitates the refinement of segmentation maps during the online retraining phase, thereby streamlining the execution of the WSSS task. The empirical results are compelling; WeakTr achieves a mean Intersection over Union (mIoU) of 78.4% on the validation set of the PASCAL VOC 2012 dataset and 50.3% on the COCO 2014 validation set, establishing a new state of the art within the domain.

Several key aspects merit further discussion:

1. ViT's Attention Mechanism: The exploration highlights that different attention heads in ViTs can discern varied parts of the image, a property harnessed to construct discriminative CAMs. This finding underscores the potential of ViTs in tasks beyond semantic segmentation, where a comprehensive understanding of contextual cues is beneficial.
2. Practical Implementations: The research contributes not only on a theoretical front but also demonstrates practical applications of these insights. The reported benchmarks provide a tangible measure of the advancements, and the availability of the source code enhances reproducibility and encourages further exploration by the research community.
3. Future Directions: Expanding upon these advances, future research could explore the integration of multi-scale features in ViTs and the efficacy of attention-based mechanisms across various weakly-supervised tasks. Moreover, exploring the computational efficiency of WeakTr relative to CNN-based methodologies could yield insights for optimizing large-scale deployments.

In summary, Zhu et al.'s exploration of using plain ViTs for WSSS presents a significant stride in leveraging Transformer architectures for complex image understanding tasks. The work not only sets a new performance benchmark for WSSS but also broadens the scope of ViT applications, prompting further inquiry into their capabilities across broader areas within computer vision. This paper thereby serves as a catalyst for subsequent innovations in supervised learning paradigms and beyond.