Rethinking Semantic Segmentation: A Prototype View (2203.15102v2)

Abstract: Prevalent semantic segmentation solutions, despite their different network designs (FCN based or attention based) and mask decoding strategies (parametric softmax based or pixel-query based), can be placed in one category, by considering the softmax weights or query vectors as learnable class prototypes. In light of this prototype view, this study uncovers several limitations of such parametric segmentation regime, and proposes a nonparametric alternative based on non-learnable prototypes. Instead of prior methods learning a single weight/query vector for each class in a fully parametric manner, our model represents each class as a set of non-learnable prototypes, relying solely on the mean features of several training pixels within that class. The dense prediction is thus achieved by nonparametric nearest prototype retrieving. This allows our model to directly shape the pixel embedding space, by optimizing the arrangement between embedded pixels and anchored prototypes. It is able to handle arbitrary number of classes with a constant amount of learnable parameters. We empirically show that, with FCN based and attention based segmentation models (i.e., HRNet, Swin, SegFormer) and backbones (i.e., ResNet, HRNet, Swin, MiT), our nonparametric framework yields compelling results over several datasets (i.e., ADE20K, Cityscapes, COCO-Stuff), and performs well in the large-vocabulary situation. We expect this work will provoke a rethink of the current de facto semantic segmentation model design.

• The paper introduces a nonparametric segmentation model that uses non-learnable prototypes derived from mean pixel features to redefine class representations.
• It demonstrates a prototype-based classification strategy that maps pixel embeddings to the nearest prototype, enhancing scalability for large-vocabulary tasks.
• Experimental results show mIoU improvements up to 1.2 percentage points on benchmarks such as ADE20K, Cityscapes, and COCO-Stuff, emphasizing robust performance.

Insights into Nonparametric Semantic Segmentation: A Prototype-Based Perspective

This paper proposes a novel approach to semantic segmentation by introducing a nonparametric framework that uses non-learnable prototypes to redefine how class representations are formed and utilized in segmentation models. At the core of this approach is a departure from traditional parametric methods, which actively learn class-specific weights or query vectors for pixel-wise prediction. Instead, this paper leverages a prototype-based paradigm, where dense predictions are achieved via a nearest prototype retrieval strategy.

Key Contributions

The paper outlines several critical contributions to the field of semantic segmentation:

1. Non-Learnable Prototype Representation: Unlike existing methods that represent each class using a single learned vector, this framework introduces a set of non-learnable prototypes determined by the mean features of several training pixels within a class. This allocation allows for handling an arbitrary number of classes with a constant amount of learnable parameters.
2. Prototype-Based Classification: By viewing classes as sets of prototypes, the model shapes the pixel embedding space without directly relying on parametric assumptions. The classification is achieved by mapping embedded pixels to the nearest class prototype.
3. Scalability and Flexibility: The nonparametric approach is inherently scalable for large-vocabulary segmentation tasks, effectively managing datasets with numerous classes efficiently by eschewing a proportional increase in parameters.

Experimental Findings

The proposed model demonstrates compelling efficacy across several benchmark datasets, including ADE20K, Cityscapes, and COCO-Stuff, and fares particularly well in large-vocabulary settings. Empirical results show significant improvements in mean Intersection over Union (mIoU) scores compared to various baseline models across traditional and transformer-based architectures. For instance, consistent mIoU improvements of up to 1.2 percentage points were observed, highlighting the potential of nonparametric strategies in delivering enhanced performance.

Discussion and Implications

The research strongly suggests that prototype-based learning can provide a robust alternative to traditional segmentation models. It potentially offers a pathway to more interpretable and generalizable models by reducing reliance on heavily parametric structures. Moreover, it opens up avenues for further investigation, including the integration of non-learnable prototype strategies with unsupervised learning techniques or enhancing model interpretability through prototypes that resemble real observations.

Future Directions

Moving forward, the transition to nonparametric frameworks could bring about more resilient models, especially in diverse and dynamic environments typical of real-world applications such as autonomous vehicles or robotics. Moreover, bridging the gap between image-wise classification and pixel-wise segmentation through unified prototype schemes could lead to more cohesive learning paradigms.

This exploration into a prototype view for semantic segmentation challenges existing norms, encouraging the community to reconsider prevailing methodologies and embrace more flexible, data-driven paradigm shifts to enhance both performance and understanding in semantic segmentation tasks.