Sparse Instance Activation for Real-Time Instance Segmentation (2203.12827v1)

Abstract: In this paper, we propose a conceptually novel, efficient, and fully convolutional framework for real-time instance segmentation. Previously, most instance segmentation methods heavily rely on object detection and perform mask prediction based on bounding boxes or dense centers. In contrast, we propose a sparse set of instance activation maps, as a new object representation, to highlight informative regions for each foreground object. Then instance-level features are obtained by aggregating features according to the highlighted regions for recognition and segmentation. Moreover, based on bipartite matching, the instance activation maps can predict objects in a one-to-one style, thus avoiding non-maximum suppression (NMS) in post-processing. Owing to the simple yet effective designs with instance activation maps, SparseInst has extremely fast inference speed and achieves 40 FPS and 37.9 AP on the COCO benchmark, which significantly outperforms the counterparts in terms of speed and accuracy. Code and models are available at https://github.com/hustvl/SparseInst.

Sparse Instance Activation for Real-Time Instance Segmentation

The paper presents a novel framework for real-time instance segmentation known as SparseInst. The key innovation lies in the introduction of instance activation maps (IAM), which serve as a new object representation approach, departing from traditional methods that rely heavily on bounding boxes or dense centers. This paper outlines how IAM can effectively highlight informative regions for each object, which are then used for feature aggregation, thereby improving both recognition and segmentation tasks.

The SparseInst framework significantly reduces the need for complex post-processing steps such as non-maximum suppression (NMS), thanks to the one-to-one prediction style enabled by bipartite matching. This simplification results in impressive real-time performance metrics: SparseInst achieves an inference speed of 40 FPS and an average precision (AP) of 37.9 on the COCO benchmark.

Technical Contributions

1. Instance Activation Maps (IAM): IAMs are weighted maps designed to emphasize the most informative parts of objects, thereby facilitating instance-level feature extraction. This approach allows for the efficient segmentation of foreground objects without reliance on bounding box predictions.
2. End-to-End Framework: SparseInst utilizes a fully convolutional framework free from traditional detector dependencies. The architecture comprises a backbone, an encoder to enhance multi-scale representation, and a decoder to compute IAMs, ultimately enabling real-time instance segmentation.
3. Bipartite Matching: By employing this algorithm, SparseInst simplifies label assignment by matching each prediction with a target, thereby supporting IAMs in highlighting individual objects effectively. This also circumvents the need for NMS during inference.
4. Efficient Design: SparseInst is optimized for speed and accuracy, marked by sparse object predictions, single-level outputs, compact network architecture, and the absence of time-consuming post-processing steps.

Experimental Results

SparseInst outperforms existing real-time instance segmentation methods both in speed and accuracy. The experimentation on COCO dataset demonstrates the model's superior performance, setting new standards among state-of-the-art real-time approaches, including YOLACT, YOLACT++, and SOLOv2. SparseInst's architecture ensures robust segmentation results even in complex scenes with minimal latency.

Implications and Future Work

The introduction of IAMs marks a significant step towards more efficient instance segmentation workflows. The ability to directly highlight object regions simplifies the computational processing related to object detection. This efficiency makes SparseInst particularly useful for applications in autonomous driving and robotics, where real-time processing is critical.

Looking towards future advancements, further optimization of classification capabilities may address the residual classification and duplication errors highlighted through TIDE analysis. Additionally, extending this architecture could explore enhancements in computational efficiency and extended applicability across diverse datasets and deployment environments.

Overall, SparseInst provides a compelling demonstration of how instance segmentation can be both simplified and accelerated without compromising on performance, setting a precedent for subsequent innovations in the field.