Sparse Object-level Supervision for Instance Segmentation with Pixel Embeddings (2103.14572v3)

Abstract: Most state-of-the-art instance segmentation methods have to be trained on densely annotated images. While difficult in general, this requirement is especially daunting for biomedical images, where domain expertise is often required for annotation and no large public data collections are available for pre-training. We propose to address the dense annotation bottleneck by introducing a proposal-free segmentation approach based on non-spatial embeddings, which exploits the structure of the learned embedding space to extract individual instances in a differentiable way. The segmentation loss can then be applied directly to instances and the overall pipeline can be trained in a fully- or weakly supervised manner. We consider the challenging case of positive-unlabeled supervision, where a novel self-supervised consistency loss is introduced for the unlabeled parts of the training data. We evaluate the proposed method on 2D and 3D segmentation problems in different microscopy modalities as well as on the Cityscapes and CVPPP instance segmentation benchmarks, achieving state-of-the-art results on the latter. The code is available at: https://github.com/kreshuklab/spoco

• The paper’s main contribution is a framework that reduces dense annotation needs by leveraging sparse, object-level supervision for instance segmentation.
• It employs a novel differentiable instance selection method with a self-supervised consistency loss to enhance segmentation accuracy under positive-unlabeled settings.
• Experimental results on benchmarks like CVPPP and Cityscapes demonstrate that this approach outperforms traditional methods in both fully and weakly supervised scenarios.

Sparse Object-level Supervision for Instance Segmentation with Pixel Embeddings

In their research, Wolny et al. address the challenge of instance segmentation in computer vision, particularly within the domain of biomedical imaging where dense annotations are arduous to procure. Most contemporary instance segmentation methods rely on heavily annotated datasets for training, which is demanding and resource-intensive, especially for biomedical images requiring domain expertise.

The authors introduce an innovative approach that bypasses the dense annotation requirement by utilizing a proposal-free segmentation method predicated on non-spatial embeddings. This methodology leverages the structure of the learned embedding space to differentiate individual instances in a differentiable manner, enabling the application of segmentation loss directly at the instance level. A significant advancement presented in this work is the capability to train the model in both fully- and weakly-supervised settings.

Focusing on the complex scenario of positive-unlabeled (PU) supervision, the paper introduces a self-supervised consistency loss for unlabeled training data regions, enhancing the efficiency and accuracy of the segmentation. This approach is experimentally applied to both 2D and 3D segmentation challenges in various microscopy modalities and benchmarks such as Cityscapes and CVPPP, with state-of-the-art results achieved on the latter.

Contributions and Methodology

1. Sparse Supervision Framework: The paper's principal contribution is a framework that reduces the dependency on comprehensive annotations by incorporating a sparse object-level supervision strategy, particularly suitable for positive-unlabeled configurations. This reduces the labeling burden significantly.
2. Differentiable Instance Selection: A novel differentiable method for selecting instances from non-spatial embeddings is introduced, allowing instance-level loss application and optimizing segmentation accuracy during training.
3. Embedding Consistency Loss: A consistency loss is employed, drawing inspiration from contrastive learning, to ensure that embeddings remain coherent across augmented views of the input, even within unlabeled regions. This consistency loss reinforces the model's generalization capacity across sub-domains without exhaustive labeling.
4. Graph-Based Clustering for Final Segmentation: The embedding-based approach is augmented by a graph-based partitioning method to efficiently convert pixel embeddings to final instances, demonstrating flexibility and speed improvements.

Experimental Results

The proposed framework was rigorously evaluated across multiple datasets:

• CVPPP Dataset: The framework surpassed existing benchmarks by achieving superior Symmetric Best Dice scores, demonstrating its efficacy in handling variances in natural images.
• Cityscapes: Even in urban scene understanding, the methodology outperformed traditional discriminative loss frameworks, especially in a semi-supervised setting.
• Microscopy Data: Both light and electron microscopy data demonstrated significant performance improvements, particularly when employing weakly supervised settings, attesting to the approach's adaptability in varying imaging conditions.
• Transfer Learning: The framework proved adept at transfer learning, as seen from its application in moving from source to target biomedical domains with minimal additional annotations.

Implications and Future Work

This paper's contributions are crucial for advancing instance segmentation, particularly in fields where generating dense annotations are impractical or limited by resource constraints. By empowering AI models to learn from sparsely annotated data, this work paves the way for more efficient AI deployment across sectors like medical imaging, where it is not feasible to annotate every instance comprehensively.

Moving forward, the authors suggest exploring fully self-supervised pre-training paradigms via extended augmentation schemes, aiming to further diminish the dependency on labeled data and enhance the versatility of AI systems in diverse application areas. This work signifies a progressive step towards making AI more adaptive and less reliant on exhaustive data labeling, particularly within specialized domains like biomedicine.