Hierarchical Open-vocabulary Universal Image Segmentation (2307.00764v2)

Abstract: Open-vocabulary image segmentation aims to partition an image into semantic regions according to arbitrary text descriptions. However, complex visual scenes can be naturally decomposed into simpler parts and abstracted at multiple levels of granularity, introducing inherent segmentation ambiguity. Unlike existing methods that typically sidestep this ambiguity and treat it as an external factor, our approach actively incorporates a hierarchical representation encompassing different semantic-levels into the learning process. We propose a decoupled text-image fusion mechanism and representation learning modules for both "things" and "stuff". Additionally, we systematically examine the differences that exist in the textual and visual features between these types of categories. Our resulting model, named HIPIE, tackles HIerarchical, oPen-vocabulary, and unIvErsal segmentation tasks within a unified framework. Benchmarked on over 40 datasets, e.g., ADE20K, COCO, Pascal-VOC Part, RefCOCO/RefCOCOg, ODinW and SeginW, HIPIE achieves the state-of-the-art results at various levels of image comprehension, including semantic-level (e.g., semantic segmentation), instance-level (e.g., panoptic/referring segmentation and object detection), as well as part-level (e.g., part/subpart segmentation) tasks. Our code is released at https://github.com/berkeley-hipie/HIPIE.

• The paper introduces HIPIE, a unified framework that integrates semantic, instance, and part segmentation to achieve state-of-the-art performance.
• It employs a hierarchical representation with decoupled text-image fusion to distinctly handle 'things' and 'stuff' based on their visual features.
• The innovative training strategy uses hierarchical prompts for open-vocabulary predictions, enabling effective segmentation of unseen part classes.

Overview of Hierarchical Open-vocabulary Universal Image Segmentation

In the paper titled "Hierarchical Open-vocabulary Universal Image Segmentation," the authors present a novel approach to open-vocabulary image segmentation, introducing a unified model named HIPIE (HIerarchical, oPen-vocabulary, and unIvErsal segmentation) that addresses various segmentation tasks. These tasks span from semantic-level segmentation, instance-level segmentation to part-level segmentation, under a single cohesive framework. The model showcases nuanced adaptability, achieving commendable performance across a wide array of benchmarks.

Key Contributions

1. Hierarchical Representation: Unlike existing segmentation methodologies that often bypass inherent segmentation ambiguities due to varied granularity, HIPIE embraces these complexities by incorporating a hierarchical representation. This design decisively incorporates different semantic levels during the learning process, which enables the model to segment visual scenes from a multi-level granularity perspective.
2. Decoupled Text-Image Fusion: The authors propose a decoupled mechanism in text-image fusion and representation learning, specifically distinguishing between "things" (countable, well-defined objects) and "stuff" (unstructured, mostly background elements). They determine that "stuff" exhibits higher visual and textual feature similarity, suggesting distinct handling in the learning process for each category.
3. Unified Framework for Diverse Segmentation Tasks: HIPIE achieves state-of-the-art (SOTA) results on over 40 datasets, underscoring its ability to handle a variety of segmentation tasks, including semantic segmentation, panoptic/referring segmentation, object detection, and part/subpart segmentation. This diversity demonstrates the model’s robustness in both semantic comprehension and task execution.
4. Novelty in Training and Inference Techniques: The model introduces innovative training techniques for part segmentation, using hierarchical prompts that allow for open-vocabulary predictions. During training, it supervises both part and instance labels, enabling the inference stage to seamlessly integrate instance, part, and subpart segmentations through distinct forward passes. This separation allows for the generation of segmentations across unseen part classes by creative prompting.

Strong Numerical Results

HIPIE’s effectiveness is highlighted by its superior quantitative performance. On tasks such as open-vocabulary panoptic segmentation, instance segmentation, and hierarchical segmentation, HIPIE consistently outperforms competing methods. For example, on MSCOCO and ADE20K datasets, it set new SOTA bars in several evaluation metrics like PQ, AP for masks and boxes, and mIoU.

Implications and Future Directions

The HIPIE framework presents significant implications for advancing the adaptability and precision of image segmentation models:

• Practical Applications: In domains requiring detailed scene understanding, such as autonomous vehicles or medical imaging, HIPIE offers a potential leap in accuracy and detail by leveraging hierarchical granularity in segmentation.
• Enhanced Model Generalization: By supporting various segmentation tasks without task-specific adjustments, HIPIE reduces the need for separate models, fostering efficient storage and computation in deployment environments such as on-device AI.

In terms of future directions, expanding the training data with even broader semantic coverage or integrating additional high-quality segmentation datasets like SA-1B could further enhance HIPIE’s generalization in recognizing uncommon object classes. Additionally, exploring its application in video tasks, such as object tracking, could extend the versatility of this hierarchical segmentation approach.

This paper’s methodological advances provide a solid foundation for further exploration into universal segmentation models that can comprehensively tackle the segmentation challenges posed by complex, real-world visual scenes.