OW-Rep: Open World Object Detection with Instance Representation Learning (2409.16073v2)

Abstract: Open World Object Detection(OWOD) addresses realistic scenarios where unseen object classes emerge, enabling detectors trained on known classes to detect unknown objects and incrementally incorporate the knowledge they provide. While existing OWOD methods primarily focus on detecting unknown objects, they often overlook the rich semantic relationships between detected objects, which are essential for scene understanding and applications in open-world environments (e.g., open-world tracking and novel class discovery). In this paper, we extend the OWOD framework to jointly detect unknown objects and learn semantically rich instance embeddings, enabling the detector to capture fine-grained semantic relationships between instances. To this end, we propose two modules that leverage the rich and generalizable knowledge of Vision Foundation Models(VFM). First, the Unknown Box Refine Module uses instance masks from the Segment Anything Model to accurately localize unknown objects. The Embedding Transfer Module then distills instance-wise semantic similarities from VFM features to the detector's embeddings via a relaxed contrastive loss, enabling the detector to learn a semantically meaningful and generalizable instance feature. Extensive experiments show that our method significantly improves both unknown object detection and instance embedding quality, while also enhancing performance in downstream tasks such as open-world tracking.

• The paper introduces instance representation learning using vision foundation models to enhance detection of novel objects in open-world scenarios.
• It leverages semantic masks from the Segment Anything Model to improve localization accuracy and enrich the feature space.
• Experimental results demonstrate superior performance in object detection and extend usability to tasks like open-world tracking.

The paper "Open-World Object Detection with Instance Representation Learning" addresses the challenge faced by deep learning-based object detectors in recognizing and relating objects that are not part of their training data, a problem humans handle naturally. This challenge is significant in open-world scenarios where these models must detect and understand novel objects and their interrelations.

The authors propose a novel method aimed at improving Open World Object Detection (OWOD) by training an object detector capable of both identifying novel objects and extracting semantically enriched features. Traditional OWOD methods struggle to capture the nuanced relationships between detected objects, which is vital for understanding complex scenes and performing tasks such as class discovery and tracking.

Key contributions of the paper include:

1. Use of Vision Foundation Models (VFM): The approach leverages existing VFMs to enhance the detector's capabilities. Specifically, semantic masks from the Segment Anything Model are utilized to supervise box regression for unknown objects, improving localization accuracy.
2. Instance Representation Learning: The method transfers instance-wise similarities from VFM features to the detector’s instance embeddings. This transfer is pivotal in developing a semantically rich feature space, enhancing the model’s understanding and detection capabilities in an open-world environment.
3. Experimental Validation: Extensive experiments validate the proposed method's ability to learn robust and generalizable features. The results demonstrate superior performance compared to existing OWOD-based feature extraction methods.
4. Application to Additional Tasks: The enhanced features derived from the model also expand its usability to related tasks, particularly open-world tracking. This indicates that the improved feature space not only aids in object detection but also supports diverse applications requiring scene understanding.

Overall, the paper presents a significant advancement in the field of open-world object detection by integrating instance representation learning with VFMs to build object detectors that are more adaptable and effective in real-world, dynamic environments.