Semantic Relation Reasoning for Shot-Stable Few-Shot Object Detection (2103.01903v2)

Abstract: Few-shot object detection is an imperative and long-lasting problem due to the inherent long-tail distribution of real-world data. Its performance is largely affected by the data scarcity of novel classes. But the semantic relation between the novel classes and the base classes is constant regardless of the data availability. In this work, we investigate utilizing this semantic relation together with the visual information and introduce explicit relation reasoning into the learning of novel object detection. Specifically, we represent each class concept by a semantic embedding learned from a large corpus of text. The detector is trained to project the image representations of objects into this embedding space. We also identify the problems of trivially using the raw embeddings with a heuristic knowledge graph and propose to augment the embeddings with a dynamic relation graph. As a result, our few-shot detector, termed SRR-FSD, is robust and stable to the variation of shots of novel objects. Experiments show that SRR-FSD can achieve competitive results at higher shots, and more importantly, a significantly better performance given both lower explicit and implicit shots. The benchmark protocol with implicit shots removed from the pretrained classification dataset can serve as a more realistic setting for future research.

• The paper introduces Semantic Relation Reasoning (SRR) to improve few-shot object detection performance by integrating semantic and visual data.
• It uses dynamic relation graphs to enhance semantic embeddings and bridge the gap between textual and image-based information.
• Experimental results show that SRR-FSD maintains robust detection accuracy even with very limited novel class data.

Semantic Relation Reasoning for Shot-Stable Few-Shot Object Detection

Few-shot object detection has been a persistent challenge due to the inherent long-tail distribution of real-world data, resulting in scarce samples available for novel classes. This scarcity significantly affects the performance of detectors designed to identify objects from these novel classes. In response, the paper proposes a conceptual advancement for few-shot object detection (FSOD) by leveraging semantic relations between base and novel classes, rather than solely depending on visual information.

Approach and Methodology

The paper introduces the concept of Semantic Relation Reasoning (SRR) for FSOD, implemented in the new detector called SRR-FSD. This approach entangles semantic relations with visual data to improve object detection in limited-shot scenarios. The central proposition is to create semantic embeddings of class concepts using LLMs trained on large text corpora. The image representations in the detector are then projected onto this semantic space.

To enhance the semantic embeddings for FSOD, the paper addresses the limitations of using raw embeddings directly by proposing the use of dynamic relation graphs. These graphs are learned from the image data, facilitating the reasoning process that augments the semantic space. This augmentation aims to reduce the bias introduced by the domain gap between vision and language, thus enabling the SRR-FSD to remain robust across varying shot conditions of novel classes.

Key Findings

The SRR-FSD showcases significant improvements over existing FSOD methods, particularly at lower shot levels. Experimental results demonstrate that SRR-FSD achieves competitive results in scenarios with higher numbers of shots, but more notably excels with fewer explicit and implicit shots. Specifically, SRR-FSD maintains its performance effectively even when novel class data is minimal, a critical benefit when encountering rare classes in practical applications.

Furthermore, the paper suggests an alternative evaluation protocol for FSOD, wherein implicit shots of novel classes are deliberately excluded from the pretrained classification dataset. This more realistic setting reveals the robustness of SRR-FSD in scenarios where prior exposure to novel classes is completely removed.

Implications and Future Research

The introduction of semantic relation reasoning in FSOD marks a significant step forward, as it allows detectors to mediate between semantic and visual information to improve shot-stability. Practically, this could lead to better detection of scarce, rare objects, enhancing applications across various domains such as wildlife monitoring, autonomous driving, and other AI-driven tasks.

Theoretically, the dynamic relation graph mechanism offers a promising avenue for bridging linguistic and visual domains, which could lead to advancements in zero-shot and few-shot learning methodologies. Future research could explore the generalizability of these principles to broader AI tasks beyond object detection, including object recognition and context understanding in multitask learning environments.