Exploit Bounding Box Annotations for Multi-label Object Recognition (1504.05843v2)

Abstract: Convolutional neural networks (CNNs) have shown great performance as general feature representations for object recognition applications. However, for multi-label images that contain multiple objects from different categories, scales and locations, global CNN features are not optimal. In this paper, we incorporate local information to enhance the feature discriminative power. In particular, we first extract object proposals from each image. With each image treated as a bag and object proposals extracted from it treated as instances, we transform the multi-label recognition problem into a multi-class multi-instance learning problem. Then, in addition to extracting the typical CNN feature representation from each proposal, we propose to make use of ground-truth bounding box annotations (strong labels) to add another level of local information by using nearest-neighbor relationships of local regions to form a multi-view pipeline. The proposed multi-view multi-instance framework utilizes both weak and strong labels effectively, and more importantly it has the generalization ability to even boost the performance of unseen categories by partial strong labels from other categories. Our framework is extensively compared with state-of-the-art hand-crafted feature based methods and CNN based methods on two multi-label benchmark datasets. The experimental results validate the discriminative power and the generalization ability of the proposed framework. With strong labels, our framework is able to achieve state-of-the-art results in both datasets.

• The paper introduces a novel multi-view multi-instance framework that integrates bounding box annotations ('strong labels') to enhance multi-label object recognition in CNNs.
• Evaluation on PASCAL VOC datasets shows the framework achieves superior performance and state-of-the-art mAP by combining feature and label views.
• Integrating strong labels improves generalization capabilities, even for unseen categories, demonstrating practical benefits for complex visual recognition tasks.

Exploit Bounding Box Annotations for Multi-label Object Recognition

This paper investigates the use of bounding box annotations to enhance multi-label object recognition in convolutional neural networks (CNNs). It introduces a novel multi-view multi-instance framework by integrating local information through bounding box annotations—conceptualized as 'strong labels'—to optimize feature extraction from multi-label images.

Research Context and Methodology

CNNs have proven effective in single-object recognition tasks; however, their performance can be suboptimal in multi-label scenarios where multiple objects with varying scales, locations, and categories are present in a single image. This paper proposes transforming the multi-label recognition problem into a multi-class, multi-instance learning problem. By treating each image as a bag and its object proposals as instances, a more localized processing is enabled, thus allowing the incorporation of spatial configurations and local similarities.

The framework employs two distinct views:

1. Feature View: Proposals are represented using the standard CNN feature extraction approach, where CNN features are generated from object proposals using a pre-trained network.
2. Label View: Bounding box annotations are utilized to construct a large-margin nearest neighbor (LMNN) CNN, emphasizing local spatial relations through a neighborhood encoding technique that uses ground truth objects as a candidate pool.

The novelty lies in the label view, which indirectly utilizes bounding box annotations to enhance the generalization capability, enabling performance improvements even on unseen categories by extrapolating partial strong labels from other categories.

Key Results

In evaluations on the PASCAL VOC 2007 and 2012 datasets, which are benchmarks for multi-label object recognition tasks:

• The proposed framework achieved superior performance compared to several state-of-the-art methods. Utilizing both feature and label views (FeV+LV-20) exhibited significant gains over methods relying solely on traditional CNN feature extraction.
• The framework demonstrated robustness in recognizing unseen categories, validating the generalization capabilities of local spatial encoding through strong labels.
• With a fusion of the very-deep 16-layer CNN model, the results reached state-of-the-art mAP levels, underscoring the efficacy of the multi-view approach.

Implications and Future Directions

The research illustrates that integrating strong labels within a multi-instance, multi-view framework can substantially enhance object recognition in multi-label settings. Practically, leveraging bounding box annotations allows for improved object detection capabilities and a more nuanced understanding of scene compositions. Theoretically, it provides a pathway for extending CNN architectures towards more complex recognition tasks.

Future work may focus on enhancing scalability through proposal selection methods to filter noisy data and improve computational efficiency. Additionally, a fully automated approach to establish candidate pools derived solely from proposals could further reduce dependency on explicitly annotated labels, thus expanding applicability in domains where annotation costs are prohibitive.

The paper contributes meaningfully to the discourse on CNN optimization for complex visual recognition tasks, highlighting valuable intersectional strategies between feature representation and label encoding.