The Application of Two-level Attention Models in Deep Convolutional Neural Network for Fine-grained Image Classification (1411.6447v1)

Abstract: Fine-grained classification is challenging because categories can only be discriminated by subtle and local differences. Variances in the pose, scale or rotation usually make the problem more difficult. Most fine-grained classification systems follow the pipeline of finding foreground object or object parts (where) to extract discriminative features (what). In this paper, we propose to apply visual attention to fine-grained classification task using deep neural network. Our pipeline integrates three types of attention: the bottom-up attention that propose candidate patches, the object-level top-down attention that selects relevant patches to a certain object, and the part-level top-down attention that localizes discriminative parts. We combine these attentions to train domain-specific deep nets, then use it to improve both the what and where aspects. Importantly, we avoid using expensive annotations like bounding box or part information from end-to-end. The weak supervision constraint makes our work easier to generalize. We have verified the effectiveness of the method on the subsets of ILSVRC2012 dataset and CUB200_2011 dataset. Our pipeline delivered significant improvements and achieved the best accuracy under the weakest supervision condition. The performance is competitive against other methods that rely on additional annotations.

Two-Level Attention Models in Deep Convolutional Networks for Fine-Grained Image Classification

The paper "The Application of Two-level Attention Models in Deep Convolutional Neural Network for Fine-grained Image Classification" explores the challenges of fine-grained classification, which necessitates distinguishing between categories defined by subtle differences. To overcome these challenges, the authors have proposed a sophisticated method leveraging a deep convolutional neural network (CNN) combined with two-level attention mechanisms: object-level and part-level.

Overview of the Problem and Method

Fine-grained classification is inherently challenging due to substantial intra-class variance and minimal inter-class variance. Categories such as specific bird species or dog breeds can only be distinguished by examining local and subtle features, which can be heavily impacted by variations in pose, scale, and rotation. Conventional fine-grained classification systems typically involve first identifying the foreground object or parts (where) and then extracting discriminative features (what).

The proposed method integrates three types of attention: bottom-up attention to propose candidate patches, object-level top-down attention to select relevant patches, and part-level top-down attention to localize discriminative parts. The pipeline utilizes these attention mechanisms to train domain-specific deep networks, ultimately improving both the selection of areas (where) and feature extraction (what). A significant advantage is that this method avoids the use of expensive annotations like bounding boxes or part information, thus generalizing under weak supervision.

Pipeline and Methodology

The method is organized into two primary stages of attention:

1. Object-Level Attention:
   • Patch Selection: A CNN trained on the ILSVRC2012 1K categories is converted to a FilterNet, which selects patches relevant to a basic-level category.
   • DomainNet Training: The filtered patches are used to train another CNN, known as DomainNet, which focuses on the relevant domain (e.g., dog breeds).
2. Part-Level Attention:
   • Part Detector: The trained DomainNet exhibits clustering patterns in mid-layer activations, with certain neurons responding to specific parts. The authors employ spectral clustering to detect parts using these neuron clusters.
   • Part-Based Classifier: The patches selected by the part detector are used to train a Support Vector Machine (SVM) classifier, which focuses on these discriminative parts.

These two attentions are then fused to provide a final prediction by combining both object-level and part-level insights.

Experimental Results

The effectiveness of this method was validated on subsets of the ILSVRC2012 dataset (dogs and birds) and the CUB200-2011 dataset. Key findings include:

• On the ILSVRC2012 datasets, the method achieved a top-1 error rate reduction from 40.1% to 28.1% for the dog subset and from 21.1% to 11.0% for the bird subset, demonstrating significant improvements under weak supervision.
• On the CUB200-2011 dataset, under the weakest supervision conditions, the method achieved 69.7% accuracy, competitive with other methods that rely on stronger annotations. When utilizing VGGNet for feature extraction, accuracy improved to 77.9%.

Implications and Future Directions

This work illustrates the potential for improving fine-grained classification using an attention mechanism that requires minimal supervision. By leveraging domain-specific and part-specific features learned through CNNs, the method provides a robust solution to identifying subtle differences in categories.

The paper also opens several avenues for future research:

• Enhanced Part-Level Attention: Further optimizing the part-detection capability by considering multiple layers and combining their insights could mitigate ambiguities inherent to single-layer clustering.
• Integration of Object and Part-Level Attention: Developing a more seamless integration process for the two attention mechanisms might result in more effective pose normalization and enhanced feature representation.

Conclusion

The integration of bidirectional attention models with deep CNNs presents a promising approach to fine-grained classification by effectively highlighting and utilizing domain-relevant and discriminative features. The method's efficiency under weak supervision conditions suggests its viability for scalable, real-world applications without necessitating exhaustive manual annotations.