Deep Learning for Fine-Grained Image Analysis: A Survey (1907.03069v1)

Abstract: Computer vision (CV) is the process of using machines to understand and analyze imagery, which is an integral branch of artificial intelligence. Among various research areas of CV, fine-grained image analysis (FGIA) is a longstanding and fundamental problem, and has become ubiquitous in diverse real-world applications. The task of FGIA targets analyzing visual objects from subordinate categories, \eg, species of birds or models of cars. The small inter-class variations and the large intra-class variations caused by the fine-grained nature makes it a challenging problem. During the booming of deep learning, recent years have witnessed remarkable progress of FGIA using deep learning techniques. In this paper, we aim to give a survey on recent advances of deep learning based FGIA techniques in a systematic way. Specifically, we organize the existing studies of FGIA techniques into three major categories: fine-grained image recognition, fine-grained image retrieval and fine-grained image generation. In addition, we also cover some other important issues of FGIA, such as publicly available benchmark datasets and its related domain specific applications. Finally, we conclude this survey by highlighting several directions and open problems which need be further explored by the community in the future.

• The paper surveys deep learning techniques for fine-grained image analysis across recognition, retrieval, and generation tasks, highlighting key challenges and benchmark datasets.
• It categorizes recognition approaches into localization-classification subnetworks, end-to-end feature encoding, and external information paradigms to address subtle inter-class variations.
• The review outlines promising future directions such as AutoML, few-shot learning, hashing methods, and addressing real-world adaptation challenges.

Deep Learning Advances in Fine-Grained Image Analysis

This survey paper provides a comprehensive overview of recent advances in fine-grained image analysis (FGIA) using deep learning techniques. The paper categorizes FGIA tasks into fine-grained image recognition, fine-grained image retrieval, and fine-grained image generation, and discusses domain-specific applications, benchmark datasets, and future research directions.

FGIA Problem and Challenges

FGIA deals with analyzing visual objects from subordinate categories, such as species of birds or models of cars. The key challenge lies in the small inter-class variations and large intra-class variations. Generic image analysis distinguishes between coarse categories (e.g., birds, dogs, oranges) with significant visual differences. FGIA, however, requires capturing subtle differences within a meta-category, such as distinguishing between different breeds of dogs (e.g., Husky, Samoyed, Alaska) by analyzing minute variations in features like ears, noses, and tails. As shown in Figure 1, the visual variance of different species of Tern can be subtle.

Benchmark Datasets for FGIA

The paper reviews commonly used fine-grained benchmark datasets, including:

• Oxford Flower: A flower dataset with 102 categories and associated captions [Flowers08].
• CUB200-2011: A popular bird dataset with bounding box, part annotations, attribute labels, and text descriptions [WahCUB200_2011].
• Stanford Dog: A dog dataset with bounding box annotations [Khosla11stanforddogs].
• Stanford Car: A car dataset with bounding box annotations [cars].
• FGVC Aircraft: An aircraft dataset with bounding box annotations and hierarchical labels [airplanes].
• Birdsnap: A bird dataset with bounding box, part annotations, and attribute labels [Birdsnap14].
• Fru92: A fruit dataset with hierarchical labels [vegfru].
• Veg200: A vegetable dataset with hierarchical labels [vegfru].
• iNat2017: A large-scale dataset of plants and animals with bounding box annotations and hierarchical labels [inat2017].
• RPC: A retail product dataset with bounding box annotations and hierarchical labels [rpc].

The CUB200-2011 dataset is particularly popular for evaluating FGIA approaches. Recent datasets like iNat2017 and RPC present new challenges with large-scale data, hierarchical structures, domain gaps, and long-tail distributions, reflecting real-world complexity.

Fine-Grained Image Recognition Paradigms

The paper organizes fine-grained recognition approaches into three paradigms:

1. Localization-Classification Subnetworks: These methods use a localization subnetwork to identify key parts of objects, followed by a classification subnetwork for recognition. Earlier works relied on dense part annotations, while recent techniques utilize image labels to achieve accurate part localization via attention mechanisms and multi-stage strategies.
2. End-to-End Feature Encoding: This paradigm focuses on learning discriminative feature representations directly using deep models. Bilinear CNNs [TsungYu15ICCV] are representative, encoding higher-order statistics of convolutional activations. Subsequent works address the high dimensionality of bilinear features using tensor sketching and specific loss functions.
3. External Information: This paradigm incorporates external information like web data, multi-modal data, or human-computer interactions. Web data is used to augment training data, addressing the scarcity of labeled fine-grained images [bohancvpr17, xiaxiaoaaai19]. Multi-modal data, such as text descriptions and knowledge graphs, is leveraged to improve fine-grained recognition accuracy [fgtextcvpr2016, yuxinpengcvpr2017]. Human-in-the-loop systems combine human and machine intelligence for iterative recognition [yincvpr16, wisdomtpami16].

Fine-Grained Image Retrieval

Fine-grained image retrieval aims to retrieve images of the same sub-category as a query image. Unlike generic image retrieval, it focuses on subtle differences within similar objects. Early deep learning approaches used pre-trained CNN models to select meaningful deep descriptors [Wei16scda]. Recent methods explore supervised metric learning with novel loss functions and weakly-supervised localization modules [xiawuijcai18, xiawuaaai19].

Fine-Grained Image Generation

Fine-grained image generation synthesizes realistic images within specific fine-grained categories using deep generative models like GANs [gan14nips]. CVAE-GAN [CVAEiccv17] combines a variational auto-encoder with a generative adversarial network to model images as compositions of labels and latent attributes. Generating images from text descriptions [AttnGANcvpr18] has gained popularity, using attention mechanisms to synthesize fine-grained details based on relevant words.

Domain-Specific Applications

Deep learning based FGIA techniques are applied in diverse domains such as clothes/shoes retrieval [sketchretrievaliccv17], fashion image recognition [deepfashion16], and product recognition [rpc]. Face identification and person/vehicle re-identification are also considered instances of fine-grained recognition at different granularity levels.

Future Directions

The survey concludes by highlighting potential research directions:

• Automatic Fine-Grained Models: Using AutoML [automlnips] and NAS [nassurvey] to develop tailor-made deep models for FGIA.
• Fine-Grained Few-Shot Learning: Developing methods that can learn new fine-grained concepts from very few examples [pcmFSFG].
• Fine-Grained Hashing: Exploring hashing techniques for efficient large-scale fine-grained data retrieval [surveyhashtpami, wujunhashingijcai16].
• Fine-Grained Analysis in Realistic Settings: Addressing challenges like domain adaptation, knowledge transfer, long-tailed distributions, and resource constraints in real-world applications.

Conclusion

This survey provides a comprehensive overview of deep learning based FGIA techniques, highlighting recent advances, challenges, and future research directions. Despite significant progress, many unsolved problems remain, offering opportunities for further research and application development in this field.