Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Ugly (1707.00600v4)

Abstract: Due to the importance of zero-shot learning, i.e. classifying images where there is a lack of labeled training data, the number of proposed approaches has recently increased steadily. We argue that it is time to take a step back and to analyze the status quo of the area. The purpose of this paper is three-fold. First, given the fact that there is no agreed upon zero-shot learning benchmark, we first define a new benchmark by unifying both the evaluation protocols and data splits of publicly available datasets used for this task. This is an important contribution as published results are often not comparable and sometimes even flawed due to, e.g. pre-training on zero-shot test classes. Moreover, we propose a new zero-shot learning dataset, the Animals with Attributes 2 (AWA2) dataset which we make publicly available both in terms of image features and the images themselves. Second, we compare and analyze a significant number of the state-of-the-art methods in depth, both in the classic zero-shot setting but also in the more realistic generalized zero-shot setting. Finally, we discuss in detail the limitations of the current status of the area which can be taken as a basis for advancing it.

• The paper introduces a unified ZSL benchmark with a new AWA2 dataset and standardized evaluation protocols.
• It systematically evaluates diverse ZSL methods, detailing performance differences in classic and generalized settings.
• The study discusses current limitations and offers guidance for future research toward more robust zero-shot learning approaches.

Zero-Shot Learning - A Comprehensive Evaluation of the Good, the Bad and the Ugly

The paper "Zero-Shot Learning - A Comprehensive Evaluation of the Good, the Bad and the Ugly" by Xian et al. offers a thorough analysis of the current state of zero-shot learning (ZSL) with the aim of standardizing evaluation protocols and highlighting key insights derived from a wide range of methods applied across multiple datasets.

Summary of Key Contributions

The paper's contributions can be categorized into three main areas:

1. Proposing New Benchmarks and Protocols:
   • Recognizing the absence of a standardized ZSL benchmark, it defines a unified benchmark protocol. This includes consistent evaluation protocols and data splits across publicly available datasets used for ZSL tasks.
   • Introduces a new dataset, Animals with Attributes 2 (AWA2), to replace the original AWA1 dataset with analogous labels but publicly available images.
2. Comprehensive Evaluation of Methods:
   • Systematic comparison and in-depth evaluation of numerous state-of-the-art ZSL methods. These include classic zero-shot learning (ZSL) settings and generalized zero-shot learning (GZSL) settings.
   • Statistical and robustness tests across various methods and datasets, including detailed analysis and ranking based on results.
3. Discussion of Limitations and Future Directions:
   • Analyzes the limitations of the current status quo, providing a basis for further advancements in the field.
   • Highlights the impact of flawed evaluation protocols and the necessity for tuning hyperparameters on a validation class split that is distinct from training classes.

Methods Evaluated

The research evaluates a diverse array of ZSL frameworks which are categorized into distinct approaches:

1. Linear Compatibility Learning Frameworks:
   • ALE, DEVISE, SJE, and ESZSL learn bi-linear compatibility functions between visual and semantic spaces.
   • SAE incorporates reconstruction constraints in compatibility learning, demonstrated by its superior performance in certain cases.
2. Nonlinear Compatibility Learning Frameworks:
   • LATEM involves piecewise linear compatibility, optimizing it through latent variable settings.
   • CMT utilizes nonlinear multi-layer neural networks for learning to map images into a semantic space.
3. Attribute Classifier Learning:
   • Methods like DAP and IAP predict attributes first and then infer class labels, showing inferior performance compared to compatibility learning frameworks.
4. Hybrid Models:
   • Models such as SSE, CONSE, and SYNC blend elements of different learning approaches, often embedding images and attributes into a common intermediate space. SYNC outperforms others in specific settings like large-scale ImageNet due to its effective handling of hierarchical embeddings and robustness.
5. Generative Models:
   • GFZSL and its transductive variant GFZSL-tran leverage Gaussian class-conditional distributions and EM-based transductive learning, showing potential in scenarios where auxiliary information for unseen classes is necessary.

Evaluation Insights

Benchmarks and Dataset Splits:

• The introduction of AWA2 helps to mitigate the issues with AWA1 which did not have publicly available images.
• The research also proposes new dataset splits ensuring none of the test classes overlap with classes used for pre-training deep learning models.

Performance and Robustness:

• The evaluation illustrates that models like GFZSL and ALE consistently outperform others on standardized tasks.
• In the generalized ZSL setting, methods optimized for both seen and unseen classes (such as ALE and DEVISE) show superior performance using the harmonic mean as an evaluation metric.

Theoretical and Practical Implications

The paper underscores the necessity for standardized evaluation protocols in ZSL and highlights that max-margin compatibility learning frameworks generally offer more robust performance across diverse datasets. Additionally, the results imply that future research should focus on generative approaches that can handle more realistic, less restrictive ZSL settings and improve generalized learning capabilities.

Conclusion

By providing a comprehensive evaluation, the paper illuminates various strengths and weaknesses of existing zero-shot learning methodologies. Additionally, the introduction of the AWA2 dataset and new benchmark protocols represent significant steps toward more consistent and reliable evaluations in the ZSL domain. Future research is expected to build on these findings, potentially leading to more advanced models capable of better handling the complexities inherent in zero-shot learning tasks.