Deep Long-Tailed Learning: A Survey (2110.04596v2)

Abstract: Deep long-tailed learning, one of the most challenging problems in visual recognition, aims to train well-performing deep models from a large number of images that follow a long-tailed class distribution. In the last decade, deep learning has emerged as a powerful recognition model for learning high-quality image representations and has led to remarkable breakthroughs in generic visual recognition. However, long-tailed class imbalance, a common problem in practical visual recognition tasks, often limits the practicality of deep network based recognition models in real-world applications, since they can be easily biased towards dominant classes and perform poorly on tail classes. To address this problem, a large number of studies have been conducted in recent years, making promising progress in the field of deep long-tailed learning. Considering the rapid evolution of this field, this paper aims to provide a comprehensive survey on recent advances in deep long-tailed learning. To be specific, we group existing deep long-tailed learning studies into three main categories (i.e., class re-balancing, information augmentation and module improvement), and review these methods following this taxonomy in detail. Afterward, we empirically analyze several state-of-the-art methods by evaluating to what extent they address the issue of class imbalance via a newly proposed evaluation metric, i.e., relative accuracy. We conclude the survey by highlighting important applications of deep long-tailed learning and identifying several promising directions for future research.

Deep Long-Tailed Learning: A Survey

Deep long-tailed learning presents a significant challenge in the field of visual recognition due to its inherently imbalanced class distribution. The challenges stem from the fact that in long-tailed datasets, a small number of classes—referred to as head classes—contain a disproportionately large number of samples, while the majority of classes—known as tail classes—are underrepresented. This imbalance often leads to biased deep models that perform well on head classes but poorly on tail classes. Therefore, deep long-tailed learning aims to address these biases and enhance performance across all classes.

This survey systematically categorizes existing methodologies into three main types: class re-balancing, information augmentation, and module improvement. Each category comprises various subtypes, aiming to tackle the imbalance problem from unique angles.

Class Re-balancing

Class re-balancing encompasses strategies, such as re-sampling, class-sensitive learning, and logit adjustment, to rectify class distribution imbalances. Re-sampling, such as class-balanced sampling or progressive-balanced sampling, modifies the sample frequencies in each mini-batch to better represent tail classes. Class-sensitive learning alters loss functions to emphasize tail classes, overcoming their under-representation in gradient updates. Notably, the Balanced Softmax loss and LDAM have demonstrated robust performance by utilizing label frequencies to re-weight losses and margins adaptively. Logit adjustment methods, on the other hand, post-process biased model predictions, using known training label frequencies to offset predicted logits towards a more balanced stance.

Information Augmentation

Incorporating external knowledge or enriching existing data is at the core of information augmentation methods. Techniques in this category include transfer learning and data augmentation. Transfer learning leverages pre-trained models and knowledge distillation to transfer insights from head to tail classes. Self-supervised pre-training (SSP) and methods like M2m exemplify this category by enhancing tail data representativeness. Meanwhile, data augmentation, particularly methods like Mixup, aims to artificially increase the variety of tail-class samples by creating synthetic data points, effectively countering the data scarcity in tail classes.

Module Improvement

Long-tailed learning also benefits from improvements at the module level—affecting both the feature extractor and the classifier. Representation learning, exemplified by parametric contrastive learning (PaCo), helps learn balanced feature spaces even from imbalanced data distributions. Additionally, innovations in classifier design, such as using cosine classifiers or causal inference-based adjustments, mitigate the bias within classifier weights. Decoupled training, which isolates representation learning from classifier training, has revealed itself as an effective approach to achieving balanced model performance across head and tail classes. Ensemble learning methods, like SADE, further boost results by combining insights from multiple experts, covering a broader spectrum of possible class distributions.

Empirical Evaluation and Implications

The survey's empirical analysis utilizes a new metric, relative accuracy, to fairly evaluate the effectiveness of different methods in handling class imbalance. State-of-the-art ensemble methods, particularly SADE and RIDE, emerge as leading performers, demonstrating substantial improvements in handling the imbalance without sacrificing the performance of head classes.

Future Directions

Several promising directions emerge from this survey. Future research could explore test-agnostic methods where the test distribution is unknown or explore new settings, such as multi-domain and robust long-tailed learning. Additionally, expanding long-tailed learning applications to continuous label spaces in regression and extending to video data indicates uncharted territories.

Conclusion

This comprehensive survey consolidates the advances in deep long-tailed learning, offering a taxonomy that aids understanding of the strategies to address class imbalance. It highlights implications for future research and encourages ongoing exploration in overcoming the challenges posed by long-tailed data in various domains.