Decoupling Representation and Classifier for Long-Tailed Recognition (1910.09217v2)

Abstract: The long-tail distribution of the visual world poses great challenges for deep learning based classification models on how to handle the class imbalance problem. Existing solutions usually involve class-balancing strategies, e.g., by loss re-weighting, data re-sampling, or transfer learning from head- to tail-classes, but most of them adhere to the scheme of jointly learning representations and classifiers. In this work, we decouple the learning procedure into representation learning and classification, and systematically explore how different balancing strategies affect them for long-tailed recognition. The findings are surprising: (1) data imbalance might not be an issue in learning high-quality representations; (2) with representations learned with the simplest instance-balanced (natural) sampling, it is also possible to achieve strong long-tailed recognition ability by adjusting only the classifier. We conduct extensive experiments and set new state-of-the-art performance on common long-tailed benchmarks like ImageNet-LT, Places-LT and iNaturalist, showing that it is possible to outperform carefully designed losses, sampling strategies, even complex modules with memory, by using a straightforward approach that decouples representation and classification. Our code is available at https://github.com/facebookresearch/classifier-balancing.

Decoupling Representation and Classifier for Long-Tailed Recognition: A Formal Analysis

The paper "Decoupling Representation and Classifier for Long-Tailed Recognition," authored by Kang et al., advances the paper of long-tailed distributions in visual recognition by proposing a novel decoupling strategy that separates the processes of representation learning and classification. This approach contrasts with conventional methods that integrate both processes into a single learning framework. The paper's findings provide significant insights into addressing class imbalance in datasets and set new benchmarks for long-tailed recognition tasks.

Introduction and Motivation

The visual world inherently features long-tailed distributions where a few classes (head classes) have abundant instances while many classes (tail classes) have limited instances. This imbalance presents a challenge for deep learning models, as standard learning algorithms tend to favor head classes, thereby degrading performance on tail classes. The traditional practice of jointly learning representations and classifiers through methods such as loss re-weighting, data re-sampling, or transfer learning has limitations in effectively handling this imbalance.

Core Hypothesis

The authors propose decoupling representation learning from classification as a means to better manage long-tailed recognition. This decoupling approach enables independent optimization of the feature extractor and the classifier, allowing for more nuanced handling of data imbalance. Specifically, the paper focuses on determining whether high-quality representations can be obtained using standard instance-balanced sampling and if adjusting the classifier alone can achieve superior recognition performance, particularly for tail classes.

Methodology

Sampling Strategies

The paper explores several sampling strategies:

1. Instance-Balanced Sampling: Instances are sampled uniformly without considering class distribution.
2. Class-Balanced Sampling: Each class is sampled uniformly, irrespective of the number of instances per class.
3. Square-Root Sampling: A compromise between instance-balanced and class-balanced sampling, proportional to the square root of the class frequencies.
4. Progressively-Balanced Sampling: A dynamic approach that transitions from instance-balanced to class-balanced sampling over the training epochs.

Classifier Adjustments

Different approaches to classifier adjustments are examined:

1. Classifier Retraining (cRT): Retraining the classifier using class-balanced sampling while keeping the representations fixed.
2. Nearest Class Mean (NCM): A non-parametric approach using cosine similarity to classify based on nearest mean representations.
3. $\tau$-Normalization: Adjusting classifier weight magnitudes by scaling them based on a temperature parameter $\tau$ to achieve balanced decision boundaries.
4. Learnable Weight Scaling (LWS): Learning scaling factors directly from the data.

Experimental Setup

The paper conducts extensive experiments on three major long-tailed datasets: ImageNet-LT, Places-LT, and iNaturalist. ResNet and ResNeXt architectures are utilized to evaluate the efficacy of the proposed decoupling method. Metrics include top-1 accuracy on balanced test sets and specific performances on many-shot, medium-shot, and few-shot splits.

Key Results

1. Joint vs. Decoupled Learning: The decoupled learning scheme consistently outperforms the conventional joint learning approach. For instance, on the ImageNet-LT dataset, decoupled methods yield significantly better accuracy for both medium-shot and few-shot classes.
2. Instance-Balanced Sampling: Surprisingly, instance-balanced sampling alone is sufficient for learning high-quality representations, as evidenced by its superior performance across several evaluation metrics.
3. Classifier Adjustments: Adjusting the classifier through $\tau$-normalization or retraining (cRT) markedly improves long-tailed recognition performance. Notably, $\tau$-normalization effectively balances classifier weights, yielding a performance boost without additional training complexity.
4. Superior Performance: The proposed decoupling approach sets new state-of-the-art results on all three benchmarks, outperforming existing methods that use more complex loss functions and sampling strategies.

Theoretical and Practical Implications

The findings challenge the prevailing belief that complex sampling strategies or loss functions are essential for long-tailed recognition. Instead, the paper demonstrates that decoupling and straightforward classifier adjustments can achieve higher efficacy. This has significant implications for designing future AI models, suggesting a pivot towards simpler, modular approaches that facilitate targeted optimization.

Future Directions

Potential future developments include exploring adaptive methods for dynamically adjusting $\tau$, investigating the effectiveness of decoupling in other domains such as text and speech recognition, and extending the approach to multi-label and hierarchical classification tasks. The paper also opens avenues for reducing computational overhead in training by leveraging efficient sampling and optimization techniques.

In conclusion, the paper by Kang et al. provides a compelling argument for the decoupling of representation learning and classification in long-tailed recognition. By systematically exploring and validating this approach, the paper not only advances theoretical understanding but also offers practical strategies for improving classification performance in imbalanced datasets.