Long-tail learning via logit adjustment (2007.07314v2)

Abstract: Real-world classification problems typically exhibit an imbalanced or long-tailed label distribution, wherein many labels are associated with only a few samples. This poses a challenge for generalisation on such labels, and also makes na\"ive learning biased towards dominant labels. In this paper, we present two simple modifications of standard softmax cross-entropy training to cope with these challenges. Our techniques revisit the classic idea of logit adjustment based on the label frequencies, either applied post-hoc to a trained model, or enforced in the loss during training. Such adjustment encourages a large relative margin between logits of rare versus dominant labels. These techniques unify and generalise several recent proposals in the literature, while possessing firmer statistical grounding and empirical performance.

• The paper introduces logit adjustment techniques that recalibrate predictions using label frequencies to combat bias in long-tailed datasets.
• It integrates adjustments directly into the loss function, ensuring Fisher consistency and providing a unifying statistical framework.
• Empirical results on CIFAR-10-LT and ImageNet variations demonstrate improved balanced error rates over competing methods.

Long-Tail Learning via Logit Adjustment: A Formal Overview

The paper addresses the challenge of classification in contexts with long-tailed label distributions, where many labels are infrequent. This imbalance often leads to biased model predictions, favoring dominant labels. The authors propose modifications to the traditional softmax cross-entropy training to adapt to these scenarios, focusing on logit adjustment through label frequencies as a core strategy.

Key Contributions

1. Logit Adjustment Techniques: The authors introduce two variants of logit adjustment. The first is a post-hoc adjustment applied after model training, while the second integrates logit adjustments directly into the training loss. These adjustments promote a wider margin between frequent and infrequent labels, offering a unifying statistical framework for several existing techniques.
2. Statistical Validity and Consistency: Unlike prior methods, the proposed techniques possess a strong theoretical foundation, ensuring Fisher consistency for minimizing the balanced error. This is significant in long-tail settings where traditional error metrics can be misleading.
3. Empirical Validation: The paper verifies its claims through extensive experiments on synthetic and real-world datasets, including CIFAR and ImageNet variations. The results emphasize the superiority of logit adjustment over alternatives like weight normalization and other loss modifications.

Numerical Results and Claims

The authors report strong empirical performance, with the proposed methods showing improved balanced error rates over existing approaches such as adaptive margins and weight normalization. For instance, on CIFAR-10-LT, the proposed logit-adjusted loss achieves a balanced error of 56.11%, outperforming several competing methods.

Theoretical and Practical Implications

Theoretically, the logit adjustment strategies offer a coherent approach to addressing class imbalance by essentially recalibrating the decision boundary to reflect balanced class probabilities. Practically, this approach enables straightforward adjustments to existing models, allowing better generalization on rare classes without requiring drastic changes to model architecture or training protocol.

Future Directions

The discourse on logit adjustment opens avenues for further exploration, particularly in its applications to settings with varying imbalance levels or in conjunction with data augmentation techniques. Additionally, integrating the proposed methods with high-capacity models or exploring tuning strategies for the adjustment parameter $\tau$ could yield deeper insights and further performance enhancements.

In summary, the paper provides a well-grounded, statistically rigorous approach to tackling long-tail learning challenges, offering both theoretical insights and practical benefits. The methodology paves the way for future advancements in balancing performance across diverse label distributions in real-world classification tasks.