Learning Prototype Classifiers for Long-Tailed Recognition

Abstract: The problem of long-tailed recognition (LTR) has received attention in recent years due to the fundamental power-law distribution of objects in the real-world. Most recent works in LTR use softmax classifiers that are biased in that they correlate classifier norm with the amount of training data for a given class. In this work, we show that learning prototype classifiers addresses the biased softmax problem in LTR. Prototype classifiers can deliver promising results simply using Nearest-Class- Mean (NCM), a special case where prototypes are empirical centroids. We go one step further and propose to jointly learn prototypes by using distances to prototypes in representation space as the logit scores for classification. Further, we theoretically analyze the properties of Euclidean distance based prototype classifiers that lead to stable gradient-based optimization which is robust to outliers. To enable independent distance scales along each channel, we enhance Prototype classifiers by learning channel-dependent temperature parameters. Our analysis shows that prototypes learned by Prototype classifiers are better separated than empirical centroids. Results on four LTR benchmarks show that Prototype classifier outperforms or is comparable to state-of-the-art methods. Our code is made available at https://github.com/saurabhsharma1993/prototype-classifier-ltr.

• The paper presents prototype classifiers that mitigate softmax bias by using Euclidean distances instead of dot products with classifier weights.
• It introduces channel-dependent temperature scaling and logit adjustment to enhance performance, particularly for minority classes.
• Extensive experiments show the method's robustness, rapid convergence, and superior tail class accuracy across various benchmarks.

Prototype Classifiers for Long-Tailed Recognition: Technical Summary

Background and Motivation

Long-tailed recognition (LTR) presents significant challenges due to power-law distributions prevalent in real-world visual datasets, resulting in severe class imbalance. Conventional classifiers utilizing softmax activation suffer from biased decision boundaries, predominantly favoring head classes. This bias arises because the softmax classifier norm is strongly correlated with class size, encoding class priors through weight magnitudes. Recent research has sought to mitigate these effects with data resampling, loss reshaping, ensemble models, and knowledge transfer—all with varying degrees of success.

The paper, "Learning Prototype Classifiers for Long-Tailed Recognition" (2302.00491), introduces an alternative paradigm: prototype classifiers, which assign classes based on distances in representation space rather than dot products with classifier weights. Prototype classifiers, especially those utilizing learnable prototypes, decouple classifier norm from class size, thereby countering the head class bias inherent in softmax approaches.

Figure 1: Softmax classifiers encode class priors via weight magnitude, yielding biased boundaries; prototype classifiers leverage Euclidean distances to learnable prototypes, producing fairer decision boundaries.

Prototype Classifier Formulation

The proposed model operates in two stages. Feature representations are learned with auxiliary softmax classifiers on imbalanced data. Subsequently, these representations are fixed, and prototypes are initialized as empirical centroids. A learnable prototype $c_y$ per class $y$ is used, with class probabilities determined via negative Euclidean distance:

$\log p(y|g(x)) \propto -\frac{1}{2}d(g(x),c_y)$

where $d(g(x),c_y)$ is the Euclidean distance in representation space.

Gradient Analysis

A salient theoretical result demonstrates that the gradient of the negative log likelihood objective with respect to a prototype $c_y$ is proportional to the misclassification probability. The gradient direction aligns with $g(x) - c_y$ for positive samples and reverses for negatives. Critically, the gradient norm is independent of the distance itself, yielding robust, stable optimization not susceptible to outlying samples. Contrastingly, squared Euclidean distance amplifies outlier influence, corroborated by substantial empirical performance drops.

Enhancements: Channel-Dependent Temperatures and Logit Adjustment

Recognizing that feature channels may have distinct variance scales, the model introduces channel-dependent temperature parameters $T_i$:

$$d_{CDT}(g(x),y) = \sqrt{\sum_{i=1}^d \frac{(g(x)-c_y)_i^2}{T_i}}$$

This equates to learning a diagonal Mahalanobis metric in the representation space, offering feature scaling and selection capabilities.

Logit adjustment further penalizes head classes during prototype learning:

$$L_{xy} = -\log \frac{e^{-\frac{1}{2} d(g(x),c_y) + \tau\log N_y}}{\sum_{y'} e^{-\frac{1}{2}d(g(x),c_{y'}) + \tau\log N_{y'}}}$$

where $N_y$ is class sample count and $\tau$ is the adjustment weight. This improves tail class recall, as validated by ablation studies.

Empirical Evaluation

Prototype classifiers are evaluated on CIFAR100-LT, ImageNet-LT, iNaturalist18, and CIFAR10-LT. Models use fixed feature backbones (e.g., ResNet-32, ResNeXt50, ResNet-50), initialized via end-to-end training with strong weight decay and balanced sampling.

Ablation Studies

• Nearest-Class-Mean (NCM): Employing empirical centroids (no learning) surges past softmax accuracy due to unbiased norms.
• Learnable Prototypes: Further boosts accuracy, particularly on tail classes.
• Channel-Dependent Temperatures and Logit Adjustment: Combined, these refinements attain maximal accuracy, especially for minority classes.

Norm Visualization

Figure 2: Prototype classifier achieves equinorm prototype vectors, eliminating classifier norm correlation with class size—a key improvement over softmax and empirical centroids.

Prototype Evolution

Figure 3: Training increases prototype separation, both in Euclidean and angular terms. Minority classes exhibit reduced separation due to the Minority Collapse effect but are still substantially improved over centroids.

Distance Metric Comparison

Empirical results confirm that Euclidean distance is optimal for prototype learning, with squared Euclidean rendering models highly sensitive to outliers and cosine distance underperforming on tail classes.

Backbone Agnosticism

Prototype classifiers consistently outperform alternate classifier-learning schemes across all tested feature backbones, confirming their robustness and transferability.

Benchmark Results

Extensive evaluation across benchmarks shows that prototype classifiers outperform or are comparable to state-of-the-art long-tailed recognition solutions (e.g., cRT, LogitAdjust, DRO-LT, WD + MaxNorm, PaCO) without reliance on self-supervised pretraining or ensembling. The model is particularly effective for tail class accuracy, addressing limitations of conventional methods.

Practical and Theoretical Implications

The proposed prototype classifier framework establishes a principled geometric alternative to the softmax paradigm for imbalanced data. Its stability and bias-resistance suggest broader applicability to scenarios where classifier norm–class size coupling is detrimental. Model speed—prototype learning converges in a single epoch—makes it adaptable for rapid deployment.

Channel-wise temperature scaling hints at further integration with metric learning, feature selection, and domain adaptation strategies. As prototype classifiers are a layer-level design, they are compatible with contemporary backbone advances, ensembling, and self-supervised representation learning.

Future research might extend prototype learning to more general metric spaces or combine it with ensemble architectures for enhanced robustness.

Conclusion

Prototype classifiers, formulated as learnable centroids in representation space with Euclidean distances and channel-dependent temperature scaling, demonstrate superior performance in long-tailed recognition tasks. They circumvent the biased classifier norm problem inherent in softmax models, exhibit stable optimization dynamics robust to outliers, and produce better-separated class representations. These properties are validated both theoretically and empirically across multiple benchmark datasets. The approach offers substantial practical advantages and lays the foundation for further innovation in classifier design for imbalanced data regimes.