- The paper proposes a theoretically principled approach that leverages Neural Collapse geometry to equalize class-wise losses at convergence.
- It reformulates loss reweighting as an inverse problem, providing a closed-form, convex solution that is computationally efficient and compatible with large datasets.
- Empirical evaluations on benchmark long-tailed datasets demonstrate significant accuracy improvements by enforcing balanced loss and optimal ETF alignment.
Rethinking Loss Reweighting in Imbalanced Learning: An Inverse Problem Perspective Grounded in Neural Collapse Geometry
Motivation and Limitations of Current Loss Reweighting
Loss reweighting is a staple approach for mitigating the performance degradation in deep neural networks that arises when training data exhibit a long-tailed class distribution. Despite widespread adoption, the majority of reweighting methods in the literature remain fundamentally heuristic, with design decisions driven primarily by empirical class frequencies or observed model behavior, rather than a principled target. The lack of a clearly defined and justified objective for reweighting impedes both systematic analysis and theoretically sound improvement of these methods.
The phenomenon of Neural Collapse (NC), which characterizes the terminal dynamics of deep networks trained on balanced datasets, reveals a canonical geometric configuration: class means and classifier weights become equiangular and symmetrically dispersed as a Simplex Equiangular Tight Frame (ETF) in the embedding space. Under this geometric alignment, the average per-class losses are enforced to be equal. This observation suggests a concrete optimization target for reweighting strategies that current loss reweighting techniques neglect.
Neural Collapse-Inspired Equal-Loss Objective
The authors leverage the NC phenomenon to identify a rigorously specified design goal for reweighting: exactly equalize class-wise average losses at convergence. The key result, formalized as Theorem 3.1, proves that under the ETF-aligned geometry of neural collapse, the average loss across classes is necessarily equal. Conversely, Theorem 3.2 demonstrates that persistent loss imbalance during training fundamentally precludes the emergence of neural collapse geometry, thus directly obstructing optimal generalization and feature structure, especially in long-tailed regimes (2605.10047).
Based on these results, loss reweighting should no longer be approached as a mere class-frequency adjustment, but should explicitly target the minimization of inter-class loss imbalance as measured by a tailored coefficient. This approach ensures that the learned embedding and classifier weights retain the theoretically desirable NC structure, which is empirically linked to improved generalization for both head and tail classes.
Departing from the conventional “forward” prescription of weights based on dataset statistics, the paper introduces a principled “inverse-problem” framework for loss reweighting. The weight assignment for each class is recast as the solution to an optimization problem: infer weights such that the reweighted empirical per-class losses converge to their global mean, explicitly including a Tikhonov regularizer to allow integration with existing heuristic or prior weights if desired.
The crucial property of this formulation is that it yields a strictly convex per-class quadratic objective, which admits a closed-form solution for the optimal class-wise weights as a function of the current network outputs and running loss statistics. This solution is computationally efficient and suitable for online or batch-wise updates. Moreover, it is directly compatible with plug-and-play augmentation of existing pipelines, ensuring deployability in large-scale training regimes.
To further counteract the under-optimization of tail classes caused by their lower batch occurrence, the method augments the class weights by a macro-level, batch-frequency-aware scaling factor, parameterized by a hyperparameter controlling compensation strength. This hybridization addresses both micro-level (batch-local) and macro-level (global) sources of reweighting inefficacy.
Empirical Evaluation and Quantitative Results
The proposed method is subjected to rigorous evaluation on canonical long-tailed classification benchmarks including CIFAR-10-LT, CIFAR-100-LT, iNaturalist, and ImageNet-LT, employing ResNet backbones and adhering to standard protocols.
Strong numerical results are reported:
- On CIFAR-100-LT under IF=100, the method achieves 47.9% accuracy compared to 41.6% for cross-entropy and consistently outperforms all baselines, including class-balanced and dynamic reweighting competitors, by margins of up to 7.3 percentage points.
- On ImageNet-LT and iNaturalist, applying the method to both cross-entropy and the state-of-the-art GLMC backbone yields the highest documented accuracies, with improvements of 1.3–1.6% over previous bests despite stricter imbalance.
- The method preserves or improves NC1-NC3 metrics throughout training, yielding tighter ETF alignment and lower class imbalance coefficients at convergence than all tested alternatives.
Ablation studies confirm that both the batch-wise inverse optimization and macro-level compensation components are required for maximal effect, with complementary benefits.
Theoretical and Practical Implications
This work resolves a major source of ambiguity in loss reweighting by introducing an explicit, theoretically justified optimization target. The explicit connection between neural collapse geometry and loss balance clarifies both the symptoms and root causes of inferior performance in heuristic reweighing under class imbalance. Furthermore, by demonstrating that loss imbalance is the sole obstruction to terminal phase ETF alignment, the paper establishes a new standard for loss reweighting methods in long-tailed learning: any effective method should, at minimum, enforce convergence of class-wise average losses.
The proposed inverse-problem framework is extensible and general: it can be integrated with any base loss and existing regularization, and the closed-form per-batch adaptation ensures minimal computational overhead. Practically, this unlocks more robust deployment of deep models in data regimes characterized by strong imbalance and rare events.
Theoretically, the results motivate future work on generalizing NC-inspired objectives to non-classification domains (e.g., multi-label or structured prediction) and on extending the inverse optimization approach to other forms of training heterogeneity (e.g., instance noise, distributional shift).
Future Directions
Directions for further investigation include:
- Extending loss balancing to multi-task, multi-label, or hierarchical label structures where the concept of ETF may require redefinition.
- Learning adaptive regularization or macro-compensation coefficients for online or federated settings.
- Analyzing the implications of NC alignment under adversarial or highly non-stationary data streams, where loss imbalance may emerge dynamically.
Conclusion
By formalizing loss reweighting as an inverse problem targeting the minimization of class-wise loss imbalance, and grounding this objective in the geometry of neural collapse, this work advances both the theoretical understanding and empirical effectiveness of imbalance learning techniques. The resulting method is distinguished by its mathematical rigor, computational efficiency, and superior empirical performance, and it reframes the loss reweighting paradigm to align with the best-understood phenomena in deep representation learning (2605.10047).
