Explaining the regression–classification performance gap for Sven

Investigate and explain the performance gap between regression and classification tasks when training neural networks with Sven, determining why improvements are significant on regression but more modest on classification.

Background

Across experiments, Sven consistently outperforms standard first-order methods on regression problems but only matches them on MNIST classification.

The authors highlight this discrepancy as a key direction, suggesting that differences in loss geometry and the singular value spectra may play roles, and explicitly defer a thorough investigation.

Clarifying this gap would inform when and how to deploy Sven for classification objectives.

References

An important direction for future work is understanding the performance gap between regression and classification settings. While Sven significantly outperforms standard first-order methods on regression tasks, the improvement is more modest for classification, and we leave a thorough investigation of this distinction to future work.

— Sven: Singular Value Descent as a Computationally Efficient Natural Gradient Method  (2604.01279 - Bright-Thonney et al., 1 Apr 2026) in Section 6 (Conclusion)