Dice Question Streamline Icon: https://streamlinehq.com

Loss landscape degradation for large spline grids under LBFGS

Investigate and characterize the optimization loss landscape for KANs trained with LBFGS at large spline grid sizes (e.g., G ≈ 1000), and establish conditions under which line search becomes inefficient or fails due to adverse landscape properties.

Information Square Streamline Icon: https://streamlinehq.com

Background

Training time for KANs increases sharply at very large grid sizes, and the authors suspect the LBFGS line search struggles because of a degraded loss landscape.

A systematic analysis of the optimization landscape and the interplay with grid size is needed to confirm this conjecture and to guide practical optimization strategies.

References

We conjecture that the loss landscape becomes bad for G=1000, so line search with trying to find an optimal step size within maximal iterations without early stopping.

— KAN: Kolmogorov-Arnold Networks (2404.19756 - Liu et al., 30 Apr 2024) in Subsection 2.4, For accuracy: Grid Extension (footnote to Figure 2 bottom right)