Free-Knots Kolmogorov-Arnold Network: On the Analysis of Spline Knots and Advancing Stability (2501.09283v1)

Abstract: Kolmogorov-Arnold Neural Networks (KANs) have gained significant attention in the machine learning community. However, their implementation often suffers from poor training stability and heavy trainable parameter. Furthermore, there is limited understanding of the behavior of the learned activation functions derived from B-splines. In this work, we analyze the behavior of KANs through the lens of spline knots and derive the lower and upper bound for the number of knots in B-spline-based KANs. To address existing limitations, we propose a novel Free Knots KAN that enhances the performance of the original KAN while reducing the number of trainable parameters to match the trainable parameter scale of standard Multi-Layer Perceptrons (MLPs). Additionally, we introduce new a training strategy to ensure $C^2$ continuity of the learnable spline, resulting in smoother activation compared to the original KAN and improve the training stability by range expansion. The proposed method is comprehensively evaluated on 8 datasets spanning various domains, including image, text, time series, multimodal, and function approximation tasks. The promising results demonstrates the feasibility of KAN-based network and the effectiveness of proposed method.

• The paper presents the Free-Knots KAN, decoupling trainable parameters from grid size to boost flexibility and expressive power over traditional KANs.
• It employs a novel free-knot approach combined with second derivative regularization to significantly stabilize training by mitigating oscillatory activations.
• Empirical results show FR-KAN's superior performance on image classification, multimodal tasks, and symbolic regression while reducing computational complexity.

Free-Knots Kolmogorov-Arnold Network: On the Analysis of Spline Knots and Advancing Stability

The article introduces an innovative approach to enhancing Kolmogorov-Arnold Networks (KANs) through the development of Free-Knots Kolmogorov-Arnold Networks (FR-KAN). The primary motivation for this research is to address the inherent challenges that KANs face, particularly their constrained expressive power due to fixed grid segments, heavy trainable parameters, and training instability arising from activation oscillations.

The researchers first delve into the theoretical underpinning of KANs, highlighting the limitations imposed by the traditional fixed-knot structure of B-splines in KANs. By leveraging the Kolmogorov-Arnold Theorem, which suggests that multivariate continuous functions can be reduced into superpositions of univariate functions, KANs offer a potential avenue for high interpretability within AI models. However, traditional implementations struggle to meet practical requirements due primarily to inefficient parameterization and unstable training regimes.

The authors propose a novel solution by introducing the concept of Free-Knots, which decouples the number of trainable parameters from the grid size $G$ and B-spline order $K$. This allows the network to operate with an adaptable grid that can expand beyond the fixed pre-set, granting the model increased flexibility and expressive power. The free-knot approach allows the KAN to more effectively introduce unique knots tailored dynamically during training, resulting in a tighter bound on the number of knots than previously achieved with fixed-knot models.

In addressing training instability, the paper introduces a unique second derivative regularization strategy, which aims to mitigate oscillation issues common in KANs. This method works alongside an expanded grid range, avoiding the potential over-concentration of grid points that may cause oscillatory behavior within the activation functions. The result is a remarkably stable training process with smoother function approximations, as evidenced in the authors' comprehensive evaluation across multiple tasks.

The FR-KAN model was rigorously tested across diverse datasets, including image classification (e.g., CIFAR10, CIFAR100), multimodal applications (e.g., AVMNIST, MIMIC-III), and symbolic regression tasks (e.g., Feynman dataset). The experiments consistently demonstrate that the FR-KAN outperforms both traditional Multi-Layer Perceptrons (MLPs) and preceding KAN variants, showcasing its robustness and efficacy across various domains. Notably, FR-KAN exhibits particularly strong performance in scenarios requiring complex function approximations, an area where traditional MLPs or even fixed-knot KANs may falter.

The theoretical contributions, such as deriving a tight upper bound for spline knots and demonstrating reduced computational complexity through neuron grouping, present FR-KAN as an adaptable, scalable model positioned to meet the increasing demands of machine learning tasks. By reducing parameter size to levels similar to MLPs without sacrificing expressive power, FR-KAN resolves the practical scalability issues that have hampered broader KAN adoption.

In conclusion, this paper makes significant strides in advancing the utility and performance of KANs. Although it greatly enhances training stability and parameter efficiency, future research could focus on optimizing computational aspects, particularly addressing recursive complexity in the forward and backward passes of B-spline computations. Such efforts could continue improving the algorithm's scalability for even more extensive datasets or tasks, integrating seamlessly into the evolving landscape of AI-driven solutions.