---
title: Free-Knots Kolmogorov-Arnold Networks
url: https://www.emergentmind.com/topics/free-knots-kolmogorov-arnold-networks-fr-kan
type: topic
---

# Free-Knots Kolmogorov-Arnold Networks

Free-Knots Kolmogorov-Arnold Networks (FR-KAN) are a class of neural architectures derived from Kolmogorov-Arnold Networks (KAN), extending them by promoting the knot locations in the B-spline parameterization to trainable variables. This dynamic allocation of knot positions enables adaptive grid refinement according to the target function's geometric complexity, offering enhanced expressivity, statistical efficiency, and training stability compared to fixed-grid KAN variants. FR-KANs have demonstrated state-of-the-art empirical performance across symbolic regression, scientific machine learning, and diverse real-world data modalities [2601.18672][2501.09283][2404.19756].

## 1. Theoretical Foundation and Formulation

Kolmogorov-Arnold Networks are based on the Kolmogorov-Arnold representation theorem, which states that any continuous multivariate function $f:[0,1]^n\to\mathbb{R}$ can be expressed as a superposition of univariate functions:
\[
f(x_1,\dots,x_n) = \sum_{q=0}^{2n} \Psi_q\left(\sum_{p=1}^n \Phi_{q,p}(x_p)\right)
\]
KAN instantiates each univariate map $\Phi$ by a learnable B-spline of order $K$ over a fixed grid of knots, optionally augmented with a SiLU shortcut. In its free-knot generalization, FR-KAN, the knot vector $\kappa_i=(\kappa_{i,1},\dots,\kappa_{i,G+1})$ for each input dimension is optimized jointly with spline coefficients.

A typical FR-KAN layer is parameterized as:
\[
f(x;\theta,\kappa)_n = \sum_{i=1}^{n_{\rm in}} \sum_{m=1}^{G+K} b_{n,i,m} B_m^{(K)}(x_i;\kappa_i)
+ r_{n,i} {\rm SiLU}(x_i)
\]
where $B_m^{(K)}(\cdot;\kappa_i)$ denotes the $m$-th order-$K$ B-spline basis function with knot vector $\kappa_i$, and both the weights $\theta$ and knots $\kappa$ are trainable parameters [2601.18672][2501.09283].

## 2. Knot Adaptation via Importance Density Functions

FR-KAN formulates knot allocation as a density estimation problem and introduces Importance Density Functions (IDF) to guide the knot distribution. Given a batch of samples $\{x^{(s)}_i\}_{s=1}^{N_b}$, importance weights $w_s=\varphi(x^{(s)})$ are computed, typically using curvature-based surrogates. The empirical probability mass function for each input point is then $P(x_i^{(s)})=w_s/\sum_t w_t$.

To adapt to geometric complexity, the curvature-based IDF is defined as:
\[
\varphi_{\rm curv}(x) = |\partial^2 \Phi(x)/\partial x^2| + \epsilon
\]
with a corresponding normalized density in the continuous limit:
\[
\rho(x) = \frac{ |f''(x)| + \epsilon }{ \int_a^b (|f''(s)| + \epsilon) ds }
\]
where $\epsilon>0$ is a small constant to avoid vanishing density [2601.18672]. The knot locations are optimized so the induced grid's empirical cumulative distribution (CDF) matches the quantiles of the IDF. The grid-matching loss is given by:
\[
L_{\rm grid}(\kappa_i) = \frac{1}{2} \sum_{j=1}^{G+1} (F_i(\kappa_{i,j}) - q_j)^2
\]
where $F_i(\zeta)=\int_a^\zeta \rho(s) ds$ and $q_j = j/(G+1)$ are uniform quantiles. The loss gradient is explicitly derived, providing efficient backpropagation for knot updates.

## 3. Optimization and Stability Mechanisms

The training objective for FR-KAN is augmented with a grid-matching term:
\[
L(\theta,\kappa) = L_{\rm pred}(\theta,\kappa) + \lambda L_{\rm grid}(\kappa)
\]
where $L_{\rm pred}$ is the prediction loss (e.g., MSE, cross-entropy, or a physics-informed residual for PDEs) and $\lambda\geq 0$ controls the adaptation strength [2601.18672]. Training optimizes both spline weights and knot positions: Adam is used for $\theta$, and SGD with a smaller learning rate for $\kappa$ is empirically effective.

To ensure smoothness and mitigate oscillations, FR-KAN introduces a $C^2$ continuity regularizer:
\[
\mathcal{L}_{C^2} = \lambda\sum_{i,j} \left\lVert \frac{\partial^2}{\partial x^2} c_{j,i}(x) \right\rVert_2^2
\]
enforced via finite differences at grid points. A wide initialization range $[a,b]=[-R,R]$ with $R\approx10$–$20$ is critical for avoiding NaN divergence and reducing grid clustering. Spline parameters are grouped among neurons to reduce parameter overhead and match standard MLPs for scalability [2501.09283].

## 4. Empirical Performance and Comparison

FR-KANs have been evaluated on synthetic function fitting, regression tasks drawing from the Feynman equations, time series prediction, image classification (MNIST, CIFAR-10/100, STL-10), text (AG News), and multimodal datasets (AVMNIST, MIMIC-III) [2501.09283].

Performance benchmarks highlight:

- On 10 synthetic functions, curvature-based FR-KAN reduces median relative $L^2$ error by 25.3% (Wilcoxon $p=0.042$) over input-density baseline.
- On 15 Feynman regression tasks, it improves error by 9.4% ($p=1.5\times10^{-4}$).
- With the 2D Helmholtz PDE, average relative error drop is 23.3% (e.g., 34.2%, 3.15%, 27.7%, 28.1% for different frequencies) [2601.18672].
- On real datasets, FR-KAN matches or exceeds ReLU-MLP accuracy, attaining +3–5% improvement on CIFAR-100/STL-10 and halving RMSEs on Feynman regression tasks.

Ablation analyses demonstrate that free-knot allocation (versus fixed grid) increases the maximum number of knots per layer and thus expressive power, while $C^2$ regularization cuts accuracy variance in half and enhances convergence reliability [2501.09283]. Stability with expanded interval grids ($R\gg1$) enables deeper networks to avoid instability and grid collapse.

## 5. Computational Complexity and Implementation

Curvature-based IDF estimation at each training step requires Hessian diagonal evaluations, which can be performed via automatic differentiation at $O(n_{\rm out} \cdot n_{\rm in} \cdot N_b)$ cost or finite differences. Quantile computation for grid updating requires $O(N_b\log N_b)$ per knot group. Empirical wall-clock overhead is typically 5–15% above the input-density baseline, but this amortizes for deeper/wider tasks:
- Synthetic function: 10.8% overhead (15.47s$\to$17.13s)
- Feynman regression: 14.8% (15.36s$\to$17.62s)
- Helmholtz PDE: 5% (61.21s$\to$64.30s) [2601.18672].

Grouping spline parameters across neurons allows the per-layer parameter count to remain near $O(d_{\rm in} d_{\rm out} + h(G+K))$, comparable to standard MLPs, and below the original KAN's $O(d_{\rm in} d_{\rm out} (G+K))$ scaling.

## 6. Theoretical Properties and Scalability

KANs, and by extension FR-KANs, admit formal universal approximation guarantees. Theoretically, for a KAN with splines of degree $k-1$ and $G$ grid intervals, the $C^m$-norm approximation error scales as:
\[
\|f - f_G\|_{C^m} \leq C G^{-k-1+m}
\]
and empirically, loss scaling for KANs is $\ell\propto N^{-(k+1)}$, outperforming ReLU MLPs which saturate at $\alpha\approx 1$ [2404.19756]. FR-KAN's knot adaptivity further concentrates resolution where needed, suggesting improved parameter efficiency. Spline-knot bounds are theoretically established, with the maximum knot count per layer raised from $(G+K)$ to $h(G+K)$ by free-knot mechanisms [2501.09283]. $C^2$ regularization and wide ranges ensure smooth, stable activations.

## 7. Interpretability and Extensions

FR-KANs remain interpretable: each edge function is a 1D spline map, and trainable knots facilitate direct visualization of function adaptivity and learning strategy. Extensions such as group-shared splines, regularizers for interpretability and sparsity, and alternative kernelizations (Fourier, Rational, RBF) enable broad applicability and further hybridization [2501.09283][2404.19756].

A plausible implication is that FR-KANs, by tightly coupling grid adaptation to geometric features of the target, offer a principled path for scientific machine learning applications demanding accuracy, efficiency, and transparency in low- and moderate-dimensional settings.

Source: https://www.emergentmind.com/topics/free-knots-kolmogorov-arnold-networks-fr-kan