Kolmogorov-Arnold Network Deep Kernels

• DKL-KAN is a hybrid model that integrates Kolmogorov-Arnold Networks with deep kernel learning to capture complex, non-smooth functions.
• The architecture uses trainable activation splines—combining scaled silu and B-spline residuals—to accurately model discontinuities and calibrate predictive uncertainty.
• Empirical results demonstrate that DKL-KAN excels on small-to-medium regression tasks, outperforming standard GP and DKL-MLP in uncertainty calibration and jump modeling.

Kolmogorov-Arnold Network Deep Kernels (DKL-KAN) define a family of scalable and expressive models for supervised learning where the input transformation for a deep kernel Gaussian process is modeled by a Kolmogorov-Arnold Network (KAN). Integrating the structural depth of deep neural networks with the flexibility of Gaussian process kernels, DKL-KAN offers a principled approach to learning complex functions, attaining calibrated predictive uncertainty and handling discontinuities effectively within the Gaussian process (GP) framework (Zinage et al., 2024).

1. Theoretical Foundation: Kolmogorov-Arnold Networks in Deep Kernel Learning

DKL-KAN is grounded in Kolmogorov’s superposition theorem, which states that any continuous function $f:[0,1]^n \to \mathbb{R}$ can be decomposed as a finite sum of univariate functions:

$$f(x) = \sum_{k=1}^{2n+1} \psi_k\left(\sum_{l=1}^n \phi_{l,k}(x_l)\right).$$

In the network representation, for each input $x \in \mathbb{R}^n$ and output dimension $i=1, \ldots, m$:

$$\phi_i(x) = \sum_{k=1}^{2n+1} \psi_{k,i}(u_k), \quad u_k = \sum_{l=1}^n \theta_{k,l} x_l,$$

where $\theta_{k,l}$ are trainable, and the outer $\psi_{k,i}(\cdot)$ are one-dimensional activation splines—a parametric blend of a scaled silu nonlinearity and a trainable B-spline residual:

$$\psi_{k,i}(z) = w_b^{k,i} b(z) + w_s^{k,i} s(z), \quad b(z) = \operatorname{silu}(z), ~ s(z) = \sum_j c_{k,i,j} B_j(z).$$

This design yields a feature map $\phi(x;\beta) \in \mathbb{R}^d$ where $\beta$ aggregates KAN parameters.

KANs thus provide a structurally rich, non-linear, and learnable mapping suitable for kernelization, overcoming some expressivity constraints of standard MLPs in Deep Kernel Learning.

2. DKL-KAN Deep Kernel Definition and Marginal Likelihood Optimization

The DKL-KAN kernel is formalized by composing a parametric base kernel with the KAN feature map:

$$f(x) = \sum_{k=1}^{2n+1} \psi_k\left(\sum_{l=1}^n \phi_{l,k}(x_l)\right).$$0

with $$f(x) = \sum_{k=1}^{2n+1} \psi_k\left(\sum_{l=1}^n \phi_{l,k}(x_l)\right).$$1 commonly chosen as an RBF kernel:

$$f(x) = \sum_{k=1}^{2n+1} \psi_k\left(\sum_{l=1}^n \phi_{l,k}(x_l)\right).$$2

The joint hyperparameter set $$f(x) = \sum_{k=1}^{2n+1} \psi_k\left(\sum_{l=1}^n \phi_{l,k}(x_l)\right).$$3, encompassing both GP kernel and KAN weights, is optimized by maximizing the exact GP log-marginal likelihood:

$$f(x) = \sum_{k=1}^{2n+1} \psi_k\left(\sum_{l=1}^n \phi_{l,k}(x_l)\right).$$4

where $$f(x) = \sum_{k=1}^{2n+1} \psi_k\left(\sum_{l=1}^n \phi_{l,k}(x_l)\right).$$5. Gradients are computed via backpropagation and the Adam optimizer (lr=0.075, decay=0.997, 2500 epochs, early-stop patience=1000) drives joint learning for both sets of parameters.

Key gradients:

• $$f(x) = \sum_{k=1}^{2n+1} \psi_k\left(\sum_{l=1}^n \phi_{l,k}(x_l)\right).$$6
• $$f(x) = \sum_{k=1}^{2n+1} \psi_k\left(\sum_{l=1}^n \phi_{l,k}(x_l)\right).$$7

3. DKL-KAN Architectural Variants and Scalability Frameworks

Two variants of DKL-KAN are analyzed:

• DKL-KAN1 (Neuron-matched): Three hidden layers with [1000, 500, 50] neurons each, matching the reference DKL-MLP baseline in layer structure.
• DKL-KAN2 (Parameter-matched): Layer sizes [256, 128, 64], tuned for approximate numerical parity in $$f(x) = \sum_{k=1}^{2n+1} \psi_k\left(\sum_{l=1}^n \phi_{l,k}(x_l)\right).$$8-parameters with DKL-MLP.

Scalability is addressed via structured GP approximations:

• Low-dimensional ($$f(x) = \sum_{k=1}^{2n+1} \psi_k\left(\sum_{l=1}^n \phi_{l,k}(x_l)\right).$$9) inputs: Use KISS-GP with $x \in \mathbb{R}^n$0 grid inducing points, exploiting $x \in \mathbb{R}^n$1 interpolation with $x \in \mathbb{R}^n$2 complexity.
• High-dimensional ($x \in \mathbb{R}^n$3) inputs: Employ SKIP—product kernel interpolation, factoring the kernel as $x \in \mathbb{R}^n$4, then applying KISS to each factor and recombining via element-wise product, preserving linear scaling in $x \in \mathbb{R}^n$5.

Variant	Hidden Layers (neurons per layer)	Parameter Count
DKL-MLP	[1000, 500, 50]	~0.53M
DKL-KAN1	[1000, 500, 50]	~5.3M
DKL-KAN2	[256, 128, 64]	~0.44M

4. Empirical Evaluation: Accuracy, Scalability, and Uncertainty

Benchmarking on UCI regression datasets (ranging $x \in \mathbb{R}^n$6k–$x \in \mathbb{R}^n$7k, $x \in \mathbb{R}^n$8 up to $x \in \mathbb{R}^n$9) establishes the following:

• Small datasets ($i=1, \ldots, m$0k): DKL-KAN1 consistently yields the lowest RMSE. For example, on the Solar dataset, RMSE values are: GP=1.07, DKL-MLP=1.41, DKL-KAN1=1.04.
• Large datasets ($i=1, \ldots, m$1k): DKL-MLP outperforms DKL-KAN in test RMSE and runtime, except for a small subset (e.g. Ctslice with $i=1, \ldots, m$2k: DKL-KAN1=3.11 vs DKL-MLP=3.19).
• Training and prediction times: On NVIDIA A100 GPU, DKL-MLP trains in 10–20 s for small sets, 100–260 s for large; DKL-KAN1 is 25–50 s for small, comparable for large-scale.

Special focus was given to discontinuity modeling:

• Standard GP: Overly smooth predictions, failing to represent sharp jumps.
• DKL-MLP: Captures the jump, but outputs overconfident, low-variance predictions throughout.
• DKL-KAN: Accurately models discontinuities and delivers rising epistemic uncertainty in regions without direct observations.

5. Implementation Protocols and Hyperparameter Choices

Core components include:

• Base kernel: RBF with signal variance $i=1, \ldots, m$3 and individual lengthscales $i=1, \ldots, m$4.
• Inducing point grids: $i=1, \ldots, m$5 for KISS-GP, per-dimension grids for SKIP.
• Optimization: Adam, learning rate $i=1, \ldots, m$6, decay $i=1, \ldots, m$7, up to 2500 epochs, early-stopping patience 1000.
• Data normalization: Empirical CDF prior to training.
• Software stack: GPyTorch + PyTorch, NVIDIA A100 40GB GPU.

6. Practical Recommendations and Use Cases

DKL-KAN is preferred in contexts marked by modest training set sizes ($i=1, \ldots, m$8k), non-smooth target functions, or the necessity for well-calibrated predictive uncertainty—especially relevant near data-poor or discontinuous regions. For massive datasets or targets with predominantly smooth structure, DKL-MLP offers superior scalability and empirical performance.

Use DKL-KAN when	Use DKL-MLP when
Non-smooth/discontinuous target	Very large $i=1, \ldots, m$9 or high-$$\phi_i(x) = \sum_{k=1}^{2n+1} \psi_{k,i}(u_k), \quad u_k = \sum_{l=1}^n \theta_{k,l} x_l,$$0 data
Need calibrated epistemic uncertainty	Smooth, continuous target function
Modest training set size ($$\phi_i(x) = \sum_{k=1}^{2n+1} \psi_{k,i}(u_k), \quad u_k = \sum_{l=1}^n \theta_{k,l} x_l,$$1k)	Scalability is paramount

A plausible implication is that future development of DKL-KAN architectures, or enhanced optimization/backward-compatibility with extreme-scale data regimes, could extend the regime in which KANs are competitive or dominant.

7. Summary and Outlook

Kolmogorov-Arnold Network Deep Kernels represent a flexible deep kernel learning paradigm, integrating universal function approximation properties of KANs with the probabilistic, nonparametric character of Gaussian Process inference. Experimental results indicate strong empirical advantages in small-to-medium regression settings, notably in uncertainty calibration and discontinuity modeling, while identifying current scalability limits compared to classic DKL-MLP approaches (Zinage et al., 2024). This suggests an active research frontier in the design of deep Gaussian process models wherein architectural expressivity and scalable approximate inference are co-optimized.