---
title: Kolmogorov-Arnold Network Deep Kernels
url: https://www.emergentmind.com/topics/kolmogorov-arnold-network-deep-kernels-dkl-kan
type: topic
---

# Kolmogorov-Arnold Network Deep Kernels

Kolmogorov-Arnold Network Deep Kernels (DKL-KAN) define a family of scalable and expressive models for supervised learning where the input transformation for a deep kernel Gaussian process is modeled by a Kolmogorov-Arnold Network (KAN). Integrating the structural depth of deep neural networks with the flexibility of Gaussian process kernels, DKL-KAN offers a principled approach to learning complex functions, attaining calibrated predictive uncertainty and handling discontinuities effectively within the Gaussian process (GP) framework [2407.21176].

## 1. Theoretical Foundation: Kolmogorov-Arnold Networks in Deep Kernel Learning

DKL-KAN is grounded in Kolmogorov’s superposition theorem, which states that any continuous function $f:[0,1]^n \to \mathbb{R}$ can be decomposed as a finite sum of univariate functions:
$$
f(x) = \sum_{k=1}^{2n+1} \psi_k\left(\sum_{l=1}^n \phi_{l,k}(x_l)\right).
$$
In the network representation, for each input $x \in \mathbb{R}^n$ and output dimension $i=1, \ldots, m$:
$$
\phi_i(x) = \sum_{k=1}^{2n+1} \psi_{k,i}(u_k), \quad u_k = \sum_{l=1}^n \theta_{k,l} x_l,
$$
where $\theta_{k,l}$ are trainable, and the outer $\psi_{k,i}(\cdot)$ are one-dimensional activation splines—a parametric blend of a scaled silu nonlinearity and a trainable B-spline residual:
$$
\psi_{k,i}(z) = w_b^{k,i} b(z) + w_s^{k,i} s(z), \quad b(z) = \operatorname{silu}(z), ~ s(z) = \sum_j c_{k,i,j} B_j(z).
$$
This design yields a feature map $\phi(x;\beta) \in \mathbb{R}^d$ where $\beta$ aggregates KAN parameters.

KANs thus provide a structurally rich, non-linear, and learnable mapping suitable for kernelization, overcoming some expressivity constraints of standard MLPs in Deep Kernel Learning.

## 2. DKL-KAN Deep Kernel Definition and Marginal Likelihood Optimization

The DKL-KAN kernel is formalized by composing a parametric base kernel with the KAN feature map:
$$
k_{\mathrm{DKL-KAN}}(x, x'; \alpha, \beta) = k_{\text{base}}(\phi(x; \beta), \phi(x'; \beta) \mid \alpha)
$$
with $k_{\text{base}}$ commonly chosen as an RBF kernel:
$$
k_{\text{base}}(u, u') = \sigma^2 \exp\left(-\frac{1}{2} \sum_{j=1}^d \frac{(u_j - u'_j)^2}{\ell_j^2}\right).
$$
The joint hyperparameter set $\gamma = (\alpha, \beta)$, encompassing both GP kernel and KAN weights, is optimized by maximizing the exact GP log-marginal likelihood:
$$
L(\gamma) = -\frac{1}{2} y^\top [K_{\gamma} + \sigma_y^2 I]^{-1} y - \frac{1}{2} \log |K_{\gamma} + \sigma_y^2 I| - \frac{n}{2} \log 2\pi,
$$
where $[K_\gamma]_{ij} = k_{\mathrm{DKL-KAN}}(x_i, x_j \mid \gamma)$. Gradients are computed via backpropagation and the Adam optimizer (lr=0.075, decay=0.997, 2500 epochs, early-stop patience=1000) drives joint learning for both sets of parameters.

Key gradients:
- $\frac{\partial L}{\partial \alpha} = \operatorname{tr}\left(\frac{\partial L}{\partial K} \frac{\partial K}{\partial \alpha}\right)$
- $\frac{\partial L}{\partial \beta} = \sum_{i,j} \left(\frac{\partial L}{\partial K_{ij}}\right) \frac{\partial k(\phi(x_i), \phi(x_j))}{\partial \phi(x_i)} \frac{\partial \phi(x_i)}{\partial \beta}$

## 3. DKL-KAN Architectural Variants and Scalability Frameworks

Two variants of DKL-KAN are analyzed:
- **DKL-KAN1** (Neuron-matched): Three hidden layers with [1000, 500, 50] neurons each, matching the reference DKL-MLP baseline in layer structure.
- **DKL-KAN2** (Parameter-matched): Layer sizes [256, 128, 64], tuned for approximate numerical parity in $\beta$-parameters with DKL-MLP.

Scalability is addressed via structured GP approximations:
- **Low-dimensional ($d \leq 4$) inputs**: Use KISS-GP with $m \approx n$ grid inducing points, exploiting $W K(U,U) W^\top$ interpolation with $O(n + h(m))$ complexity.
- **High-dimensional ($d > 4$) inputs**: Employ SKIP—product kernel interpolation, factoring the kernel as $k(X,X)=\otimes_{d=1}^{D}k^{(d)}(X_{:,d},X_{:,d})$, then applying KISS to each factor and recombining via element-wise product, preserving linear scaling in $n$.

| Variant         | Hidden Layers (neurons per layer) | Parameter Count      |
|-----------------|-----------------------------------|---------------------|
| DKL-MLP         | [1000, 500, 50]                   | ~0.53M              |
| DKL-KAN1        | [1000, 500, 50]                   | ~5.3M               |
| DKL-KAN2        | [256, 128, 64]                    | ~0.44M              |

## 4. Empirical Evaluation: Accuracy, Scalability, and Uncertainty

Benchmarking on UCI regression datasets (ranging $n=1$k–$63$k, $d$ up to $385$) establishes the following:
- **Small datasets ($n < 10$k):** DKL-KAN1 consistently yields the lowest RMSE. For example, on the Solar dataset, RMSE values are: GP=1.07, DKL-MLP=1.41, DKL-KAN1=1.04.
- **Large datasets ($n > 20$k):** DKL-MLP outperforms DKL-KAN in test RMSE and runtime, except for a small subset (e.g. Ctslice with $n=53$k: DKL-KAN1=3.11 vs DKL-MLP=3.19).
- **Training and prediction times:** On NVIDIA A100 GPU, DKL-MLP trains in 10–20 s for small sets, 100–260 s for large; DKL-KAN1 is 25–50 s for small, comparable for large-scale.

Special focus was given to discontinuity modeling:
- **Standard GP:** Overly smooth predictions, failing to represent sharp jumps.
- **DKL-MLP:** Captures the jump, but outputs overconfident, low-variance predictions throughout.
- **DKL-KAN:** Accurately models discontinuities and delivers rising epistemic uncertainty in regions without direct observations.

## 5. Implementation Protocols and Hyperparameter Choices

Core components include:
- **Base kernel:** RBF with signal variance $\sigma^2$ and individual lengthscales $\ell_d$.
- **Inducing point grids:** $m \approx n$ for KISS-GP, per-dimension grids for SKIP.
- **Optimization:** Adam, learning rate $0.075$, decay $0.997$, up to 2500 epochs, early-stopping patience 1000.
- **Data normalization:** Empirical CDF prior to training.
- **Software stack:** GPyTorch + PyTorch, NVIDIA A100 40GB GPU.

## 6. Practical Recommendations and Use Cases

DKL-KAN is preferred in contexts marked by modest training set sizes ($n \leq 20$k), non-smooth target functions, or the necessity for well-calibrated predictive uncertainty—especially relevant near data-poor or discontinuous regions. For massive datasets or targets with predominantly smooth structure, DKL-MLP offers superior scalability and empirical performance.

| Use DKL-KAN when | Use DKL-MLP when |
|------------------|------------------|
| Non-smooth/discontinuous target | Very large $n$ or high-$d$ data |
| Need calibrated epistemic uncertainty | Smooth, continuous target function |
| Modest training set size ($n \leq 20$k) | Scalability is paramount           |

A plausible implication is that future development of DKL-KAN architectures, or enhanced optimization/backward-compatibility with extreme-scale data regimes, could extend the regime in which KANs are competitive or dominant.

## 7. Summary and Outlook

Kolmogorov-Arnold Network Deep Kernels represent a flexible deep kernel learning paradigm, integrating universal function approximation properties of KANs with the probabilistic, nonparametric character of Gaussian Process inference. Experimental results indicate strong empirical advantages in small-to-medium regression settings, notably in uncertainty calibration and discontinuity modeling, while identifying current scalability limits compared to classic DKL-MLP approaches [2407.21176]. This suggests an active research frontier in the design of deep Gaussian process models wherein architectural expressivity and scalable approximate inference are co-optimized.

Source: https://www.emergentmind.com/topics/kolmogorov-arnold-network-deep-kernels-dkl-kan