---
title: 'InfinityKAN: Adaptive Variational KAN Architecture'
url: https://www.emergentmind.com/topics/infinitykan
type: topic
---

# InfinityKAN: Adaptive Variational KAN Architecture

InfinityKAN is a variational machine learning architecture that extends Kolmogorov–Arnold Networks (KANs) by adaptively learning an effectively infinite number of basis functions per univariate transformation during training. This adaptive mechanism replaces fixed architectural hyperparameters with variationally optimized variables, rendering the number of active bases (and thus network capacity) a learnable quantity. The model is grounded in the Kolmogorov–Arnold Theorem but overcomes the practical limitations of fixed-basis designs by leveraging variational inference and a novel smooth basis-selection mechanism [2507.02466].

## 1. Foundations: KANs and the Kolmogorov–Arnold Theorem

The Kolmogorov–Arnold representation states that any continuous bounded function $f: [0,1]^d \to \mathbb{R}$ can be represented exactly as
\[
f(x_1, \dots, x_d) = \sum_{q=1}^{2d+1} \phi'_q\left( \sum_{p=1}^d \phi_{q p}(x_p) \right)
\]
for suitable continuous univariate functions $\phi'_{q}, \phi_{q p}: \mathbb{R} \to \mathbb{R}$. KANs instantiate this result by parameterizing each inner function $\phi_{q p}$ as a linear combination of $n$ preselected basis functions $\{\varphi^n_k\}_{k=1}^n$, i.e.,
\[
\phi^n_{q p}(x) = \sum_{k=1}^n \theta^n_{q p k} \, \varphi^n_k(x)
\]
with coefficients $\theta^n_{qpk}$ trained via backpropagation. The value of $n$ critically affects expressivity and overfitting trade-offs but must be hand-tuned for each task [2507.02466].

## 2. Variational Formulation and Inference in InfinityKAN

InfinityKAN tackles the architectural constraint of fixed $n$ by casting both weights and basis counts as random variables within a joint probabilistic model. For a dataset $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^D$, the latent variables are $\theta = \{\theta^{\ell n}_{q p k}\}$ (basis coefficients, layer-indexed) and $\lambda = \{\lambda_\ell\}$ (the canonical rates controlling the number of bases per layer).

The joint distribution is formulated as:
\[
p(Y, \theta, \lambda \mid X) = p(Y \mid X, \theta, \lambda) \, p(\theta) \, p(\lambda)
\]
with
- $p(\theta^{\ell n}_{q p k}) = \mathcal{N}(0, \sigma_\ell^2 I)$ (Gaussian prior for weights),
- $p(\lambda_\ell) = \mathcal{P}(\lambda_\ell; \eta_\ell)$ (Poisson prior for basis count),
- $p(Y \mid X, \theta, \lambda)$ being the standard Gaussian (regression) or softmax (classification) likelihood where the KAN’s active basis count is $K_\ell = 2\lambda_\ell + 1$ per layer.

Variational inference is used for posterior approximation:
\[
q(\theta, \lambda) = \prod_{\ell=1}^L q(\lambda_\ell) \prod_{\text{active } \theta} q(\theta) \prod_{\text{inactive } \theta} p(\theta)
\]
where $q(\lambda_\ell) = \mathcal{P}(\lambda_\ell; \bar{\lambda}_\ell)$ and $q(\theta^{\ell n}_{qpk}) = \mathcal{N}(\bar{\theta}^{\ell n}_{qpk}, I)$ for active coefficients. The training objective maximizes the ELBO:
\[
\mathcal{L}_{\rm ELBO} = \sum_{i=1}^D \ln p\left(y_i \mid x_i, \theta = \bar{\theta}, \lambda = \bar{\lambda}\right)
+ \sum_{\ell=1}^L \left[\ln p(\bar{\lambda}_\ell; \eta_\ell) - \ln q(\bar{\lambda}_\ell; \bar{\lambda}_\ell)\right]
+ \sum_{\ell, q, p, k} \ln p(\bar{\theta}_{qpk}^{\ell K_\ell}; 0, \sigma_\ell^2 I)
\]
Optimization is performed by SGD on variational means $\{\bar{\theta}, \bar{\lambda}\}$ [2507.02466].

## 3. InfinityKAN Architecture and Basis Adaptivity

Each layer in InfinityKAN implements a generalized KAN, but with each univariate function parameterized as a potentially infinite linear combination:
\[
\phi^\ell_{q p}(x) = \lim_{n \to \infty} \sum_{k=1}^n \theta^{\ell n}_{qpk} \, \varphi^n_k(x)
\]
For practical training and inference, the series is truncated adaptively using a learnable $K_\ell$ for each layer, with an additional smooth “window” function:
\[
w_k^{K_\ell} = \left(1 + e^{-\beta K_\ell + \beta \gamma |x_k|}\right)^{-1} \mathbbm{1}_{x_k \in [-K_\ell, K_\ell]}
\]
This window enables differentiable masking of inactive bases and supports backpropagation-friendly optimization over $K_\ell$.

The $\ell$th layer output is computed as:
\[
x_q^\ell = \sum_{p=1}^{d_{\ell-1}} \sum_{k=1}^{K_\ell} \theta^{\ell,K_\ell}_{q p k} \, w^{K_\ell}_k \, \varphi^{K_\ell}_k(x_p^{\ell-1})
\]
where $\varphi^{K_\ell}_k$ can be chosen from ReLU-steps, Chebyshev polynomials, or Fourier modes. This generalizes classical KANs without prescribing basis cardinality a priori.

## 4. Training Algorithm and Optimization Techniques

Training of InfinityKAN proceeds via stochastic gradient descent applied to the ELBO, using the following procedure, as excerpted (Algorithm 1) [2507.02466]:
```python
# InfinityKAN Training
Input: Dataset D, basis family {φ_k}, interpolation ℐ
Initialize variational parameters {barθ, barλ}
for epoch = 1…N do
    for (x, y) in D do
        for ℓ = 1…L do
            K_ℓ ← 2·Poisson(η_ℓ; barλ_ℓ) + 1     # sample active window
            w^ℓ_k ← window(K_ℓ, x_k)              # smooth mask
            θ^ℓ ← ℐ(θ'^ℓ)                         # interpolate old θ
        end
        ŷ ← InfinityKAN(x; {θ, w, K})             # forward pass
        loss ← –ELBO(ŷ, y, {barθ, barλ})           # eq. (7)
        ← backpropagate gradients ∇_{barθ, barλ}
        ← SGD-step updates of barθ, barλ
    end
end
```
Special techniques include:
- A soft window to smooth the effective change in basis count.
- Mean-field variational approximations.
- First-order approximations to the variational expectation.
The discrete $\lambda$ variable is handled via reparameterization and optimization over its mean parameter $\bar{\lambda}$ [2507.02466].

## 5. Empirical Performance and Robustness Analyses

InfinityKAN was evaluated on a range of tasks, including synthetic regression, image classification (MNIST, Fashion-MNIST, CIFAR-10/100, EuroSAT), and graph classification (MUTAG, NCI1, PROTEINS, ENZYMES, REDDIT-BINARY). The variant Infinity-GKAN extends these ideas to graph neural architectures. Empirical results establish:

- On regression and classification benchmarks, InfinityKAN matches or improves upon best-tuned fixed-$n$ KANs and MLPs, adaptively learning basis numbers $K_\ell \approx 5$–20 per layer.
- On graph classification, Infinity-GKAN outperforms both fixed-basis KANs and conventional MLP graph encoders.
- Ablation studies across the window function parameters $(\beta, \gamma)$, basis choice (ReLU, Chebyshev, Fourier), variational priors, and width/depth show robust, task-adaptive selection of $K_\ell$ and basis type [2507.02466].

Empirical visualizations (see Figure 1 in the source) corroborate smooth, monotonic evolution and stabilization of the learned $K_\ell$ during training. The ELBO remains Lipschitz-continuous in $K_\ell$, indicating stable optimization.

## 6. Significance and Theoretical Properties

InfinityKAN eliminates the need for manual selection of basis set cardinality in KANs, preserving the full expressive power of the Kolmogorov–Arnold Theorem in a learnable, scalable framework. The architecture is fully compatible with modern backpropagation and variational inference pipelines and is practically competitive with standard MLPs and fixed-KAN architectures across diverse tasks. The use of a Poisson prior and adaptive windows for basis selection allows the architecture to allocate model capacity dynamically, reducing overfitting risk and computation associated with unnecessarily large basis sets.

A plausible implication is that this framework generalizes to any machine learning scenario in which function approximation benefits from infinite or adaptive expansion sets, provided a suitable variational or amortized optimization method can be specified [2507.02466].

## 7. Limitations and Open Directions

Despite empirical success and theoretical soundness, several open questions remain regarding optimal prior selection for $\lambda$, the theoretical limits of adaptively truncated univariate basis expansions, and the transferability of the approach to broader network architectures. Further, the extension of InfinityKAN principles to object types other than scalar function approximation, or to structured data beyond graphs, remains an open area for investigation. Robustness against adversarial basis selection and convergence guarantees under non-Gaussian priors also warrant further study [2507.02466].

Source: https://www.emergentmind.com/topics/infinitykan