---
title: Self-Optimizing Kolmogorov-Arnold Network
url: https://www.emergentmind.com/topics/self-optimizing-kolmogorov-arnold-network-so-kan
type: topic
---

# Self-Optimizing Kolmogorov-Arnold Network

Searching arXiv for the cited SO-KAN/KAN optimization papers to ground the article in current preprints.
Self-Optimizing Kolmogorov-Arnold Network (SO-KAN) denotes a class of Kolmogorov-Arnold Network formulations in which architectural degrees of freedom or synaptic nonlinearities are optimized automatically rather than fixed by manual design. In the recent literature, the label covers at least three distinct realizations: GA-KAN, which automates KAN design with a genetic algorithm for classification; overprovisioned KANs with differentiable sparsification, which treat architecture search as an end-to-end optimization problem under a minimum-description-length objective; and physical KANs built from Synaptic Nonlinear Elements (SYNEs), which adapt trainable nonlinear device responses in closed loop with a differentiable digital twin [2501.17411] [2512.12448] [2601.15340].

## 1. Mathematical and architectural basis

KANs were introduced to address the issue of interpretability in multilayer perceptrons. Their construction is motivated by the Kolmogorov-Arnold representation theorem, stated in one of the cited formulations as
$$
f(x_1,\dots,x_n)=\sum_{q=1}^{2n+1}\Phi_q\!\Bigl(\sum_{p=1}^n \phi_{q,p}(x_p)\Bigr),
$$
for continuous $f:[0,1]^n\to\mathbb R$, where the $\phi_{q,p}$ and $\Phi_q$ are univariate functions. A relaxed KAN replaces the fixed $2n+1$ width by arbitrary layers and widths,
$$
\mathrm{KAN}(\mathbf x)=\bigl(\Phi_{L-1}\circ\cdots\circ\Phi_1\circ\Phi_0\bigr)(\mathbf x),
$$
with each $\Phi_l$ an $n_{l+1}\times n_l$ matrix of univariate activation functions $\phi_{l,j,i}$ [2501.17411].

The architectural distinction from a conventional MLP is that KANs place the nonlinear transformation on edges rather than at neurons. In the spline-based formulation, each edge activation is
$$
\phi(x)=w_b\,b(x)+w_s\sum_{i=0}^{G+k-1}c_i\,B_{i,k}(x),
$$
where $G$ is the number of spline grid intervals, $k=3$ is fixed, $b(x)$ is a fixed basis, and $w_b,w_s,c_i$ are trainable. In the physical realization, the same functional role is assigned to device-native synaptic nonlinearities $\phi_{p,q}$ and outer functions $\psi_q$, with linear summation at neurons and trainable nonlinear response curves at synapses [2501.17411] [2601.15340].

This shared mathematical template makes “self-optimization” meaningful at multiple levels. It can refer to automatic selection of depth, width, connectivity, and spline resolution; to learned binary gates on edges and summation nodes; or to continual retuning of physical nonlinear transfer curves. A plausible implication is that SO-KAN is better understood as an optimization paradigm for KANs than as a single canonical architecture.

## 2. Principal realizations of self-optimization

The current literature presents several non-equivalent mechanisms for making KANs self-optimizing.

| Realization | Optimized objects | Optimization mechanism |
|---|---|---|
| GA-KAN | Connectivity bits, effective depth, grid parameter $G$ | Genetic algorithm, sparse decoding, validation-loss fitness |
| Differentiable SO-KAN | KAN parameters $\theta$, edge gates, node gates | Hard-concrete relaxation, MDL-style complexity penalty, gradient training |
| Physical SO-KAN | $V_{\rm tune1}$, $V_{\rm tune2}$, input range, gain $G$, device count | Differentiable digital twin, hardware-in-the-loop reprogramming, adaptive pruning |

GA-KAN is framed as an automated optimization approach that requires no human intervention in the design process. Its core contributions are automated optimization, a new encoding strategy, and a new decoding process, with the aim of improving accuracy and interpretability while reducing the number of parameters [2501.17411].

Bagrow and Bongard’s formulation studies overprovisioned architectures combined with sparsification to learn compact, interpretable KANs without sacrificing accuracy. The defining step is differentiable sparsification, which turns architecture search into an end-to-end optimization problem. Overprovisioning and sparsification are reported as synergistic, with the combination outperforming either alone [2512.12448].

The physical realization extends the self-optimizing idea into hardware. There, the network does not merely prune software parameters; it trains the synaptic nonlinearity itself, using SYNE devices whose voltage-current response curves are tuned during learning. The paper characterizes this as learning nonlinear heterogeneity in physical KANs, with closed-loop adaptation to the underlying device physics [2601.15340].

A common misconception is that SO-KAN names a single training rule. The literature instead uses the term for several families of automatic adaptation mechanisms layered onto the KAN formalism.

## 3. Genetic architecture search and sparse decoding

In GA-KAN, the search space is encoded directly into a chromosome. A maximal depth $d$ and maximal neurons per layer $u$ are fixed, with layer sizes labeled $n_0=n$ and $n_{d+1}=m$. For each layer $l=0,\dots,d$, the presence or absence of each of the $n_l\times n_{l+1}$ connections is encoded by a bit $g_{l,i,j}\in\{0,1\}$. The spline grid parameter $G\in[1,64]$ is encoded in $b_{\rm grid}=6$ bits, and the actual depth is encoded in $b_{\rm depth}=2$ bits representing depth minus one. The total chromosome length is
$$
b_{\rm total}=\sum_{l=0}^d (n_l\,n_{l+1})+b_{\rm grid}+b_{\rm depth}.
$$
This directly couples neural architecture search to KAN-specific spline resolution [2501.17411].

The decoding process enforces sparsity and variable depth. The depth bits are applied as zero masks that zero out all $n_l\times n_{l+1}$ bits in layers beyond the chosen depth. If, after masking, a layer’s bits are all zero, the layer is dropped altogether by a degradation mechanism. The resulting network is therefore a sparse KAN of depth at most $4$, with only those edges retained for which $g_{l,i,j}=1$. The final step reads the 6 grid bits, maps them to $G$, and builds each edge activation’s spline with $G$ intervals [2501.17411].

Fitness evaluation is not purely structural. Each decoded individual is trained on $\mathcal D_{\rm train}$ for $N_{\rm steps}=20$ epochs with LBFGS, while the minimal validation loss is tracked:
$$
\mathrm{fitness}(x)=\min_{t=1\ldots N_{\rm steps}}L_{\rm val}^{(t)}.
$$
Invalid networks with no path from input to output are assigned $\mathrm{fitness}=+\infty$, and the GA minimizes this loss. The reported GA parameters are population size $N=100$, generations $G=20$, crossover rate $r_c=0.9$, and mutation rate $r_m=0.5$ [2501.17411].

This scheme realizes self-optimization by discrete search rather than differentiable relaxation. It automates connectivity, effective depth, and spline granularity, while retaining the original KAN emphasis on symbolic interpretability.

## 4. Differentiable sparsification and MDL-style optimization

A second SO-KAN formulation replaces evolutionary search with differentiable gating. The objective is to learn both the KAN parameters $\theta$ and binary gates on every edge and summation node so as to minimize prediction error plus a complexity penalty. In summary form, training minimizes
$$
L = \mathrm{MSE} + \beta\cdot\frac{\log n}{n}\cdot\sum E[\tilde z_j],
$$
where the expected gate count functions as a model-complexity term, and with $c\equiv 1$ this quantity is the expected number of open gates [2512.12448].

Hard binary gates are replaced by a hard-concrete relaxation. For each gate,
$$
u\sim \mathrm{Uniform}(0,1),\qquad
s_j=\sigma\!\left(\frac{\log u-\log(1-u)+\alpha_j}{\tau}\right),
$$
$$
\hat S_j=s_j(\zeta-\gamma)+\gamma,\qquad
\tilde z_j=\mathrm{clip}(\hat S_j;0,1),
$$
with fixed $\tau=2/3$, $\gamma=-0.1$, and $\zeta=1.1$. The expected open-gate probability is
$$
E[\tilde z_j]=\sigma\!\left(\alpha_j-\tau\log(-\gamma/\zeta)\right).
$$
At test time the method thresholds deterministically,
$$
\hat z_j = I[E[\tilde z_j] > 1/2],
$$
so every gate is exactly $0$ or $1$, preserving interpretability [2512.12448].

The gating mechanism is coupled to heavy overprovisioning. A trunk KAN with layer sizes such as $[2,4,4,4,2]$ is augmented with DenseNet-style forward connections, so that every layer $\ell+1$ sees the concatenation of $x^{(0)},x^{(1)},\dots,x^{(\ell)}$. Every such edge and node carries its own gate. Training initializes gate logits at $\alpha_j=-1$, uses Adam with learning rate $10^{-3}$, and may include a warm-up phase with $\beta\leftarrow 0$ before sparsity is imposed. Early stopping is triggered if the fraction of gates decisively in $\{0,1\}$ exceeds $0.99$ [2512.12448].

This realization makes architecture search continuous and end to end. Depth, width, and connectivity emerge from thresholded gate configurations rather than from an explicit chromosome. The single hyperparameter $\beta$ then governs the error-size trade-off within a principled MDL framework.

## 5. Hardware-native SO-KAN and SYNE devices

The physical realization of SO-KAN moves self-optimization into the substrate. SYNE-based KANs operate at room temperature, $0.1$–$1$ microampere currents, and $2$ MHz speeds, with no observed degradation over $10^{13}$ measurements and months-long timescales. Instead of training linear synaptic weights while treating device nonlinearities as fixed, the method trains the synaptic nonlinearity itself [2601.15340].

In the SYNE implementation, each synapse is realized as a combination of device responses. For a given synapse $(p,q)$, the input voltage $V_{\rm in}=x_p$ is applied to constituent SYNE devices. Each device has independent tuning voltages $V_{\rm tune1}^i$ and $V_{\rm tune2}^i$ that shape its $I$–$V$ curve, and each measured output current $I_{\rm out}^i$ is scaled by a trainable gain $G^i$. The synapse output is approximated as
$$
\phi_{p,q}(x_p)\approx \sum_{i=1}^n G^i\cdot I_{\rm out}^i + \mathrm{bias},
$$
with outer functions $\psi_q$ realized similarly or by a small MLP in the digital twin [2601.15340].

Training proceeds through a differentiable digital twin: a data-driven MLP with three hidden layers and ReLU maps $[V_{\rm in},V_{\rm tune1},V_{\rm tune2}]$ to $I_{\rm out}$, with training MSE $\sim 4\times 10^{-7}$ on held-out SYNE data. Each SYNE device has five trainable parameters: $V_{\rm tune1}$ and $V_{\rm tune2}$ in $[-0.6\,\mathrm V,+2\,\mathrm V]$, the learned input range $[V_{{\rm in},\min},V_{{\rm in},\max}]$, and output gain $G\in[0,3]$, with one bias per neuron. Regression uses MSE, classification uses binary cross-entropy, and optimization uses Adam under hardware-motivated constraints such as gain clipping and penalties for tuning voltages outside the high-SNR range [2601.15340].

The self-optimizing loop is explicitly hardware-in-the-loop. After each training epoch, updated tuning parameters are reprogrammed into the SYNE array or time-multiplexed via FPGA, and fresh $I$–$V$ measurements are taken. The method also incorporates expressivity-guided architecture selection via the $\epsilon$-packing expressivity metric, adaptive pruning of low-impact synaptic nonlinearities, and on-device re-tuning as aging or drift alters the device curves. This makes self-optimization a joint adaptation of network structure and physical nonlinear response rather than a purely software pruning procedure [2601.15340].

## 6. Empirical behavior, interpretability, and limitations

GA-KAN is validated on two toy datasets and five UCI classification benchmarks. On Iris and Wine, GA-KAN and the standard KAN both reach $100.00\%$ accuracy, but GA-KAN uses far fewer parameters; on Rice, WDBC, and Raisin, GA-KAN improves or matches the standard KAN while also reducing parameter count [2501.17411].

| Dataset | GA-KAN | Reference KAN |
|---|---|---|
| Iris | 100.00%; 156 params | KAN[4,9,3]: 100.00%; 882 params |
| Wine | 100.00%; 390 params | KAN[13,27,3]: 100.00%; 6,912 params |
| Rice | 95.14%; AUC 0.985; 585 params | KAN[7,15,2]: 94.23%; AUC 0.984; 1,620 params |
| WDBC | 100.00%; AUC 1.000; 1,560 params | KAN[30,61,2]: 100.00%; AUC 1.000; 27,328 params |
| Raisin | 90.00%; AUC 0.938; 1,064 params | KAN[7,15,2]: 86.67%; AUC 0.935; 3,240 params |

For interpretability, GA-KAN extracts symbolic formulae in two stages: `auto_symbolic` first chooses among a library of primitives by $R^2$ fit, after which a manual optional stage can inspect shapes, fix repeated patterns, and retrain coefficients. The Iris example is reported as
$$
\begin{aligned}
z_1 &= 59.08 - 34.97\,x_2,\\
z_2 &= 376.42\,x_1 -147.15\,x_2-288.70\,x_3-1295.19\,\sin(1.95\,x_4-5.44)-116.67,\\
z_3 &= -247.23\,x_2+1953.46\,x_4-303.35\,\sin(1.61\,x_1-0.28)-1229.87,
\end{aligned}
$$
with $\mathrm{Softmax}(z_1,z_2,z_3)$ giving class probabilities [2501.17411].

The differentiable SO-KAN literature reports a similar accuracy-complexity pattern under a different optimization regime. On Nguyen symbolic-regression benchmarks, Baseline and FC-Only fit with $R^2\approx 1$ but retain $\approx 60$ trunk activations; Gates-Only with $\beta=0.01$ prunes to $\approx 12$ activations with $R^2\simeq 0.999$; and the Full model with $\beta=0.01$ often prunes the trunk entirely, using $0$ trunk $+1$ forward connection with $R^2=1.000$. On the Ikeda map, the Full model with $\beta=0.1$ prunes to $19\%$ of activations while multi-step RMSE rises only $2.3\%$. On real-world regression tasks, the Full setting with $\beta=0.01$–$0.1$ cuts to $1.3$–$12.9\%$ of activations and reduces RMSE by $8$–$15\%$ [2512.12448].

The physical SO-KAN results extend the same theme to device efficiency. In $2$D composite regression with architecture $[2,1,1]$ and $12$ SYNEs per synapse, the model uses $92$ trainable parameters and $72$ SYNEs with test MSE $\approx 1.4\times 10^{-2}$, while a software MLP to match the MSE requires $\approx 3{,}000$ parameters. In classification with architecture $[2,2,1]$ and $8$ SYNEs per synapse, the model uses $243$ parameters and $192$ devices, reaches $99.0\%$ accuracy on yin-yang and $97.5\%$ on “5×checker,” and is compared with an MLP baseline needing $\sim 20{,}000$ parameters. For Li-ion battery end-of-life prediction, architecture $[5,12,1]$ uses $863$ devices and $1087$ parameters to achieve normalized MSE $=3.73\times 10^{-3}$ on a $10$-battery test set, while a matching MLP is reported as $[5,300\times 5,1]\approx 360$k parameters [2601.15340].

The main limitations are method-specific. Current GA-KAN is costly because fitness evaluation requires training KANs and has been tested on low-dimensional data; proposed extensions include surrogate fitness, weight-sharing, GPU-gradient acceleration, multi-objective GA, and on-device deployment via further quantization or pruning [2501.17411]. In the differentiable setting, the trade-off between expressivity and compactness is controlled by $\beta$, and for the ecosystem task heavy pruning requires longer training to recover the attractor [2512.12448]. In the physical setting, continual re-optimization is integral rather than incidental, because hardware drift and aging are part of the operating regime [2601.15340].

Taken together, these results show that “self-optimizing” in SO-KAN can mean automated neural architecture search, differentiable architecture pruning under a complexity penalty, or hardware-native adaptation of synaptic nonlinearities. The unifying theme is that KAN expressivity is not treated as a fixed handcrafted design, but as an object of optimization in its own right.

Source: https://www.emergentmind.com/topics/self-optimizing-kolmogorov-arnold-network-so-kan