KAN-CL: Per-Knot Importance Regularization for Continual Learning with Kolmogorov-Arnold Networks

Abstract: Catastrophic forgetting remains the central obstacle in continual learning (CL): parameters shared across tasks interfere with one another, and existing regularization methods such as EWC and SI apply uniform penalties without awareness of which input region a parameter serves. We propose KAN-CL, a continual learning framework that exploits the compact-support spline parameterization of Kolmogorov-Arnold Networks (KANs) to perform importance-weighted anchoring at per-knot granularity. Deployed as a classification head on a convolutional backbone with standard EWC regularization on the backbone (bbEWC) KAN-CL achieves forgetting reductions of 88% and 93% over a head-only KAN baseline on Split-CIFAR-10/5T and Split-CIFAR-100/10T respectively, while matching or exceeding the accuracy of all baselines on both benchmarks. We further provide a Neural Tangent Kernel (NTK) analysis showing that KAN's spline locality induces a structural rank deficit in the cross-task NTK, yielding a forgetting bound that holds even in the feature-learning regime. These results establish that combining an architecture with natural parameter locality (KAN head) with a complementary backbone regularizer (bbEWC) yields a compositional and principled approach to catastrophic forgetting.

• The paper introduces KAN-CL, a method that applies per-knot importance regularization on B-spline coefficients to precisely mitigate catastrophic forgetting.
• Empirical results on Split-CIFAR benchmarks show up to 93% reduction in forgetting while maintaining or improving accuracy compared to conventional methods.
• Theoretical NTK analysis demonstrates that KAN-CL’s architectural decoupling yields reduced cross-task interference, offering a tighter bound on forgetting than traditional MLP approaches.

KAN-CL: Per-Knot Importance Regularization for Continual Learning with Kolmogorov-Arnold Networks

Introduction and Motivation

Catastrophic forgetting remains a fundamental limitation in continual learning (CL), where neural models struggle to preserve performance on previous tasks when sequentially trained on new ones. Standard regularization-based methods, such as Elastic Weight Consolidation (EWC) and Synaptic Intelligence (SI), constrain parameter drift in proportion to a global importance measure, but do not leverage spatial or taskwise locality in the parameterization. The introduction of Kolmogorov-Arnold Networks (KANs), which utilize learnable univariate B-spline functions on edges, enables structural parameter locality: each spline coefficient is only active on a compact input interval, offering the potential for spatially precise regularization.

This work proposes KAN-CL, which exploits the unique parameterization of KANs to provide per-knot importance-weighted regularization. Coupled with EWC regularization applied to a convolutional feature backbone (bbEWC), the hybrid system achieves state-of-the-art forgetting reduction while maintaining or improving accuracy on benchmark CL tasks. Theoretical analysis shows that KAN's locality induces a rank deficit in cross-task Neural Tangent Kernel (NTK), yielding provably tighter forgetting bounds in both fixed-feature and feature-learning regimes.

Theoretical Framework

KANs parameterize neural functions with univariate B-splines per edge, resulting in coefficients $c_{ijk}$ that only influence inputs falling within specific knot intervals. The per-knot Fisher information decomposition (Lemma 1) precisely quantifies the contribution of each coefficient, and Theorem 1 shows that, under disjoint task marginal supports, cross-task Fisher scores are structurally zero outside task-local knots. The NTK for the spline components (Equation 1) further reflects this architectural decoupling, giving rise to a strict upper bound on cross-task interference (Theorem 2):

• Rank advantage: The cross-task NTK for a KAN head can be strictly lower-rank than that of an MLP head, reflecting reduced parameter sharing and task entanglement.
• Forgetting bound: The NTK-based forgetting bound for KANs is strictly tighter than for conventional MLPs in settings with approximate or strictly disjoint feature supports.
• Robustness to feature learning: The structural rank gap persists even when backbone features are evolving, due to $\theta$-independent basis functions, and is empirically preserved in the hybrid setting.

KAN-CL Algorithm

The KAN-CL+bbEWC method applies two regularizers to non-overlapping network components:

• KAN-CL (Head regularizer): For each spline coefficient, task-level Fisher and activation-mass are computed post-task and normalized before being combined into a per-knot importance weight. Downstream, gradients on each coefficient are masked and a quadratic anchor penalty is applied, tying each coefficient to its post-task value in proportion to its importance. This spatially specific regularization leverages KAN's parameter locality.
• bbEWC (Backbone regularizer): Online EWC is applied to all convolutional backbone and feature normalization parameters, without affecting the KAN head.

An extension, anchor annealing, modifies the head anchor strength in tandem with replay buffers for class-incremental learning, allowing for dynamic trade-offs between rigidity and plasticity as dictated by task difficulty.

Empirical Results

Main results on Split-CIFAR-10/5T and Split-CIFAR-100/10T demonstrate that KAN-CL+bbEWC achieves the lowest observed forgetting while matching or exceeding the best accuracy of all tested baselines. Notably, when replacing the KAN head with an MLP head while retaining the same backbone and regularizer, forgetting increases by factors of up to 37×, directly isolating the effectiveness of per-knot regularization.

Figure 1: Main results on Split-CIFAR benchmarks demonstrating KAN-CL+bbEWC (dark teal) achieves minimal forgetting and maximal or matching accuracy compared to all baselines.

Further analysis quantifies forgetting reduction relative to a head-only KAN-CL baseline, with reductions of 88% (CIFAR-10/5T) and 93% (CIFAR-100/10T).

Figure 2: Forgetting reduction of KAN-CL+bbEWC relative to KAN-CL (head only), showing dramatic gains especially on more challenging output spaces.

Replay and anchor annealing experiments show that, for class-incremental learning, dynamic adjustment of anchor strength in the presence of replay maximizes accuracy without detrimental rigidity. The method is robust across various decay rates and buffer difficulty.

Figure 3: Anchor annealing for class-IL with replay improves performance by tuning replay scale and decay, outperforming fixed regularization baselines.

Ablation studies identify the per-knot L2 anchor as the critical component—removal leads to near-full forgetting. The mask term is marginal but stabilizes early-stage gradients, and bbEWC is essential for image tasks.

Figure 4: Component ablation on representative benchmarks shows removing the anchor or backbone EWC drastically increases forgetting.

NTK analysis empirically validates the predicted reduction in cross-task coupling for KAN heads compared to MLP heads.

Figure 5: Normalized cross-task NTK norm for KAN vs. MLP head, showing KAN's consistent rank deficit especially on higher-dimension data.

Taskwise accuracy trajectories and stability-plasticity scatter plots confirm that KAN-CL+bbEWC provides flat, robust accuracy profiles across all tasks and occupies the upper-left (optimal) region of the forgetting-accuracy frontier.

Figure 6: $R[t,j]$ trajectories indicating KAN-CL+bbEWC maintains stable performance across all previously seen tasks.

Figure 7: Stability-plasticity scatter across benchmarks, with KAN-CL+bbEWC consistently achieving high accuracy and low forgetting.

Discussion and Implications

The results demonstrate that architectural design, specifically exploiting spatial parameter locality via B-splines in KANs, enables parameter-level regularization that is precise and non-intrusive. Unlike global parameter penalties, KAN-CL's per-knot anchoring protects only the coefficients actually implicated in prior tasks, preserving plasticity elsewhere. This architectural–algorithmic compositionality yields a strict Pareto improvement, achieving both high accuracy and minimal forgetting.

The NTK-based analysis both predicts and explains the observed empirical improvements, showing that per-knot regularization not only provides practical gains, but does so via a fundamentally tighter upper bound on cross-task interference. This highlights the potential for further using architectural inductive biases to inform and synergize with regularizer design.

Limitations and Future Directions:

• The capacity match between backbone and KAN head is important; scaling to larger backbones requires principled head scaling.
• The disjoint-support assumption is only approximately met in complex real-world tasks.
• Extensions to adaptive-grid KANs, larger architectures, and settings with more nuanced feature overlap represent promising avenues.
• A formal treatment of the approximate disjointness regime for empirical features is open.

Conclusion

KAN-CL+bbEWC showcases that coupling a per-knot importance regularizer with the inherent spatial locality of KANs in a hybrid CNN+KAN setting achieves state-of-the-art forgetting reduction and high accuracy, substantiated by theory and experiment. This work establishes the architectural parameterization, and not only the training objective, as a primary axis for continual learning algorithm design, suggesting fertile ground for future research.

See (2605.12306) for full theoretical derivations and experiment details.