KANs need curvature: penalties for compositional smoothness

Abstract: Kolmogorov-Arnold networks (KANs) offer a potent combination of accuracy and interpretability, thanks to their compositions of learnable univariate activation functions. However, the activations of well-fitting KANs tend to exhibit pathologically high-curvature oscillations, making them difficult to interpret, and standard regularization penalties do not prevent this. Here we derive a basis-agnostic curvature penalty and show that penalized models can maintain accuracy while achieving substantially smoother activations. Accounting for how function composition shapes curvature, we prove an upper bound on the full model's curvature relative to the curvature penalty, and use this to motivate richer forms of penalties. Scientific machine learning is increasingly bottlenecked by the trade-off between accuracy and interpretability. Results such as ours that improve interpretability without sacrificing accuracy will further strengthen KANs as a practical tool for both prediction and insight.

• The paper introduces a principled curvature penalty that directly targets activation smoothness in KANs.
• It demonstrates that curvature penalties achieve an order of magnitude reduction in activation roughness while preserving accuracy.
• The approach stabilizes training in overparameterized regimes and generalizes to multiple activation parameterizations.

Curvature Penalties for Compositional Smoothness in Kolmogorov–Arnold Networks

Introduction and Motivation

Kolmogorov–Arnold Networks (KANs) provide a compositional alternative to standard MLPs by replacing fixed nonlinear activation functions with learnable univariate splines on each network edge. This architectural choice enhances interpretability because every activation can be directly inspected, and, under appropriate structural conditions and training, these networks deliver comparable accuracy to traditional architectures. However, one of the central challenges in practical KAN adoption is that, even in cases of excellent empirical fit, the learned activations often show pathological, high-curvature, kink-like oscillations. Such activations hinder interpretability, as their visual and analytic forms do not closely resemble the atomic functional building blocks present in ground truth structure.

The paper "KANs need curvature: penalties for compositional smoothness" (2605.02190) addresses this challenge by formally analyzing why standard KAN regularization fails to enforce activation smoothness, deriving principled curvature penalties in function space, and rigorously connecting edgewise penalties to the smoothness of the composite KAN function. Furthermore, the work provides comprehensive empirical evidence that curvature-penalized KANs preserve accuracy while yielding much smoother—and thus more interpretable—activations, and stabilizing optimization in overparameterized regimes.

Failure of Existing Penalties and Formulation of a Curvature Penalty

Standard KAN regularization, including the PyKAN entropy regularizer, penalizes the average or $L^1$ norm of activations and their distributional concentration across network edges, but remains agnostic to curvature. As a result, high-frequency activation oscillations incur no additional penalization relative to smooth activations of the same mean magnitude, leading to functionally accurate but compositionally uninterpretable solutions.

Figure 1: On the target $\sin(x + y^2)$, unpenalized (A) and KAN-penalized (B) models yield oscillatory, non-smooth activations unlike ground truth; curvature-penalized models (C) yield smooth, interpretable activations.

To directly promote smoothness, the paper derives an edgewise curvature penalty based on the univariate bending energy (squared $H^2$ seminorm) integrated over each activation's support. For B-spline parameterizations, this penalty reduces to a quadratic form in the spline coefficients, plus an $L^2$ penalization of the base function (e.g., SiLU) coefficient:

$$R(f) = \sum_e \left( \| D_2(\alpha_e c_e) \|^2 + K_\mathrm{SiLU} \, \beta_e^2 \right)$$

This penalty is basis-agnostic, relying only on the differentiability and the Gram matrix of the basis, and can thus be generalized to other functional parameterizations including Gaussian RBFs.

Empirical Evaluation: Smoothness–Accuracy Tradeoffs

The curvature-penalized KANs are systematically evaluated across canonical function approximation and benchmark tasks. Empirical results conclusively demonstrate that these models maintain fitting accuracy while producing substantially smoother activations. For the benchmark $f(x, y) = \exp(\sin(\pi x) + y^2)$, curvature-penalized KANs achieve test RMSE within a factor of two of the unpenalized baseline across a large range of penalty weights, while reducing total activation curvature by more than an order of magnitude.

Figure 2: Curvature penalties bias optimization toward smoother activation basins without significant accuracy loss.

Additionally, in regimes of significant overparameterization (large grid size $G$), standard training is unstable. The curvature penalty regularizes these models and allows stable, end-to-end optimization, outperforming the standard PyKAN penalty in test error and smoothness. Notably, curvature penalization makes traditional grid-extension curricula—previously needed to control solution smoothness—no longer necessary for achieving accurate, smooth solutions at high capacity.

Figure 3: In overparameterized KANs ($G=200$), the curvature penalty yields lower test RMSE across all $\lambda$, stabilizing optimization and model selection.

Further, the penalty's utility generalizes to different optimizers (e.g., Adam, L-BFGS), network capacities, and alternative basis parameterizations (see App.~B), underlining the approach’s robustness and architectural generality.

Figure 4: Applying the curvature penalty improves training and generalization across optimizers and architectures.

Theoretical Analysis: Compositional Curvature and Upper Bounds

One of the key formal contributions is the demonstration that naively penalizing the curvature of individual activations is not equivalent to constraining the curvature of the composite KAN function. Using chain rule analysis, the paper proves that the input Hessian of a compositional KAN can be expressed as sums over edgewise Hessians modulated by upstream and downstream Jacobians. While edgewise curvature contributes to overall function curvature, compositional effects can amplify roughness when substantial intermediate gradient norms are present.

The authors derive upper bounds placing the sum of edgewise curvature penalties as a principled upper bound on the mean input-Hessian norm of the full composed network, subject to mild distributional and architectural conditions. This yields a rigorous connection between the proxy regularization and the true function-space smoothness desideratum.

Weighted Curvature Penalty and Data-Dependent Refinement

Since the compositional upper bound analysis introduces data-dependent path weights reflecting the impact of each edge’s curvature on the output, the authors subsequently propose a richer, weighted curvature penalty that retains this path sensitivity. Empirical studies confirm that, with optimal $\lambda$, the weighted penalty further halves the average test RMSE compared to the uniform penalty across multiple seeds.

Figure 5: Weighted curvature penalty reduces mean test RMSE by $\sin(x + y^2)$0 compared to the unweighted approach.

Extension to Other Bases

The penalty generalizes to activation parameterizations beyond B-splines. For example, experiments with Gaussian-RBF-based FastKANs show that the proposed penalty controls curvature and restores trainability and accuracy in regimes where standard training fails due to overparameterization.

Figure 6: Curvature penalty smooths and stabilizes activation functions in high-resolution RBF-based FastKANs.

Implications and Future Directions

The introduction of principled, basis-agnostic curvature penalties for KANs has multiple practical and theoretical implications. First, it enables transparent activations—bringing KANs closer to their goal of interpretable neural function modeling—without requiring cumbersome multi-stage training. Moreover, because the penalty is expressed in model coefficients rather than sample-dependent function evaluations, it remains flexible and compatible with both batch and streaming training regimes, as well as with neural architecture search and meta-learning. The compositional analysis and the diagonal structure of KAN Hessians further facilitate a deeper understanding of architectural smoothness constraints and their role in function composition, inviting extension to more complex settings, e.g., adaptive grids or higher-order regularization forms.

Future research may refine the relationship between compositional smoothness and task-specific generalization, explore non-quadratic penalties (e.g., $\sin(x + y^2)$1 variants for kink-tolerance), and design adaptive or automatically-tuned curvature priors based on data geometry and optimization feedback.

Conclusion

This work rigorously identifies and addresses the failure of conventional KAN regularization to yield compositional smoothness, derives a principled curvature penalty applicable to a broad class of activation parameterizations, proves its role in upper bounding overall function curvature, and demonstrates its superior empirical performance in accuracy, interpretability, and trainability across a range of tasks and architectures. With the introduction of this penalty, KANs encapsulate both powerful compositional expressivity and interpretable smoothness, reinforcing their suitability for scientific and interpretable machine learning tasks.