MetaCluster: Enabling Deep Compression of Kolmogorov-Arnold Network (2510.19105v1)

Abstract: Kolmogorov-Arnold Networks (KANs) replace scalar weights with per-edge vectors of basis coefficients, thereby boosting expressivity and accuracy but at the same time resulting in a multiplicative increase in parameters and memory. We propose MetaCluster, a framework that makes KANs highly compressible without sacrificing accuracy. Specifically, a lightweight meta-learner, trained jointly with the KAN, is used to map low-dimensional embedding to coefficient vectors, shaping them to lie on a low-dimensional manifold that is amenable to clustering. We then run K-means in coefficient space and replace per-edge vectors with shared centroids. Afterwards, the meta-learner can be discarded, and a brief fine-tuning of the centroid codebook recovers any residual accuracy loss. The resulting model stores only a small codebook and per-edge indices, exploiting the vector nature of KAN parameters to amortize storage across multiple coefficients. On MNIST, CIFAR-10, and CIFAR-100, across standard KANs and ConvKANs using multiple basis functions, MetaCluster achieves a reduction of up to 80$\times$ in parameter storage, with no loss in accuracy. Code will be released upon publication.

• The paper introduces a meta-learning framework that compresses Kolmogorov-Arnold Networks by clustering coefficient vectors to achieve up to 80× reduction in storage.
• It employs a lightweight meta-learner to map high-dimensional coefficients onto a low-dimensional manifold, facilitating efficient K-means clustering.
• Experimental results on MNIST, CIFAR-10, and CIFAR-100 demonstrate that MetaCluster maintains accuracy even in memory-constrained scenarios.

MetaCluster: Deep Compression for Kolmogorov-Arnold Networks

Introduction

Kolmogorov-Arnold Networks (KANs) have demonstrated superior expressivity and accuracy over traditional MLPs by parameterizing each edge with a vector of basis coefficients, such as B-splines or RBFs. However, this design incurs a significant parameter and memory overhead, impeding their scalability in large-scale applications. The MetaCluster framework addresses this bottleneck by introducing a meta-learning-based manifold shaping strategy, enabling effective weight sharing and deep compression of KANs without sacrificing accuracy.

MetaCluster Framework

MetaCluster operates in three stages: (1) manifold shaping via a meta-learner, (2) clustering of coefficient vectors, and (3) fine-tuning post-compression.

Manifold Shaping with Meta-Learner

A lightweight meta-learner $M_\theta$ is trained jointly with the KAN to map low-dimensional embeddings $\mathbf{z}_i \in \mathbb{R}^{d_{emb}}$ to high-dimensional coefficient vectors $\mathbf{w}_i \in \mathbb{R}^{|\mathbf{w}|}$. This constrains the coefficients to a low-dimensional manifold, making them highly amenable to clustering. The meta-learner is parameterized as a two-layer MLP:

$$M_\theta(\mathbf{z}_i) = \mathbf{W}_2 \sigma(\mathbf{W}_1 \mathbf{z}_i + b_1) + b_2$$

Empirical t-SNE visualizations confirm that the meta-learner organizes the coefficient vectors into structured manifolds, facilitating downstream clustering.

Figure 1: t-SNE visualization of KAN activation weights; meta-learner with $d_{emb}=1$ yields a 1D manifold, $d_{emb}=2$ yields a 2D sheet, while baseline KAN weights are unstructured.

K-means Clustering in Coefficient Space

After training, K-means is applied to the set of coefficient vectors, clustering them into $k$ centroids in $\mathbb{R}^{|\mathbf{w}|}$. Each edge's coefficients are replaced by the nearest centroid, and only the codebook and per-edge indices are stored. The clustering objective is:

$$\arg\min_{\mathcal C} \sum_{i=1}^{k} \sum_{M_{\theta}(\mathbf{z})\in C_i} \|M_{\theta}(\mathbf{z}) - \mathbf{c}_i\|_2^2$$

This approach amortizes the codebook storage over multiple coefficients, yielding a much higher compression factor than scalar weight sharing in MLPs.

Fine-tuning for Accuracy Recovery

The meta-learner and embeddings are discarded post-clustering. A brief fine-tuning of the centroid codebook is performed to recover any residual accuracy loss, typically requiring only a few epochs.

Compression Analysis

The compression factor $r$ for KANs is:

$$r = \frac{n |\mathbf{w}| b}{n\log_2(k) + |\mathbf{w}| k b}$$

where $n$ is the number of edges, $|\mathbf{w}|$ is the coefficient vector size, $k$ is the number of centroids, and $b$ is the bit-width. The key insight is that the index cost $n\log_2(k)$ is amortized over $|\mathbf{w}|$ coefficients per centroid, making KANs particularly well-suited for weight sharing.

Experimental Results

MetaCluster was evaluated on MNIST, CIFAR-10, and CIFAR-100 using both fully-connected and convolutional KAN architectures with B-spline, RBF, and Gram polynomial bases. Across 24 model schemes, MetaCluster consistently achieved up to $80\times$ reduction in parameter storage with no loss in accuracy.

• On CIFAR-10, MetaClusterKAN reduced memory from 3,064.95 KB to 38.32 KB (compression factor $79.9\times$) while matching the accuracy of the uncompressed model.
• In convolutional settings, MetaClusterKANConv achieved a $31.7\times$ reduction (from 13.7 MB to 435.34 KB) with negligible accuracy impact.

Ablation studies confirmed that manifold shaping is essential for high-quality clustering. Increasing the meta-learner embedding dimension degrades clusterability and post-compression accuracy, while the framework remains robust to variations in basis coefficient count and cluster count.

(Figure 2)

Figure 2: Classification accuracy vs. number of clusters for FastKAN variants; MetaCluster maintains accuracy as cluster count decreases, outperforming naive clustering.

Practical and Theoretical Implications

MetaCluster enables the deployment of KANs in memory-constrained environments and large-scale settings where parameter overhead previously limited their use. The framework is robust across architectures and basis functions, and its design is compatible with further quantization techniques for additional compression. The meta-learner is only required during training, incurring zero inference overhead.

Theoretically, MetaCluster demonstrates that meta-learned manifold shaping can overcome the curse of dimensionality in high-dimensional clustering, a challenge that has stymied previous attempts at weight sharing for KANs. The approach generalizes to other architectures with per-edge vector parameters and can be extended to multi-task or multi-modal settings.

Future Directions

Potential future developments include:

• Integration with differentiable clustering (e.g., DKM) for end-to-end joint optimization.
• Application to other high-dimensional parameterized architectures (e.g., hypernetworks, rational KANs).
• Exploration of adaptive embedding dimension selection and dynamic codebook sizing.
• Hardware-aware compression strategies leveraging MetaCluster for efficient deployment on edge devices and accelerators.

Conclusion

MetaCluster provides an effective solution to the memory inefficiency of Kolmogorov-Arnold Networks by combining meta-learned manifold shaping with weight sharing. It achieves substantial compression—up to $80\times$—without accuracy degradation, making KANs practical for large-scale and resource-constrained applications. The framework's generality and compatibility with further quantization position it as a foundational technique for future neural network compression research.