Iterative Low-Rank Kernel Updates

• Iterative low-rank kernel updates are algorithmic strategies that enforce low-dimensional kernel representations via spectral dynamics and rank constraints.
• They employ methods such as spectral ODEs, nuclear norm minimization, and ADMM to iteratively update kernels for robust, efficient learning.
• These techniques yield provable rank compression, resistance to SGD noise, and scalable solutions for neural network training, regression, and graph-based clustering.

Iterative low-rank kernel updates are algorithmic strategies that exploit spectral and optimization structure to enforce and maintain low-rank representations throughout kernel-based learning, clustering, and spectral filtering. Motivated by the prohibitive cost and statistical redundancy of generic positive-definite kernel matrices, these frameworks use explicit rank constraints or spectral dynamics to iteratively update the kernel in a low-dimensional subspace aligned with either supervised labels, geometric structure, or task-induced manifolds. Core mechanisms include spectral ODEs, nuclear norm minimization, Cholesky factorizations, and alternating minimization schemes. This approach has led to provably compressive dynamics in wide, regularized neural networks, efficient kernel approximation in multitask regression, and scalable graph-based clustering.

1. Spectral Evolution and Low-Rank Steady States in Supervised Learning

Under supervised training of wide, ℓ₂-regularized neural models, the kernel (e.g., Neural Tangent Kernel) evolves according to a deterministic matrix ODE of Riccati type. In this framework, the kernel $K(t)\in\mathbb R^{N\times N}$ evolves as

$$\dot{K}(t) = \lambda\left[(K+\lambda I)^{-1}M_Y(K+\lambda I)^{-1}K + K(K+\lambda I)^{-1}M_Y(K+\lambda I)^{-1}\right] - 2\mu K,$$

where $M_Y = Y Y^T$, $Y$ is the label matrix, $\lambda$ is the ridge parameter, and $\mu$ is the feature decay (Li et al., 1 Jan 2026).

The steady-state solution induces exact spectral pruning: the "water-filling" law sets all kernel eigenvalues $k_i$ with label-gram eigenvalue $\sigma_i \leq \tau := \lambda\mu$ to zero, while stronger modes take the closed form

$$k_i = \lambda\left(\sqrt{\frac{\sigma_i}{\lambda\mu}-1}\right)_+.$$

This mechanism provably compresses the rank of $K$ to at most $$\dot{K}(t) = \lambda\left[(K+\lambda I)^{-1}M_Y(K+\lambda I)^{-1}K + K(K+\lambda I)^{-1}M_Y(K+\lambda I)^{-1}\right] - 2\mu K,$$0—the number of supervised classes—revealing that supervised learning dynamics are inherently compressive and label-aligned.

2. Discretized Iterative Low-Rank Kernel Update Algorithms

To implement this kernel evolution in practice, the Riccati flow is discretized in the eigenbasis of $$\dot{K}(t) = \lambda\left[(K+\lambda I)^{-1}M_Y(K+\lambda I)^{-1}K + K(K+\lambda I)^{-1}M_Y(K+\lambda I)^{-1}\right] - 2\mu K,$$1, requiring only the tracking of $$\dot{K}(t) = \lambda\left[(K+\lambda I)^{-1}M_Y(K+\lambda I)^{-1}K + K(K+\lambda I)^{-1}M_Y(K+\lambda I)^{-1}\right] - 2\mu K,$$2 for those $$\dot{K}(t) = \lambda\left[(K+\lambda I)^{-1}M_Y(K+\lambda I)^{-1}K + K(K+\lambda I)^{-1}M_Y(K+\lambda I)^{-1}\right] - 2\mu K,$$3. Explicit Euler updates take the form

$$\dot{K}(t) = \lambda\left[(K+\lambda I)^{-1}M_Y(K+\lambda I)^{-1}K + K(K+\lambda I)^{-1}M_Y(K+\lambda I)^{-1}\right] - 2\mu K,$$4

with subsequent projection onto the nonnegative orthant and hard-thresholding of near-zero eigenvalues. The low-rank kernel is reconstructed as $$\dot{K}(t) = \lambda\left[(K+\lambda I)^{-1}M_Y(K+\lambda I)^{-1}K + K(K+\lambda I)^{-1}M_Y(K+\lambda I)^{-1}\right] - 2\mu K,$$5, where $$\dot{K}(t) = \lambda\left[(K+\lambda I)^{-1}M_Y(K+\lambda I)^{-1}K + K(K+\lambda I)^{-1}M_Y(K+\lambda I)^{-1}\right] - 2\mu K,$$6 collects the top eigenvectors of $$\dot{K}(t) = \lambda\left[(K+\lambda I)^{-1}M_Y(K+\lambda I)^{-1}K + K(K+\lambda I)^{-1}M_Y(K+\lambda I)^{-1}\right] - 2\mu K,$$7. This iterative scheme maintains $$\dot{K}(t) = \lambda\left[(K+\lambda I)^{-1}M_Y(K+\lambda I)^{-1}K + K(K+\lambda I)^{-1}M_Y(K+\lambda I)^{-1}\right] - 2\mu K,$$8 computational and storage burden per update and is robust to SGD noise, which is also spectrally confined to the label-induced $$\dot{K}(t) = \lambda\left[(K+\lambda I)^{-1}M_Y(K+\lambda I)^{-1}K + K(K+\lambda I)^{-1}M_Y(K+\lambda I)^{-1}\right] - 2\mu K,$$9 subspace (Li et al., 1 Jan 2026).

3. Incomplete Cholesky and Least-Angle Regression for Predictive Kernel Approximation

In multi-kernel regression, the Mklaren algorithm employs incomplete Cholesky factorizations with a least-angle regression (LAR) selection criterion to construct a low-rank approximation of multiple kernel matrices without explicitly forming their dense representations (Stražar et al., 2016). At each iteration, Mklaren selects a kernel and pivot via the LAR criterion, performs a Cholesky column update, and appends the resulting normalized feature to a combined matrix $M_Y = Y Y^T$0, which spans the active regression subspace.

The method maintains and updates per-kernel factorizations $M_Y = Y Y^T$1, a combined feature matrix $M_Y = Y Y^T$2, and regression coefficients $M_Y = Y Y^T$3 iteratively:

• Pivot selection is guided by maximizing predictive correlation with the residual.
• Column updates are performed only as needed, leveraging look-ahead pivots for efficiency.
• Feature expansion continues until a prescribed rank or convergence is achieved.

This framework has linear complexity in the number of data points and kernels when the final rank is moderate, providing scalable kernel learning for large datasets.

4. ADMM-Driven Low-Rank Kernel Updates in Graph-Based Clustering

In graph-based clustering, iterative low-rank kernel learning is effected through an alternating direction method of multipliers (ADMM) scheme that couples the learning of the graph adjacency matrix $M_Y = Y Y^T$4 and a consensus kernel $M_Y = Y Y^T$5, both encouraged to be low-rank via nuclear norm penalties (Kang et al., 2019). Given a set of $M_Y = Y Y^T$6 base kernels $M_Y = Y Y^T$7, the unified objective optimizes:

$M_Y = Y Y^T$8

subject to $M_Y = Y Y^T$9.

Each ADMM cycle sequentially updates $Y$0, $Y$1, $Y$2, $Y$3 (auxiliary variables), $Y$4, and dual variables $Y$5, $Y$6 by solving convex subproblems, including closed-form updates, proximal (singular value thresholding) steps for nuclear norms, and per-iteration quadratic programs for $Y$7.

This structure supports explicit enforcement of low rank at each step (via soft-thresholding on singular values), guarantees convergence under mild conditions, and empirically yields scalable performance for $Y$8 up to a few thousand.

5. Laplacian Spectral Filtering and Semi-Supervised Generalizations

Extensions to semi-supervised and self-supervised learning replace the label-gram $Y$9 with a graph Laplacian $\lambda$0 to drive spectral filtering. The minimization of

$\lambda$1

yields the solution (in the Laplacian eigenbasis):

$\lambda$2

where $\lambda$3 are Laplacian eigenvalues. This produces a high-rank spectral filter that retains only low-frequency (smooth) graph modes (Li et al., 1 Jan 2026).

This generalization unifies supervised label-driven low-rank kernel learning with unsupervised manifold learning, allowing for hybrid models that share the iterative update core but operate in different spectral domains.

6. Algorithmic Summaries and Computational Considerations

A summary table of the principal iterative low-rank kernel update schemes is provided below:

Framework	Core Update Mechanism	Low-Rank Enforcement
Task-Driven Kernel ODE (Li et al., 1 Jan 2026)	Spectral Riccati ODE + Euler discretization	Water-filling spectral law, projection, rank ≤ C
Mklaren (Stražar et al., 2016)	Incomplete Cholesky + LAR	Explicit column updates, active dimensionality
LKG-ADMM (Kang et al., 2019)	ADMM with nuclear norm and SVD	Singular value thresholding, explicit nuclear norm

Complexities per iteration are $\lambda$4 for the Riccati-flow-based method, $\lambda$5 in Mklaren (with $\lambda$6 total rank, $\lambda$7 look-ahead), and $\lambda$8 for LKG-ADMM dominated by SVD and matrix inversion. For larger-scale data ($\lambda$9), further approximation (e.g., Nyström, randomized SVD) is commonly required.

7. Noise Structure, Robustness, and Limitations

In supervised kernel evolution, SGD-induced noise is also spectrally low-rank, with the covariance of the instantaneous noise bounded by twice the number of classes: $\mu$0 (Li et al., 1 Jan 2026). Thus, gradient noise cannot excite directions orthogonal to the label-driven task subspace, reinforcing the effectiveness of low-rank updates and their robustness to stochastic training dynamics.

A plausible implication is that, in properly regularized, wide networks or iterative kernel schemes, complexity and memory requirements can be sharply reduced without significant predictive loss—provided the data admits a compressive target structure. However, for high-rank or truly multimodal tasks (e.g., self-supervised contexts), the spectral pruning may excessively restrict representation power. Extensions using graph Laplacians recover the ability to work in higher-rank or smooth-manifold settings.

---

References:

• "Task-Driven Kernel Flows: Label Rank Compression and Laplacian Spectral Filtering" (Li et al., 1 Jan 2026)
• "Learning the kernel matrix via predictive low-rank approximations" (Stražar et al., 2016)
• "Low-rank Kernel Learning for Graph-based Clustering" (Kang et al., 2019)