---
title: Stochastic Gradient Kernel
url: https://www.emergentmind.com/topics/stochastic-gradient-kernel
type: topic
---

# Stochastic Gradient Kernel

Stochastic gradient kernel denotes a class of kernel objects that arise when stochastic gradient updates are expressed in function space. In the neural tangent kernel (NTK) regime, the kernel is the gradient covariance at initialization, and stochastic gradient descent on network parameters becomes stochastic gradient descent in the reproducing kernel Hilbert space (RKHS) associated with that kernel. In classical RKHS learning, the stochastic gradient itself is a kernel section of the form $K(X_t,\cdot)$, while in scalable or constrained variants the update may use a projected, truncated, mixed, or random-feature approximation of that section. Across these settings, the common role of the kernel is to determine the functional direction, regularization behavior, and statistical rate of the stochastic update [2006.12297][2510.04237][2008.10020].

## 1. Terminological scope and basic operator form

In RKHS learning with a Mercer kernel $K$ and loss $\ell$, the canonical kernel-SGD iteration is
$$
f_{t+1}=f_t-\gamma_t\,\partial_u\ell(f_t(X_{t+1}),Y_{t+1})\,K(X_{t+1},\cdot).
$$
This form appears explicitly in large-scale supervised learning with general losses, where the stochastic gradient is an unbiased estimate of the population gradient and the kernel section $K(X_{t+1},\cdot)$ is the basic update direction [2510.04237]. The same representer-form update also appears in complex RKHSs for least-squares, where
$$
f_{t+1}=f_t-\eta_t\bigl(f_t(x_t)-y_t\bigr)K(x_t,\cdot),
$$
and the distinction is not the algebraic structure of the update but the use of the complex Hilbert-space gradient [2604.23017].

A second usage occurs when the stochastic gradient is not evaluated at the current iterate. In passive stochastic approximation, one observes a random sample $X_n\sim\mu(x)$ and a noisy gradient at $X_n\neq\theta_n$, then re-centers that gradient with a kernel approximation of the Dirac mass:
$$
\theta_{n+1}=\theta_n-\alpha_n\,K_h(\theta_n-X_n)\,\hat\nabla J(X_n;\theta_n).
$$
Here the kernel is not a reproducing kernel for prediction, but a localization device that makes off-policy gradient information usable at the current parameter value [2008.10020].

A third usage appears in scalable kernel methods, where the exact kernel section is replaced by a projected or stochastic surrogate. Examples include truncated kernels on spherical harmonics, random-feature Monte Carlo approximations, and doubly stochastic functional gradients. This suggests that “stochastic gradient kernel” is best understood operationally: it is the kernel object that mediates one stochastic gradient step in function space, whether exactly, approximately, or through a limiting equivalence [2410.01570][1407.5599].

## 2. The neural tangent kernel as a gradient kernel

For overparameterized two-layer networks,
$$
f(x;\theta)=\frac1{\sqrt M}\sum_{r=1}^M a_r\,\sigma(b_r^\top x+\gamma c_r),\qquad \theta=(a,B,c),
$$
initialized at $\theta_0$, the neural tangent kernel is
$$
K_\infty(x,x')=\partial_\theta f(x;\theta_0)^\top\partial_\theta f(x';\theta_0).
$$
For smooth $\sigma$ and $R=\|a^{(0)}\|$, it has the closed form
$$
K_\infty(x,x')
= \mathbb E_{b\sim\mathrm{Unif}(S^{d-1})}[\sigma(b^\top x)\sigma(b^\top x')]
+ R^2(x^\top x'+\gamma^2)\,\mathbb E_{b\sim\mathrm{Unif}}[\sigma'(b^\top x)\sigma'(b^\top x')],
$$
which defines a positive-definite kernel on $X\times X$ [2006.12297].

In the overparameterized NTK regime, parameters barely move:
$$
\theta(t)\approx\theta_0+\Delta\theta(t),\qquad
f(x;\theta(t))\approx f(x;\theta_0)+\partial_\theta f(x;\theta_0)\,\Delta\theta(t).
$$
Gradient descent on parameters then induces a linear update on the function
$$
g^{(t+1)}(x)-g^{(t)}(x)\approx-\eta\,K_\infty(x,x_t)\,\bigl(g^{(t)}(x_t)-y_t\bigr).
$$
The training dynamics of the network therefore become equivalent to stochastic gradient descent in the RKHS $H_\infty$ associated with $K_\infty$ [2006.12297].

This equivalence is the central reason the NTK is called a gradient kernel. The kernel is not introduced as an external prior; it is induced by the parameter-gradient geometry at initialization. In this sense, the stochastic gradient kernel is the functional object that converts parameter-space SGD into an RKHS recursion.

## 3. Optimal-rate theory in the NTK regime

The convergence analysis of averaged SGD in the NTK regime is formulated through the covariance operator $\Sigma_\infty$ and two structural assumptions: the source condition $g_\rho\in\mathrm{Range}(\Sigma_\infty^r)$ with $r\in[1/2,1]$, and the spectral decay $\lambda_i(\Sigma_\infty)=\Theta(i^{-\beta})$ with $\beta>1$. Together with smoothness and boundedness assumptions—$\sigma$ is $C^2$-smooth with bounded derivatives, $\|x\|\le1$, $|y|\le1$—they yield the bound
$$
\mathbb E\bigl[\|\bar g^{(T)}-g_\rho\|_{L_2(\rho_X)}^2\bigr]
=O(\lambda^{2r})
+O\!\left(\frac{\mathrm{Tr}[\Sigma_\infty(\Sigma_\infty+\lambda I)^{-1}]}{T+1}\right).
$$
Balancing the two terms with
$$
\lambda=T^{-\beta/(2r\beta+1)}
$$
gives
$$
\mathbb E\bigl[\|\bar g^{(T)}-g_\rho\|^2\bigr]
=O\!\left(T^{-2r\beta/(2r\beta+1)}\right),
$$
which is the minimax-optimal rate in $H_\infty$ [2006.12297].

The proof decomposes the problem into four steps. First, overparameterization implies an equivalence to RKHS-SGD: for all $t\le T$,
$$
\|g_{\mathrm{nn}}^{(t)}-g_{\mathrm{rf}}^{(t)}\|_\infty\le\epsilon,
$$
where $g_{\mathrm{rf}}$ is SGD in the random-feature NTK space $H_M$. Second, averaged SGD in $H_M$ is compared with the Tikhonov solution $g_{M,\lambda}$. Third, operator perturbation bounds remove the dependence on finite width by showing $\Sigma_M\approx\Sigma_\infty$ in operator norm. Fourth, the bias term behaves like $\lambda^{2r}$ while the variance behaves like $\mathrm{Tr}[\Sigma_\infty(\Sigma_\infty+\lambda I)^{-1}]/T$, and under $\lambda_i=\Theta(i^{-\beta})$ one has
$$
\mathrm{Tr}[\Sigma_\infty(\Sigma_\infty+\lambda I)^{-1}]
=\sum_i\frac{\lambda_i}{\lambda_i+\lambda}
=O(\lambda^{-1/\beta}).
$$
The optimal rate follows by balancing these terms [2006.12297].

The same framework extends to smooth approximations of ReLU. If $\sigma^{(s)}\to\mathrm{ReLU}$ with $\sigma^{(s)}\in C^2$, then $K_\infty^{(s)}\to K_\infty$ uniformly and $\Sigma_\infty^{(s)}\to\Sigma_\infty$ in operator norm. For the ReLU NTK in $d$ dimensions, the exponent becomes $\beta=1+1/(d-1)$, giving
$$
\mathbb E\bigl[\|\bar g^{(T)}-g_\rho\|^2\bigr]
=O\!\left(T^{-\frac{2r(1+1/(d-1))}{2r(1+1/(d-1))+1}}\right),
$$
and for $r=1/2$ this simplifies to $O(T^{-d/(2d-1)})$ [2006.12297]. A practical implication stated in the same line of analysis is that a smooth surrogate such as swish can inherit the same minimax-optimal rates if it is sufficiently close to ReLU at initialization.

## 4. Projected, truncated, random-feature, and distributed variants

A major branch of stochastic gradient kernel research replaces the full RKHS update by a finite-dimensional projection. On the sphere, truncated kernel SGD defines
$$
L_n=\min\{k:\dim\Pi_k^d\ge n^\theta\},\qquad
\mathcal H_{L_n}=\bigoplus_{k=0}^{L_n}H_k^d,
$$
with truncated kernel
$$
K^T_{L_n}(x,x')=\sum_{k=0}^{L_n}(\dim\Pi_k^d)^{-2s}K_k(x,x').
$$
The update is
$$
\hat f_n=\hat f_{n-1}+\gamma_n\bigl(Y_n-\hat f_{n-1}(X_n)\bigr)K^T_{L_n}(X_n,\cdot),
$$
and the averaged iterate is $\bar f_n=\frac1{n+1}\sum_{i=0}^n\hat f_i$. With constant step size $\gamma_n=\gamma_0$, truncation exponent $\theta\le1/(4sr+1)$, and the regularity condition $f^*=L_{\omega,K}^r(u^*)$, the excess risk satisfies
$$
\mathbb E\bigl[\|\bar f_n-f^*\|^2_{\mu_X}\bigr]
=O(n^{-4sr\,\theta})+O(n^{-1+\theta}),
$$
and the optimal choice $\theta=1/(4sr+1)$ yields the minimax-optimal rate $O(n^{-4sr/(4sr+1)})$. The same construction is stated to overcome the saturation problem of classical kernel SGD, while its spherical-polynomial implementation reduces storage and computational cost to $\mathcal O(n^{\frac{d}{d-1}\epsilon})$ and $\mathcal O(n^{1+\frac{d}{d-1}\epsilon})$ for sufficiently regular problems [2410.01570].

A related general-loss framework uses spherical radial basis functions and projects each stochastic gradient onto
$$
H_{L_n}=\mathrm{span}\{Y_{k,j}:k=0,\dots,L_n\},
$$
with
$$
L_n=\min\{k:\dim\Pi_k^d\ge n^\theta\}.
$$
The projected update is
$$
g_n=f_{n-1}-\gamma_n\,\partial_u\ell(f_{n-1}(X_n),Y_n)\,K^T_{L_n}(X_n,\cdot),
\qquad f_n=P_W g_n.
$$
Under the source condition $f^*=\Sigma^r g^*$ with $r\ge1/2$, the last iterate and the suffix average achieve, up to logarithmic factors, minimax-optimal excess-risk rates $n^{-2r/(2r+1)}$, and the RKHS norm converges at rate $n^{-(2r-1)/(2r+1)}$ up to logarithmic factors. Because $\dim\Pi_{L_n}^d=O(L_n^{d-1})$, the overall time is $O(n^{1+d\theta/(d-1)})$ and the memory is $O(n^\theta)$ [2510.04237].

A second branch replaces exact kernels by random-feature or doubly stochastic surrogates. Doubly stochastic functional gradients use
$$
\zeta_t(\cdot)=\ell'(f_t(x_t),y_t)\,\phi_{\omega_t}(x_t)\phi_{\omega_t}(\cdot),
$$
which is unbiased for the risk gradient because $\mathbb E_{\omega_t}[\phi_{\omega_t}(x_t)\phi_{\omega_t}(\cdot)]=k(x_t,\cdot)$. The resulting function converges pointwise at rate $O(1/t)$ and achieves excess risk $O(1/\sqrt t)$, while avoiding storage of support vectors [1407.5599]. In the mini-batch random-feature setting, the number of features $M$, the number of SGD steps $T$, the step size $\gamma$, and the batch size $b$ jointly determine implicit regularization, and with standard capacity and source assumptions one obtains the minimax-optimal rate $n^{-2r/(2r+\alpha)}$ [1807.06343]. Distributed SGM extends the same principle to partitioned data and retains the optimal rate $N^{-2r/(2r+\gamma)}$ provided the partition level is not too large [1801.07226].

These variants all preserve the defining role of the stochastic gradient kernel while modifying its representation. The exact kernel section, truncated kernel, or random-feature surrogate is the object through which the stochastic gradient is regularized and made computationally tractable.

## 5. Stability, restricted gradients, and directional bias

In fixed-dictionary kernel adaptive filtering, the stochastic gradient kernel appears through a restricted gradient on a dictionary subspace
$$
\mathcal M=\mathrm{span}\{\kappa(\cdot,u_j):j\in\mathcal J\}\subset\mathcal H.
$$
If $G$ is the Gram matrix of the dictionary and $J(\alpha)$ is the MSE cost in coordinates, the steepest-descent direction under the induced $G$-inner product is
$$
\nabla_G J(\alpha)=G^{-1}\nabla J(\alpha).
$$
This yields the Natural KLMS update
$$
\alpha_{n+1}=\alpha_n+\eta\,e_n\,G^{-1}\kappa_n,
$$
which becomes an ordinary LMS recursion in whitened coordinates. The analysis gives mean stability if
$$
0<\eta<\frac{2}{\lambda_{\max}(\tilde R_\kappa)},
$$
and mean-square stability if the covariance recursion has spectral radius $\rho(K)<1$ [1410.3595]. The kernel here is not only a representer; through $G^{-1}$ it induces the local metric of the stochastic descent.

A different phenomenon is directional bias. In kernel regression with Gram matrix $K$, full-batch GD and one-sample SGD both interpolate, but their residual vectors align with different eigendirections. Under a diagonally-dominant Gram matrix, SGD with a moderate-and-annealing step-size schedule converges along the eigenvector corresponding to the largest eigenvalue $\gamma_1$, whereas GD with a moderate or small step size converges along the eigenvector corresponding to the smallest eigenvalue $\gamma_n$. With misalignment vector $b_t=\alpha_t-\hat\alpha$, the RKHS generalization error is
$$
\Delta(f)=b^\top K b.
$$
For fixed training loss, this error is minimized when the residual aligns with the top-eigenvector, which the analysis uses to explain why the SGD estimator can have smaller generalization error than the GD estimator [2205.00061].

The same bias structure extends to complex RKHSs. Under unbiasedness, bounded variance, lower-bounded objective, smoothness, and convexity assumptions, the averaged iterate of Complex SGD satisfies
$$
\mathbb E[F(\bar f_T)]-F^*\le
\frac{\|f_0-f^*\|_{\mathcal H}^2}{\sum_{t=0}^{T-1}\eta_t}
+\frac{2\sigma^2\sum_{t=0}^{T-1}\eta_t^2}{\sum_{t=0}^{T-1}\eta_t},
$$
and with $\eta_t=c/\sqrt T$ this gives $O(1/\sqrt T)$. In the strongly convex case,
$$
\mathbb E\|f_t-f^*\|_{\mathcal H}^2
\le (1-\eta\mu)^t\|f_0-f^*\|^2+\frac{2\sigma^2\eta}{\mu},
$$
so the iterates exhibit exponential-type convergence to an $O(\eta)$ neighborhood. The same work states that directional bias results known in the real setting extend to kernel regression in complex RKHSs [2604.23017].

## 6. Applications, extensions, and recurring misconceptions

Kernelized stochastic-gradient constructions appear well beyond standard regression. In passive stochastic approximation, multi-kernel mixtures
$$
K_{\mathrm{multi}}(x;\theta)=\sum_{j=1}^m w_j K_{h_j}(x-\theta)
$$
are used with self-normalized importance weights to aggregate a batch of noisy gradients sampled at random locations. Under weak-limit conditions, the continuous-time interpolation converges to the ideal ODE
$$
\frac{d\theta}{dt}=-\nabla J(\theta),
$$
so the limit no longer depends on the sampling density or on the individual kernels; the diffusion approximation yields $\|\theta_n-\theta^*\|=O_p(\sqrt\alpha)$ for constant step size and $\mathbb E\|\theta_n-\theta^*\|^2=O(1/n)$ for decaying step size [2008.10020].

In quantum kernel alignment, Pegasos is used to learn both the SVM decision boundary and the kernel parameters simultaneously. With trainable quantum feature map $\psi_\theta(x)$, the induced kernel is
$$
K_\theta(x,x')=\bigl|\langle\psi_\theta(x)\mid\psi_\theta(x')\rangle\bigr|^2,
$$
and the stochastic procedure updates the primal SVM weights and the circuit parameters in one loop. The reported comparison states that this reduces quantum-circuit calls from $O(M^{4.67}/\epsilon^2)$ to $O(M/\epsilon)$, up to shot-noise factors, and naturally supports non-stationary data through support-vector forgetting [2304.09899].

In non-adversarial generative modeling, kernel-based training minimizes
$$
J(\theta)=\mathbb E\!\left[k(G_\theta(Z^1),G_\theta(Z^2))-k(G_\theta(Z^1),X)-k(G_\theta(Z^2),X)\right],
$$
a pure minimization problem rather than a min-max one. The corresponding stochastic gradient update is standard SGD on $\theta$, and the stochastic-approximation analysis states that batch averaging, gradient smoothing, and delayed-parameter variants do not improve the leading-order convergence behavior or the steady-state covariance [1811.09568].

A recurring misconception is that kernel SGD is intrinsically saturation-limited or that iterate averaging is always the statistically preferred strategy. The spherical T-kernel analysis states that constant-step-size truncated kernel SGD overcomes the inherent saturation problem of kernel SGD [2410.01570]. The learning-curve analysis for kernel regression on the sphere similarly states that one-pass SGD with an exponentially decaying step-size schedule achieves minimax-optimal rates up to constants across scales and avoids saturation except in a final stage of highly misspecified learning, whereas averaged SGD behaves like a qualification-$1$ filter and can saturate when the smoothness parameter is large [2505.22048]. These results do not eliminate the importance of averaging; rather, they show that the statistical role of a stochastic gradient kernel depends on the spectral filter implemented by the update schedule, truncation rule, and representation of the kernel itself.

Source: https://www.emergentmind.com/topics/stochastic-gradient-kernel