Stochastic Gradient Kernel

• Stochastic gradient kernel is defined as the kernel object that mediates one stochastic gradient step in function space, converting parameter-space SGD into an RKHS recursion.
• It plays a central role in both classical and scalable RKHS learning by leveraging kernel sections (exact, projected, or approximate) for effective regularization and convergence.
• Analysis of these kernels in the NTK regime provides minimax-optimal rates, insights into stability, convergence dynamics, and the impact of directional bias in gradient descent.

Stochastic gradient kernel denotes a class of kernel objects that arise when stochastic gradient updates are expressed in function space. In the neural tangent kernel (NTK) regime, the kernel is the gradient covariance at initialization, and stochastic gradient descent on network parameters becomes stochastic gradient descent in the reproducing kernel Hilbert space (RKHS) associated with that kernel. In classical RKHS learning, the stochastic gradient itself is a kernel section of the form $K(X_t,\cdot)$, while in scalable or constrained variants the update may use a projected, truncated, mixed, or random-feature approximation of that section. Across these settings, the common role of the kernel is to determine the functional direction, regularization behavior, and statistical rate of the stochastic update (Nitanda et al., 2020, Bai et al., 5 Oct 2025, Krishnamurthy et al., 2020).

1. Terminological scope and basic operator form

In RKHS learning with a Mercer kernel $K$ and loss $\ell$, the canonical kernel-SGD iteration is

$$f_{t+1}=f_t-\gamma_t\,\partial_u\ell(f_t(X_{t+1}),Y_{t+1})\,K(X_{t+1},\cdot).$$

This form appears explicitly in large-scale supervised learning with general losses, where the stochastic gradient is an unbiased estimate of the population gradient and the kernel section $K(X_{t+1},\cdot)$ is the basic update direction (Bai et al., 5 Oct 2025). The same representer-form update also appears in complex RKHSs for least-squares, where

$$f_{t+1}=f_t-\eta_t\bigl(f_t(x_t)-y_t\bigr)K(x_t,\cdot),$$

and the distinction is not the algebraic structure of the update but the use of the complex Hilbert-space gradient (Alpay et al., 24 Apr 2026).

A second usage occurs when the stochastic gradient is not evaluated at the current iterate. In passive stochastic approximation, one observes a random sample $X_n\sim\mu(x)$ and a noisy gradient at $X_n\neq\theta_n$, then re-centers that gradient with a kernel approximation of the Dirac mass:

$$\theta_{n+1}=\theta_n-\alpha_n\,K_h(\theta_n-X_n)\,\hat\nabla J(X_n;\theta_n).$$

Here the kernel is not a reproducing kernel for prediction, but a localization device that makes off-policy gradient information usable at the current parameter value (Krishnamurthy et al., 2020).

A third usage appears in scalable kernel methods, where the exact kernel section is replaced by a projected or stochastic surrogate. Examples include truncated kernels on spherical harmonics, random-feature Monte Carlo approximations, and doubly stochastic functional gradients. This suggests that “stochastic gradient kernel” is best understood operationally: it is the kernel object that mediates one stochastic gradient step in function space, whether exactly, approximately, or through a limiting equivalence (Bai et al., 2024, Dai et al., 2014).

2. The neural tangent kernel as a gradient kernel

For overparameterized two-layer networks,

$$f(x;\theta)=\frac1{\sqrt M}\sum_{r=1}^M a_r\,\sigma(b_r^\top x+\gamma c_r),\qquad \theta=(a,B,c),$$

initialized at $K$0, the neural tangent kernel is

$K$1

For smooth $K$2 and $K$3, it has the closed form

$K$4

which defines a positive-definite kernel on $K$5 (Nitanda et al., 2020).

In the overparameterized NTK regime, parameters barely move:

$K$6

Gradient descent on parameters then induces a linear update on the function

$K$7

The training dynamics of the network therefore become equivalent to stochastic gradient descent in the RKHS $K$8 associated with $K$9 (Nitanda et al., 2020).

This equivalence is the central reason the NTK is called a gradient kernel. The kernel is not introduced as an external prior; it is induced by the parameter-gradient geometry at initialization. In this sense, the stochastic gradient kernel is the functional object that converts parameter-space SGD into an RKHS recursion.

3. Optimal-rate theory in the NTK regime

The convergence analysis of averaged SGD in the NTK regime is formulated through the covariance operator $\ell$0 and two structural assumptions: the source condition $\ell$1 with $\ell$2, and the spectral decay $\ell$3 with $\ell$4. Together with smoothness and boundedness assumptions—$\ell$5 is $\ell$6-smooth with bounded derivatives, $\ell$7, $\ell$8—they yield the bound

$\ell$9

Balancing the two terms with

$$f_{t+1}=f_t-\gamma_t\,\partial_u\ell(f_t(X_{t+1}),Y_{t+1})\,K(X_{t+1},\cdot).$$0

gives

$$f_{t+1}=f_t-\gamma_t\,\partial_u\ell(f_t(X_{t+1}),Y_{t+1})\,K(X_{t+1},\cdot).$$1

which is the minimax-optimal rate in $$f_{t+1}=f_t-\gamma_t\,\partial_u\ell(f_t(X_{t+1}),Y_{t+1})\,K(X_{t+1},\cdot).$$2 (Nitanda et al., 2020).

The proof decomposes the problem into four steps. First, overparameterization implies an equivalence to RKHS-SGD: for all $$f_{t+1}=f_t-\gamma_t\,\partial_u\ell(f_t(X_{t+1}),Y_{t+1})\,K(X_{t+1},\cdot).$$3,

$$f_{t+1}=f_t-\gamma_t\,\partial_u\ell(f_t(X_{t+1}),Y_{t+1})\,K(X_{t+1},\cdot).$$4

where $$f_{t+1}=f_t-\gamma_t\,\partial_u\ell(f_t(X_{t+1}),Y_{t+1})\,K(X_{t+1},\cdot).$$5 is SGD in the random-feature NTK space $$f_{t+1}=f_t-\gamma_t\,\partial_u\ell(f_t(X_{t+1}),Y_{t+1})\,K(X_{t+1},\cdot).$$6. Second, averaged SGD in $$f_{t+1}=f_t-\gamma_t\,\partial_u\ell(f_t(X_{t+1}),Y_{t+1})\,K(X_{t+1},\cdot).$$7 is compared with the Tikhonov solution $$f_{t+1}=f_t-\gamma_t\,\partial_u\ell(f_t(X_{t+1}),Y_{t+1})\,K(X_{t+1},\cdot).$$8. Third, operator perturbation bounds remove the dependence on finite width by showing $$f_{t+1}=f_t-\gamma_t\,\partial_u\ell(f_t(X_{t+1}),Y_{t+1})\,K(X_{t+1},\cdot).$$9 in operator norm. Fourth, the bias term behaves like $K(X_{t+1},\cdot)$0 while the variance behaves like $K(X_{t+1},\cdot)$1, and under $K(X_{t+1},\cdot)$2 one has

$K(X_{t+1},\cdot)$3

The optimal rate follows by balancing these terms (Nitanda et al., 2020).

The same framework extends to smooth approximations of ReLU. If $K(X_{t+1},\cdot)$4 with $K(X_{t+1},\cdot)$5, then $K(X_{t+1},\cdot)$6 uniformly and $K(X_{t+1},\cdot)$7 in operator norm. For the ReLU NTK in $K(X_{t+1},\cdot)$8 dimensions, the exponent becomes $K(X_{t+1},\cdot)$9, giving

$$f_{t+1}=f_t-\eta_t\bigl(f_t(x_t)-y_t\bigr)K(x_t,\cdot),$$0

and for $$f_{t+1}=f_t-\eta_t\bigl(f_t(x_t)-y_t\bigr)K(x_t,\cdot),$$1 this simplifies to $$f_{t+1}=f_t-\eta_t\bigl(f_t(x_t)-y_t\bigr)K(x_t,\cdot),$$2 (Nitanda et al., 2020). A practical implication stated in the same line of analysis is that a smooth surrogate such as swish can inherit the same minimax-optimal rates if it is sufficiently close to ReLU at initialization.

4. Projected, truncated, random-feature, and distributed variants

A major branch of stochastic gradient kernel research replaces the full RKHS update by a finite-dimensional projection. On the sphere, truncated kernel SGD defines

$$f_{t+1}=f_t-\eta_t\bigl(f_t(x_t)-y_t\bigr)K(x_t,\cdot),$$3

with truncated kernel

$$f_{t+1}=f_t-\eta_t\bigl(f_t(x_t)-y_t\bigr)K(x_t,\cdot),$$4

The update is

$$f_{t+1}=f_t-\eta_t\bigl(f_t(x_t)-y_t\bigr)K(x_t,\cdot),$$5

and the averaged iterate is $$f_{t+1}=f_t-\eta_t\bigl(f_t(x_t)-y_t\bigr)K(x_t,\cdot),$$6. With constant step size $$f_{t+1}=f_t-\eta_t\bigl(f_t(x_t)-y_t\bigr)K(x_t,\cdot),$$7, truncation exponent $$f_{t+1}=f_t-\eta_t\bigl(f_t(x_t)-y_t\bigr)K(x_t,\cdot),$$8, and the regularity condition $$f_{t+1}=f_t-\eta_t\bigl(f_t(x_t)-y_t\bigr)K(x_t,\cdot),$$9, the excess risk satisfies

$X_n\sim\mu(x)$0

and the optimal choice $X_n\sim\mu(x)$1 yields the minimax-optimal rate $X_n\sim\mu(x)$2. The same construction is stated to overcome the saturation problem of classical kernel SGD, while its spherical-polynomial implementation reduces storage and computational cost to $X_n\sim\mu(x)$3 and $X_n\sim\mu(x)$4 for sufficiently regular problems (Bai et al., 2024).

A related general-loss framework uses spherical radial basis functions and projects each stochastic gradient onto

$X_n\sim\mu(x)$5

with

$X_n\sim\mu(x)$6

The projected update is

$X_n\sim\mu(x)$7

Under the source condition $X_n\sim\mu(x)$8 with $X_n\sim\mu(x)$9, the last iterate and the suffix average achieve, up to logarithmic factors, minimax-optimal excess-risk rates $X_n\neq\theta_n$0, and the RKHS norm converges at rate $X_n\neq\theta_n$1 up to logarithmic factors. Because $X_n\neq\theta_n$2, the overall time is $X_n\neq\theta_n$3 and the memory is $X_n\neq\theta_n$4 (Bai et al., 5 Oct 2025).

A second branch replaces exact kernels by random-feature or doubly stochastic surrogates. Doubly stochastic functional gradients use

$X_n\neq\theta_n$5

which is unbiased for the risk gradient because $X_n\neq\theta_n$6. The resulting function converges pointwise at rate $X_n\neq\theta_n$7 and achieves excess risk $X_n\neq\theta_n$8, while avoiding storage of support vectors (Dai et al., 2014). In the mini-batch random-feature setting, the number of features $X_n\neq\theta_n$9, the number of SGD steps $$\theta_{n+1}=\theta_n-\alpha_n\,K_h(\theta_n-X_n)\,\hat\nabla J(X_n;\theta_n).$$0, the step size $$\theta_{n+1}=\theta_n-\alpha_n\,K_h(\theta_n-X_n)\,\hat\nabla J(X_n;\theta_n).$$1, and the batch size $$\theta_{n+1}=\theta_n-\alpha_n\,K_h(\theta_n-X_n)\,\hat\nabla J(X_n;\theta_n).$$2 jointly determine implicit regularization, and with standard capacity and source assumptions one obtains the minimax-optimal rate $$\theta_{n+1}=\theta_n-\alpha_n\,K_h(\theta_n-X_n)\,\hat\nabla J(X_n;\theta_n).$$3 (Carratino et al., 2018). Distributed SGM extends the same principle to partitioned data and retains the optimal rate $$\theta_{n+1}=\theta_n-\alpha_n\,K_h(\theta_n-X_n)\,\hat\nabla J(X_n;\theta_n).$$4 provided the partition level is not too large (Lin et al., 2018).

These variants all preserve the defining role of the stochastic gradient kernel while modifying its representation. The exact kernel section, truncated kernel, or random-feature surrogate is the object through which the stochastic gradient is regularized and made computationally tractable.

5. Stability, restricted gradients, and directional bias

In fixed-dictionary kernel adaptive filtering, the stochastic gradient kernel appears through a restricted gradient on a dictionary subspace

$$\theta_{n+1}=\theta_n-\alpha_n\,K_h(\theta_n-X_n)\,\hat\nabla J(X_n;\theta_n).$$5

If $$\theta_{n+1}=\theta_n-\alpha_n\,K_h(\theta_n-X_n)\,\hat\nabla J(X_n;\theta_n).$$6 is the Gram matrix of the dictionary and $$\theta_{n+1}=\theta_n-\alpha_n\,K_h(\theta_n-X_n)\,\hat\nabla J(X_n;\theta_n).$$7 is the MSE cost in coordinates, the steepest-descent direction under the induced $$\theta_{n+1}=\theta_n-\alpha_n\,K_h(\theta_n-X_n)\,\hat\nabla J(X_n;\theta_n).$$8-inner product is

$$\theta_{n+1}=\theta_n-\alpha_n\,K_h(\theta_n-X_n)\,\hat\nabla J(X_n;\theta_n).$$9

This yields the Natural KLMS update

$$f(x;\theta)=\frac1{\sqrt M}\sum_{r=1}^M a_r\,\sigma(b_r^\top x+\gamma c_r),\qquad \theta=(a,B,c),$$0

which becomes an ordinary LMS recursion in whitened coordinates. The analysis gives mean stability if

$$f(x;\theta)=\frac1{\sqrt M}\sum_{r=1}^M a_r\,\sigma(b_r^\top x+\gamma c_r),\qquad \theta=(a,B,c),$$1

and mean-square stability if the covariance recursion has spectral radius $$f(x;\theta)=\frac1{\sqrt M}\sum_{r=1}^M a_r\,\sigma(b_r^\top x+\gamma c_r),\qquad \theta=(a,B,c),$$2 (Takizawa et al., 2014). The kernel here is not only a representer; through $$f(x;\theta)=\frac1{\sqrt M}\sum_{r=1}^M a_r\,\sigma(b_r^\top x+\gamma c_r),\qquad \theta=(a,B,c),$$3 it induces the local metric of the stochastic descent.

A different phenomenon is directional bias. In kernel regression with Gram matrix $$f(x;\theta)=\frac1{\sqrt M}\sum_{r=1}^M a_r\,\sigma(b_r^\top x+\gamma c_r),\qquad \theta=(a,B,c),$$4, full-batch GD and one-sample SGD both interpolate, but their residual vectors align with different eigendirections. Under a diagonally-dominant Gram matrix, SGD with a moderate-and-annealing step-size schedule converges along the eigenvector corresponding to the largest eigenvalue $$f(x;\theta)=\frac1{\sqrt M}\sum_{r=1}^M a_r\,\sigma(b_r^\top x+\gamma c_r),\qquad \theta=(a,B,c),$$5, whereas GD with a moderate or small step size converges along the eigenvector corresponding to the smallest eigenvalue $$f(x;\theta)=\frac1{\sqrt M}\sum_{r=1}^M a_r\,\sigma(b_r^\top x+\gamma c_r),\qquad \theta=(a,B,c),$$6. With misalignment vector $$f(x;\theta)=\frac1{\sqrt M}\sum_{r=1}^M a_r\,\sigma(b_r^\top x+\gamma c_r),\qquad \theta=(a,B,c),$$7, the RKHS generalization error is

$$f(x;\theta)=\frac1{\sqrt M}\sum_{r=1}^M a_r\,\sigma(b_r^\top x+\gamma c_r),\qquad \theta=(a,B,c),$$8

For fixed training loss, this error is minimized when the residual aligns with the top-eigenvector, which the analysis uses to explain why the SGD estimator can have smaller generalization error than the GD estimator (Luo et al., 2022).

The same bias structure extends to complex RKHSs. Under unbiasedness, bounded variance, lower-bounded objective, smoothness, and convexity assumptions, the averaged iterate of Complex SGD satisfies

$$f(x;\theta)=\frac1{\sqrt M}\sum_{r=1}^M a_r\,\sigma(b_r^\top x+\gamma c_r),\qquad \theta=(a,B,c),$$9

and with $K$00 this gives $K$01. In the strongly convex case,

$K$02

so the iterates exhibit exponential-type convergence to an $K$03 neighborhood. The same work states that directional bias results known in the real setting extend to kernel regression in complex RKHSs (Alpay et al., 24 Apr 2026).

6. Applications, extensions, and recurring misconceptions

Kernelized stochastic-gradient constructions appear well beyond standard regression. In passive stochastic approximation, multi-kernel mixtures

$K$04

are used with self-normalized importance weights to aggregate a batch of noisy gradients sampled at random locations. Under weak-limit conditions, the continuous-time interpolation converges to the ideal ODE

$K$05

so the limit no longer depends on the sampling density or on the individual kernels; the diffusion approximation yields $K$06 for constant step size and $K$07 for decaying step size (Krishnamurthy et al., 2020).

In quantum kernel alignment, Pegasos is used to learn both the SVM decision boundary and the kernel parameters simultaneously. With trainable quantum feature map $K$08, the induced kernel is

$K$09

and the stochastic procedure updates the primal SVM weights and the circuit parameters in one loop. The reported comparison states that this reduces quantum-circuit calls from $K$10 to $K$11, up to shot-noise factors, and naturally supports non-stationary data through support-vector forgetting (Gentinetta et al., 2023).

In non-adversarial generative modeling, kernel-based training minimizes

$K$12

a pure minimization problem rather than a min-max one. The corresponding stochastic gradient update is standard SGD on $K$13, and the stochastic-approximation analysis states that batch averaging, gradient smoothing, and delayed-parameter variants do not improve the leading-order convergence behavior or the steady-state covariance (Basioti et al., 2018).

A recurring misconception is that kernel SGD is intrinsically saturation-limited or that iterate averaging is always the statistically preferred strategy. The spherical T-kernel analysis states that constant-step-size truncated kernel SGD overcomes the inherent saturation problem of kernel SGD (Bai et al., 2024). The learning-curve analysis for kernel regression on the sphere similarly states that one-pass SGD with an exponentially decaying step-size schedule achieves minimax-optimal rates up to constants across scales and avoids saturation except in a final stage of highly misspecified learning, whereas averaged SGD behaves like a qualification-$K$14 filter and can saturate when the smoothness parameter is large (Zhang et al., 28 May 2025). These results do not eliminate the importance of averaging; rather, they show that the statistical role of a stochastic gradient kernel depends on the spectral filter implemented by the update schedule, truncation rule, and representation of the kernel itself.