Stochastic Domingos Theorems in Deep Learning

• Stochastic Domingos theorems define continuous-time kernel representations via diffusion approximations that extend deterministic gradient-flow identities.
• They reveal optimizer-specific kernel structures where SGD, SGDM, RMSprop, and Adam induce distinct temporal and geometric weightings in the learning trajectory.
• The framework links tangent-feature memory to generalization by highlighting how training dynamics capture interpolation and null-space properties in neural networks.

Stochastic Domingos theorems are interpolation formulas for models trained by stochastic optimization that extend Pedro Domingos’ deterministic claim that gradient-trained differentiable models admit a kernel-machine description. In the stochastic formulation, mini-batch training is replaced by a continuous-time diffusion approximation, the tangent kernel becomes a stochastic gradient kernel evaluated along a random training trajectory, and the resulting representation is optimizer-specific: SGD, SGDM, RMSprop, and Adam induce distinct temporal and geometric weightings in the kernel expansion (Guo et al., 14 Mar 2026). The topic sits at the intersection of NTK-style tangent-feature analysis, weak approximation of stochastic optimization, and generalization theory, and it is best understood against the earlier deterministic discussion that emphasized both the exact path-kernel identity and its interpretive limits (Courtois et al., 2022).

1. Deterministic precursor: Domingos’ interpolation formula

The deterministic precursor considers a differentiable learning machine

$$\mathcal N:\mathbb R^p\times\mathbb R^d\to\mathbb R, \qquad (x,w)\mapsto \mathcal N(x;w),$$

trained on

$\mathcal D=(x_n,y_n)_{n\in[|1,N|]}$

by continuous-time gradient flow

$$\frac{dw(t)}{dt}=-\nabla_w\mathcal L(w(t)), \qquad \mathcal L(w)=\frac1N\sum_{n=1}^N L(\hat y_n(w),y_n), \quad \hat y_n(w)=\mathcal N(x_n;w).$$

The tangent kernel at parameter $w$ is

$$K_{\mathcal N,w}(x,\tilde x) = \langle \nabla_w \mathcal N(x;w),\nabla_w \mathcal N(\tilde x;w)\rangle_{\mathbb R^d},$$

and the path kernel along the learning trajectory $w(s)$ is

$$K_{\mathcal N,\gamma}(x,\tilde x;t) = \int_0^t \langle \nabla_w \mathcal N(x;w(s)),\nabla_w \mathcal N(\tilde x;w(s))\rangle_{\mathbb R^d}\,ds.$$

The exact deterministic identity is obtained by differentiating the predictor along the gradient-flow path: $$\frac{\partial \mathcal N(x;w(t))}{\partial t} = -\frac1N\sum_{n=1}^N L'(\hat y_n(w(t)),y_n)\, K_{\mathcal N,w(t)}(x,x_n).$$ Integrating from $0$ to $t$ yields

$\mathcal D=(x_n,y_n)_{n\in[|1,N|]}$0

For squared loss, this becomes

$\mathcal D=(x_n,y_n)_{n\in[|1,N|]}$1

This deterministic result is mathematically exact, but its interpretation is qualified in two ways. First, what is exactly a kernel machine is the time derivative $\mathcal D=(x_n,y_n)_{n\in[|1,N|]}$2, not necessarily the final predictor in the sense of a fixed-coefficient RKHS expansion. Second, the coefficients in the integrated representation depend on the test point $\mathcal D=(x_n,y_n)_{n\in[|1,N|]}$3, and the “bias” term is $\mathcal D=(x_n,y_n)_{n\in[|1,N|]}$4, which also depends on $\mathcal D=(x_n,y_n)_{n\in[|1,N|]}$5. The later stochastic development inherits this caution: the relevant kernel-machine structure is path-dependent and optimizer-dependent rather than a static classical kernel expansion (Courtois et al., 2022).

2. Diffusion approximation and the stochastic gradient kernel

The stochastic extension begins from the observation that practical deep learning is trained by mini-batch methods at finite learning rate rather than by full-batch gradient flow. For SGD, the discrete iteration is

$\mathcal D=(x_n,y_n)_{n\in[|1,N|]}$6

where $\mathcal D=(x_n,y_n)_{n\in[|1,N|]}$7 is a random mini-batch and

$\mathcal D=(x_n,y_n)_{n\in[|1,N|]}$8

The bridge to a stochastic Domingos theorem is a weak SDE approximation: $\mathcal D=(x_n,y_n)_{n\in[|1,N|]}$9 with

$$\frac{dw(t)}{dt}=-\nabla_w\mathcal L(w(t)), \qquad \mathcal L(w)=\frac1N\sum_{n=1}^N L(\hat y_n(w),y_n), \quad \hat y_n(w)=\mathcal N(x_n;w).$$0

The approximation is weak in the sense that for smooth test functions $$\frac{dw(t)}{dt}=-\nabla_w\mathcal L(w(t)), \qquad \mathcal L(w)=\frac1N\sum_{n=1}^N L(\hat y_n(w),y_n), \quad \hat y_n(w)=\mathcal N(x_n;w).$$1,

$$\frac{dw(t)}{dt}=-\nabla_w\mathcal L(w(t)), \qquad \mathcal L(w)=\frac1N\sum_{n=1}^N L(\hat y_n(w),y_n), \quad \hat y_n(w)=\mathcal N(x_n;w).$$2

The central stochastic object is the stochastic gradient kernel

$$\frac{dw(t)}{dt}=-\nabla_w\mathcal L(w(t)), \qquad \mathcal L(w)=\frac1N\sum_{n=1}^N L(\hat y_n(w),y_n), \quad \hat y_n(w)=\mathcal N(x_n;w).$$3

with same-time version

$$\frac{dw(t)}{dt}=-\nabla_w\mathcal L(w(t)), \qquad \mathcal L(w)=\frac1N\sum_{n=1}^N L(\hat y_n(w),y_n), \quad \hat y_n(w)=\mathcal N(x_n;w).$$4

For adaptive optimizers, the paper also introduces the weighted stochastic gradient kernel

$$\frac{dw(t)}{dt}=-\nabla_w\mathcal L(w(t)), \qquad \mathcal L(w)=\frac1N\sum_{n=1}^N L(\hat y_n(w),y_n), \quad \hat y_n(w)=\mathcal N(x_n;w).$$5

This shift from $$\frac{dw(t)}{dt}=-\nabla_w\mathcal L(w(t)), \qquad \mathcal L(w)=\frac1N\sum_{n=1}^N L(\hat y_n(w),y_n), \quad \hat y_n(w)=\mathcal N(x_n;w).$$6 to $$\frac{dw(t)}{dt}=-\nabla_w\mathcal L(w(t)), \qquad \mathcal L(w)=\frac1N\sum_{n=1}^N L(\hat y_n(w),y_n), \quad \hat y_n(w)=\mathcal N(x_n;w).$$7 changes the theorem’s status. The deterministic formula is pathwise and exact along a gradient-flow trajectory. The stochastic theorem is an expectation-level statement about $$\frac{dw(t)}{dt}=-\nabla_w\mathcal L(w(t)), \qquad \mathcal L(w)=\frac1N\sum_{n=1}^N L(\hat y_n(w),y_n), \quad \hat y_n(w)=\mathcal N(x_n;w).$$8, with the martingale term eliminated only after taking expectation and with an explicit small-step remainder. In that precise sense, the stochastic Domingos theorems are diffusion-averaged extensions of the deterministic path-kernel identity rather than pathwise identities for the raw mini-batch iterates (Guo et al., 14 Mar 2026).

3. Optimizer-specific stochastic Domingos theorems

For SGD, the stochastic Domingos theorem states

$$\frac{dw(t)}{dt}=-\nabla_w\mathcal L(w(t)), \qquad \mathcal L(w)=\frac1N\sum_{n=1}^N L(\hat y_n(w),y_n), \quad \hat y_n(w)=\mathcal N(x_n;w).$$9

Thus the expected output is an initial predictor plus an accumulated kernel interaction term, with training samples contributing through loss derivatives and tangent-feature alignment along the stochastic trajectory.

The remaining optimizers preserve the same basic structure but alter the weighting. SGDM introduces a two-time kernel and exponential temporal memory; RMSprop introduces a preconditioned feature metric; Adam combines both effects and yields a weaker asymptotic remainder. The optimizer-specific forms are summarized below.

Optimizer	Kernel structure in the expected-output formula	Remainder
SGD	$w$0	$w$1
SGDM	$w$2, $w$3	$w$4
RMSprop	$w$5	$w$6
Adam	memory term with $w$7 plus instantaneous term with $w$8	$w$9

For SGDM, the theorem is

$$K_{\mathcal N,w}(x,\tilde x) = \langle \nabla_w \mathcal N(x;w),\nabla_w \mathcal N(\tilde x;w)\rangle_{\mathbb R^d},$$0

so momentum acts as an exponentially decaying temporal memory. For RMSprop, the expected predictor is governed by $$K_{\mathcal N,w}(x,\tilde x) = \langle \nabla_w \mathcal N(x;w),\nabla_w \mathcal N(\tilde x;w)\rangle_{\mathbb R^d},$$1, where $$K_{\mathcal N,w}(x,\tilde x) = \langle \nabla_w \mathcal N(x;w),\nabla_w \mathcal N(\tilde x;w)\rangle_{\mathbb R^d},$$2 is a diagonal exponential average of $$K_{\mathcal N,w}(x,\tilde x) = \langle \nabla_w \mathcal N(x;w),\nabla_w \mathcal N(\tilde x;w)\rangle_{\mathbb R^d},$$3 plus $$K_{\mathcal N,w}(x,\tilde x) = \langle \nabla_w \mathcal N(x;w),\nabla_w \mathcal N(\tilde x;w)\rangle_{\mathbb R^d},$$4. For Adam, the theorem adds both a momentum-memory term and an instantaneous term, with weights involving

$$K_{\mathcal N,w}(x,\tilde x) = \langle \nabla_w \mathcal N(x;w),\nabla_w \mathcal N(\tilde x;w)\rangle_{\mathbb R^d},$$5

and bias-correction-like factors

$$K_{\mathcal N,w}(x,\tilde x) = \langle \nabla_w \mathcal N(x;w),\nabla_w \mathcal N(\tilde x;w)\rangle_{\mathbb R^d},$$6

These formulas are the sense in which the stochastic Domingos theorems provide a kernel-machine representation with optimizer-specific weighting (Guo et al., 14 Mar 2026).

4. Tangent-feature memory and the meaning of the kernel-machine representation

The stochastic Domingos theorems support a feature-space memory view of learning. Training samples contribute to the prediction at test point $$K_{\mathcal N,w}(x,\tilde x) = \langle \nabla_w \mathcal N(x;w),\nabla_w \mathcal N(\tilde x;w)\rangle_{\mathbb R^d},$$7 through two quantities. The first is a loss-dependent weight,

$$K_{\mathcal N,w}(x,\tilde x) = \langle \nabla_w \mathcal N(x;w),\nabla_w \mathcal N(\tilde x;w)\rangle_{\mathbb R^d},$$8

which reduces to a residual for mean-squared error. The second is tangent-feature alignment, measured by

$$K_{\mathcal N,w}(x,\tilde x) = \langle \nabla_w \mathcal N(x;w),\nabla_w \mathcal N(\tilde x;w)\rangle_{\mathbb R^d},$$9

or by its preconditioned variant. In this picture, training stores data-dependent information in an evolving tangent geometry, and test-time prediction arises from kernel-weighted retrieval and aggregation of those stored features (Guo et al., 14 Mar 2026).

This interpretation is continuous with the deterministic Domingos discussion but not identical to a standard fixed-kernel RKHS model. The kernel is trajectory-dependent, usually parameter-dependent, and in the stochastic setting random before expectation. The coefficients are not static dual weights; they are loss derivatives accumulated along the training path. The deterministic discussion already emphasized that “approximately a kernel machine” should be read carefully because the coefficients depend on the test point and the bias term depends on the test point as well. The stochastic theorems preserve that qualification while extending the path-kernel viewpoint to mini-batch training (Courtois et al., 2022).

A plausible implication is that lazy-training and NTK regimes correspond to a special case in which the stochastic kernel changes little over training. The later paper states that when $w(s)$0, the operator viewpoint reduces toward the classical NTK picture. Outside that regime, the stochastic Domingos theorem is explicitly about an evolving tangent-feature geometry rather than a frozen kernel (Guo et al., 14 Mar 2026).

5. Generalization and the null space of the stochastic kernel operator

The paper links generalization error to the null space of the integral operator induced by the stochastic gradient kernel. The operator is

$w(s)$1

For mean-squared error, define the terminal residual

$w(s)$2

and the generalization error

$w(s)$3

The theorem states that, as $w(s)$4,

$w(s)$5

Accordingly, the residual generalization error is governed by the component of $w(s)$6 lying in $w(s)$7 (Guo et al., 14 Mar 2026).

The operator-theoretic interpretation is spectral. If

$w(s)$8

then modes with $w(s)$9 are visible to the learned tangent geometry, whereas modes in the null space are not. The paper derives the null-space result from the evolution of the residual energy, obtaining, up to $$K_{\mathcal N,\gamma}(x,\tilde x;t) = \int_0^t \langle \nabla_w \mathcal N(x;w(s)),\nabla_w \mathcal N(\tilde x;w(s))\rangle_{\mathbb R^d}\,ds.$$0,

$$K_{\mathcal N,\gamma}(x,\tilde x;t) = \int_0^t \langle \nabla_w \mathcal N(x;w(s)),\nabla_w \mathcal N(\tilde x;w(s))\rangle_{\mathbb R^d}\,ds.$$1

Under the stated stationarity assumptions, this drives $$K_{\mathcal N,\gamma}(x,\tilde x;t) = \int_0^t \langle \nabla_w \mathcal N(x;w(s)),\nabla_w \mathcal N(\tilde x;w(s))\rangle_{\mathbb R^d}\,ds.$$2 to zero in the small-step limit.

This suggests a sharp distinction between interpolation and extrapolation. In-distribution behavior is favorable when test targets align with the range of the learned kernel operator. Out-of-distribution failure arises when test behavior excites directions that remain in the null space of the learned tangent-feature memory. The paper treats this as the structural mechanism through which stochastic training retains a kernel description while still exhibiting limits on generalization (Guo et al., 14 Mar 2026).

6. Extensions, empirical illustrations, and limitations

The same path-kernel framework is used to interpret diffusion models and GANs. For diffusion models, the paper states that diffusion induces stage-wise, noise-localized corrections: the time or noise embedding makes the kernel large mainly between nearby noise levels, so denoising is represented as local correction along the diffusion schedule. For GANs, the correction is distribution-guided and shaped by discriminator geometry: the discriminator induces a density-ratio-like geometry, and the generator update is propagated through a kernel defined by generator-Jacobian alignment (Guo et al., 14 Mar 2026).

The empirical illustrations are geometric rather than asymptotic. On synthetic classification, MNIST, and CIFAR-10, normalized gradient kernels evolve toward label-homogeneous or block-diagonal structure. Tangent-feature maps $$K_{\mathcal N,\gamma}(x,\tilde x;t) = \int_0^t \langle \nabla_w \mathcal N(x;w(s)),\nabla_w \mathcal N(\tilde x;w(s))\rangle_{\mathbb R^d}\,ds.$$3 become linearly separable enough for linear SVMs to recover the learned decision rule. In out-of-distribution settings, low cross-domain tangent-kernel similarity correlates with weak transfer, and rank gaps in tangent features correlate with persistent test loss. These observations are presented as support for the claim that predictions are governed by learned tangent-feature geometry rather than by Euclidean proximity alone (Guo et al., 14 Mar 2026).

The framework also has explicit limitations. The theorems are expectation-level statements, not pathwise descriptions of every stochastic training run. They rely on weak SDE approximations and on smoothness assumptions; the later paper notes that the fully rigorous statements exclude non-smooth architectures or losses such as general ReLU or $$K_{\mathcal N,\gamma}(x,\tilde x;t) = \int_0^t \langle \nabla_w \mathcal N(x;w(s)),\nabla_w \mathcal N(\tilde x;w(s))\rangle_{\mathbb R^d}\,ds.$$4 loss. Adam carries only an $$K_{\mathcal N,\gamma}(x,\tilde x;t) = \int_0^t \langle \nabla_w \mathcal N(x;w(s)),\nabla_w \mathcal N(\tilde x;w(s))\rangle_{\mathbb R^d}\,ds.$$5 remainder, weaker than the $$K_{\mathcal N,\gamma}(x,\tilde x;t) = \int_0^t \langle \nabla_w \mathcal N(x;w(s)),\nabla_w \mathcal N(\tilde x;w(s))\rangle_{\mathbb R^d}\,ds.$$6 error obtained for SGD, SGDM, and RMSprop. More broadly, the earlier deterministic discussion emphasized that Domingos’ theorem, while mathematically correct in its setting, does not by itself explain the full behavior of modern neural networks, especially in high-dimensional or nontrivially optimized settings. The stochastic theorems fill the specific gap left open there—SGD- and Adam-type optimization—but they remain explanatory structural results rather than complete predictive theories of deep-network generalization (Courtois et al., 2022).

A common misunderstanding is to equate these results with the claim that trained neural networks are literally fixed classical kernel machines. The literature instead supports a narrower statement: along deterministic or stochastic gradient-based training, prediction updates are governed by tangent-kernel similarities to training examples, and the final predictor is an accumulated path-kernel interpolation whose geometry depends on the trajectory, the optimizer, and the learned feature memory (Guo et al., 14 Mar 2026).