NTK-ECRN: Kernel-Controlled Residual Networks

• NTK-ECRN is a deep network architecture that uses residual connections, Fourier feature embeddings, and explicit kernel eigenvalue control to manage NTK spectral properties.
• It employs layerwise scaling and stochastic depth to regulate kernel spikiness and reduce spectral bias, ensuring stable optimization even in very deep networks.
• The model bridges infinite-width NTK theory with practical applications, demonstrating improved performance in high-frequency regression and classification tasks.

NTK-Eigenvalue-Controlled Residual Networks (NTK-ECRN) are a class of deep architectures that integrate residual connections, explicit kernel eigenvalue control, and structured embeddings to enable rigorous mathematical analysis of the neural tangent kernel (NTK) during training. These models facilitate precise manipulation of kernel spectral properties, providing mechanisms to stabilize optimization, enhance generalization, and reduce spectral bias. They bridge infinite-width NTK theory with practical deep-learning architectures by incorporating Fourier feature embeddings, layerwise residual scaling, and stochastic depth. Theoretical work establishes bounds on NTK dynamics, characterizes eigenvalue evolution, and links spectral control to function-space expressivity and learning performance (Belfer et al., 2021, Mysore et al., 9 Dec 2025).

1. Model Architecture and Formal Specification

Let $x \in \mathbb{R}^d$ denote an input. The NTK-ECRN architecture is composed of the following components (Mysore et al., 9 Dec 2025):

1. Fourier Feature Embedding: Define a (learned or fixed) projection matrix $B \in \mathbb{R}^{d_f \times d}$ and set

$$\phi(x) = [\sin(2\pi Bx), \cos(2\pi Bx)] \in \mathbb{R}^{2d_f}.$$

This embedding enriches the input with high-frequency components to expand the NTK's function space.

2. Residual Blocks with Layerwise Scaling and Stochastic Depth:

$$h^{(0)} = \phi(x), \qquad h^{(l+1)} = h^{(l)} + m_l \, \alpha_l \, \sigma\big(W^l h^{(l)} + b^l\big),$$

for $l=0,\ldots,L-1$, where $W^l \in \mathbb{R}^{n \times n}$, $b^l \in \mathbb{R}^n$, $\sigma$ is a smooth activation (e.g., $\tanh$, GELU), $\alpha_l > 0$ is a layerwise scaling factor, and $B \in \mathbb{R}^{d_f \times d}$0 implements stochastic depth.

3. Linear Readout:

$B \in \mathbb{R}^{d_f \times d}$1

Key architectural features include control over the contribution of each layer to the kernel via $B \in \mathbb{R}^{d_f \times d}$2, explicit stochastic regularization through $B \in \mathbb{R}^{d_f \times d}$3, and spectral enrichment via $B \in \mathbb{R}^{d_f \times d}$4.

2. Neural Tangent Kernel Structure and Spectral Properties

For parameters $B \in \mathbb{R}^{d_f \times d}$5, the NTK is given by

$B \in \mathbb{R}^{d_f \times d}$6

The residual update yields a recursive kernel formula:

$B \in \mathbb{R}^{d_f \times d}$7

where $B \in \mathbb{R}^{d_f \times d}$8 is the Jacobian of the $B \in \mathbb{R}^{d_f \times d}$9th block's activation.

For massively overparameterized, ReLU-activated ResNets on inputs uniformly distributed on $$\phi(x) = [\sin(2\pi Bx), \cos(2\pi Bx)] \in \mathbb{R}^{2d_f}.$$0, the infinite-width NTK (ResNTK) kernel $$\phi(x) = [\sin(2\pi Bx), \cos(2\pi Bx)] \in \mathbb{R}^{2d_f}.$$1 has the exact form (Belfer et al., 2021):

$$\phi(x) = [\sin(2\pi Bx), \cos(2\pi Bx)] \in \mathbb{R}^{2d_f}.$$2

with layerwise recurrence defined via ReLU kernel functions $$\phi(x) = [\sin(2\pi Bx), \cos(2\pi Bx)] \in \mathbb{R}^{2d_f}.$$3 and parameters $$\phi(x) = [\sin(2\pi Bx), \cos(2\pi Bx)] \in \mathbb{R}^{2d_f}.$$4 (the scaling of residual blocks).

On $$\phi(x) = [\sin(2\pi Bx), \cos(2\pi Bx)] \in \mathbb{R}^{2d_f}.$$5, $$\phi(x) = [\sin(2\pi Bx), \cos(2\pi Bx)] \in \mathbb{R}^{2d_f}.$$6 is zonal and homogeneous of degree 1, so its eigenfunctions are the spherical harmonics $$\phi(x) = [\sin(2\pi Bx), \cos(2\pi Bx)] \in \mathbb{R}^{2d_f}.$$7. Eigenvalues decay polynomially with frequency $$\phi(x) = [\sin(2\pi Bx), \cos(2\pi Bx)] \in \mathbb{R}^{2d_f}.$$8 as $$\phi(x) = [\sin(2\pi Bx), \cos(2\pi Bx)] \in \mathbb{R}^{2d_f}.$$9, mirroring the Laplace kernel and classical finite-layer FC-NTK (Belfer et al., 2021).

3. Explicit Control of Kernel Spectra

The parameter $$h^{(0)} = \phi(x), \qquad h^{(l+1)} = h^{(l)} + m_l \, \alpha_l \, \sigma\big(W^l h^{(l)} + b^l\big),$$0 (and its layerwise variant $$h^{(0)} = \phi(x), \qquad h^{(l+1)} = h^{(l)} + m_l \, \alpha_l \, \sigma\big(W^l h^{(l)} + b^l\big),$$1 in ECRN) is a tuning knob that determines the contribution of residual versus skip connections (Belfer et al., 2021, Mysore et al., 9 Dec 2025). Analytical expansions near $$h^{(0)} = \phi(x), \qquad h^{(l+1)} = h^{(l)} + m_l \, \alpha_l \, \sigma\big(W^l h^{(l)} + b^l\big),$$2 (where $$h^{(0)} = \phi(x), \qquad h^{(l+1)} = h^{(l)} + m_l \, \alpha_l \, \sigma\big(W^l h^{(l)} + b^l\big),$$3) yield:

• For fixed $$h^{(0)} = \phi(x), \qquad h^{(l+1)} = h^{(l)} + m_l \, \alpha_l \, \sigma\big(W^l h^{(l)} + b^l\big),$$4, the coefficient $$h^{(0)} = \phi(x), \qquad h^{(l+1)} = h^{(l)} + m_l \, \alpha_l \, \sigma\big(W^l h^{(l)} + b^l\big),$$5 in the expansion $$h^{(0)} = \phi(x), \qquad h^{(l+1)} = h^{(l)} + m_l \, \alpha_l \, \sigma\big(W^l h^{(l)} + b^l\big),$$6 grows as $$h^{(0)} = \phi(x), \qquad h^{(l+1)} = h^{(l)} + m_l \, \alpha_l \, \sigma\big(W^l h^{(l)} + b^l\big),$$7 for large $$h^{(0)} = \phi(x), \qquad h^{(l+1)} = h^{(l)} + m_l \, \alpha_l \, \sigma\big(W^l h^{(l)} + b^l\big),$$8, resulting in a kernel that becomes increasingly spiky ("spikiness").
• For $$h^{(0)} = \phi(x), \qquad h^{(l+1)} = h^{(l)} + m_l \, \alpha_l \, \sigma\big(W^l h^{(l)} + b^l\big),$$9, $l=0,\ldots,L-1$0 for $l=0,\ldots,L-1$1 and $l=0,\ldots,L-1$2 for $l=0,\ldots,L-1$3 as $l=0,\ldots,L-1$4, yielding a stable Laplace-like kernel shape and preventing spectral localization.

Parity bias emerges when $l=0,\ldots,L-1$5 with $l=0,\ldots,L-1$6, causing odd-$l=0,\ldots,L-1$7 eigenvalues to vanish and emphasizing even harmonics.

Residual scaling and stochastic depth provide mechanisms to:

• Maintain or suppress kernel spikiness,
• Achieve stability in eigenvalue growth,
• Induce parity bias or enforce Laplace-like universality,
• Restrict the NTK's condition number,
• Precisely target function spaces of desired smoothness (Belfer et al., 2021, Mysore et al., 9 Dec 2025).

4. Fourier Feature Embedding and Role of Stochastic Depth

Fourier features expand the reproducing kernel Hilbert space (RKHS) of the NTK to encompass high-frequency functions, increasing the smallest eigenvalues $l=0,\ldots,L-1$8 associated with fine-scale components. This reduces spectral bias, accelerates fitting of non-smooth targets, and diminishes slow learning of high-frequency modes.

Stochastic depth introduces blockwise Bernoulli random variables $l=0,\ldots,L-1$9, resulting in the NTK being an expectation over mixtures of shallower kernels. In expectation, each residual block's contribution is scaled by $W^l \in \mathbb{R}^{n \times n}$0, allowing regularization of kernel growth and tractable computation of $W^l \in \mathbb{R}^{n \times n}$1 and $W^l \in \mathbb{R}^{n \times n}$2 (Mysore et al., 9 Dec 2025).

Both mechanisms contribute to empirical stability, improved generalization, and analytic tractability of kernel spectra.

5. Theoretical Analysis: NTK Dynamics, Eigenvalue Evolution, and Stability

Denote $W^l \in \mathbb{R}^{n \times n}$3 as the empirical NTK after $W^l \in \mathbb{R}^{n \times n}$4 training steps. The following bounds characterize its evolution under gradient descent (Mysore et al., 9 Dec 2025):

• Incremental NTK Bound: For $W^l \in \mathbb{R}^{n \times n}$5 $W^l \in \mathbb{R}^{n \times n}$6-Lipschitz and $W^l \in \mathbb{R}^{n \times n}$7,

$W^l \in \mathbb{R}^{n \times n}$8

thereby linking NTK changes directly to residual scaling and learning rates.

• Eigenvalue Increment: Each layer provides

$W^l \in \mathbb{R}^{n \times n}$9

Under stochastic depth $b^l \in \mathbb{R}^n$0, expectation yields

$b^l \in \mathbb{R}^n$1

Spectral control ensures a bounded condition number $b^l \in \mathbb{R}^n$2, preventing edge-of-stability instabilities and promoting uniform convergence rates across data modes. Generalization is bounded by the sum of squared errors in each eigenmode, weighted by $b^l \in \mathbb{R}^n$3.

6. Empirical Performance and Recommendations

Empirical results corroborate the theoretical framework (Mysore et al., 9 Dec 2025, Belfer et al., 2021):

• On high-frequency regression and multi-class classification, NTK-ECRN matches or outperforms both conventional NTK and standard deep architectures in $b^l \in \mathbb{R}^n$4, cross-entropy, and test accuracy.
• Kernel spikiness (constant $b^l \in \mathbb{R}^n$5) yields degraded accuracy at depth, whereas scaling $b^l \in \mathbb{R}^n$6 or $b^l \in \mathbb{R}^n$7 maintains Laplace-like spectral decay and robust generalization, even for $b^l \in \mathbb{R}^n$8.
• Fourier embeddings rapidly fit high-frequency targets, evidenced by accelerated eigenmode convergence.
• On standard benchmarks (UCI regression, CIFAR-10), NTK-ECRN achieves $b^l \in \mathbb{R}^n$9 improvements in accuracy or $\sigma$0 relative to both standard finite-width NTK and deep baselines.

A concrete guideline emerges: select depth $\sigma$1 and scale $\sigma$2 so that the coefficient $\sigma$3 in the kernel's small-angle expansion matches the Laplace or desired profile, choosing $\sigma$4 for spectral stability and parity bias as needed (Belfer et al., 2021).

7. Connections, Significance, and Design Considerations

NTK-ECRN unifies infinite-width kernel theory with tractable architectural controls for practical deep networks. The approach is rooted in explicit manipulation of kernel eigenspectra, drawing an analytic connection between network hyperparameters and function-space coverage. Its utility includes:

• Eliminating deleterious kernel localization as network depth increases,
• Enabling data-dependent kernel tailoring (spectral adaptation),
• Providing a rigorous platform for investigating kernel dynamics during training.

These models inherit the function-space expressivity of Laplace and spherical kernels (smoothness ~ $\sigma$5) while affording precise parity control and spectral steepness modulation (Belfer et al., 2021, Mysore et al., 9 Dec 2025). A plausible implication is their use as a diagnostic and design tool for robust deep learning, especially in scenarios where spectral bias or kernel degeneration at depth pose learning challenges.