Collective Kernel EFT for Pre-activation ResNets

Abstract: In finite-width deep neural networks, the empirical kernel $G$ evolves stochastically across layers. We develop a collective kernel effective field theory (EFT) for pre-activation ResNets based on a $G$-only closure hierarchy and diagnose its finite validity window. Exploiting the exact conditional Gaussianity of residual increments, we derive an exact stochastic recursion for $G$. Applying Gaussian approximations systematically yields a continuous-depth ODE system for the mean kernel $K_0$, the kernel covariance $V_4$, and the $1/n$ mean correction $K_{1,\mathrm{EFT}}$, which emerges diagrammatically as a one-loop tadpole correction. Numerically, $K_0$ remains accurate at all depths. However, the $V_4$ equation residual accumulates to an $O(1)$ error at finite time, primarily driven by approximation errors in the $G$-only transport term. Furthermore, $K_{1,\mathrm{EFT}}$ fails due to the breakdown of the source closure, which exhibits a systematic mismatch even at initialization. These findings highlight the limitations of $G$-only state-space reduction and suggest extending the state space to incorporate the sigma-kernel.

• The paper presents a novel collective kernel EFT that leverages a G-only closure hierarchy to analyze finite-width dynamics in deep pre-activation ResNets.
• It derives exact block-transition laws and a collective SDE through diagrammatic expansion to capture drift and fluctuation dynamics across layers.
• Numerical validations reveal a finite validity window and identify precise mechanisms for closure breakdown in both covariance and mean corrections.

Collective Kernel Effective Field Theory for Pre-activation ResNets

Introduction and Problem Formulation

This paper presents a systematic finite-width theory for deep pre-activation ResNets at initialization, using a collective kernel effective field theory (EFT) that leverages a $G$-only closure hierarchy. The framework advances beyond mean-field and NTK formulations by capturing empirical kernel evolution via an exact stochastic recursion. The authors systematically derive and critically examine the finite validity window of this Gaussian closure hierarchy, providing non-perturbative insights into the finite-width dynamics of deep ResNets.

The analysis is anchored on interpreting the empirical kernel $G^\ell$ as the sufficient statistic for layer-wise propagation, employing the conditional Gaussianity of residual increments as the main technical tool. The work makes explicit connections to field-theoretic approaches, establishes a diagrammatic expansion around the continuous-depth limit, and localizes sources of closure breakdown at each hierarchical level.

Exact Block Law and Conditional Dynamics

Building upon the exact conditional Gaussianity of block residual increments, the analysis derives a ghost-free Martin–Siggia–Rose–Janssen–De Dominicis (MSRJD) action for a single ResNet block. The core insight is that, in pre-activation ResNets, the increment $\eta^\ell$ (not the preactivation itself) forms an exact conditional Gaussian variable given the preceding layer. This permits an exact block-transition law for both the neuron-level variables and, via kernel empirical averaging, for the empirical kernel $G^\ell$.

The exact update for $G^\ell$ naturally separates drift and fluctuation terms:

$$G^{\ell+1} = G^\ell + H^\ell + \varepsilon^2 J^\ell,$$

where $H^\ell$ captures the cross-term and $J^\ell$ represents the Gramian of increments. The finite-$n$ stochasticity is retained exactly through conditional covariances.

Hierarchical Closure and Collective SDE

After establishing the exact recursions, the authors introduce a hierarchical closure scheme:

• (GC0) Gaussian Closure: Approximates the law of single neurons as Gaussian conditioned on the empirical kernel $G^\ell$.
• (LIN) Linearization: Expands drift kernels to first order around the mean kernel $G^\ell$0 for transport terms.
• (GC1) Next-to-Leading-Order Closure: Expands the mean drift to second order to capture mean correction $G^\ell$1 as a one-loop diagrammatic term.

This closure yields a collective SDE for $G^\ell$2 in the diffusion limit, with mean dynamics for the average kernel

$G^\ell$3

and covariance evolution for the four-point kernel fluctuation

$G^\ell$4

Here, the transport operator $G^\ell$5 encodes the linearized response, while $G^\ell$6 accumulates stochasticity from layer to layer. The mean correction $G^\ell$7 is governed by a cubic vertex-induced tadpole:

$G^\ell$8

Figure 1: Trajectories of $G^\ell$9 and $\eta^\ell$0 compared with empirical kernel dynamics and theory for $\eta^\ell$1, highlighting the agreement for $\eta^\ell$2 and showing the accumulation of error for $\eta^\ell$3 at large $\eta^\ell$4.

Numerical Results: Window of Validity and Breakdown Localization

The closure hierarchy is validated via large-scale ensemble simulations. Numerical evidence shows that the mean kernel $\eta^\ell$5 is predicted accurately to all depths. However, finite-width corrections to covariance ($\eta^\ell$6) and mean ($\eta^\ell$7) reveal distinct breakdown regimes:

• The discrepancy in $\eta^\ell$8 accumulates super-linearly with time, reaching $\eta^\ell$9 error at $G^\ell$0. This error is robust across $G^\ell$1 and $G^\ell$2 sweeps, indicating that it arises from systematic breakdowns in the GC0+LIN closure rather than mere discretization or finite-size artifacts.
• Direct empirical measurements of the true noise source $G^\ell$3 confirm the accuracy of the theory's source term, isolating the failure in the transport (drift) term.

  Figure 2: $G^\ell$4-dependence of $G^\ell$5 trajectories shows predictive error is independent of discretization, implying the breakdown is intrinsic to the closure.

At the next order, the mean correction $G^\ell$6 (driven by the one-loop tadpole) fails for off-diagonal entries by a factor of 2-3 compared to empirical data, with the error present already at initialization. The failure is entirely due to the inadequacy of the source approximation in the GC1 closure for $G^\ell$7 (i.e., the induced NLO mean correction term).

Figure 3: Validation of $G^\ell$8, demonstrating theory-experiment agreement fails for off-diagonal components due to inaccuracies in the NLO source closure.

A depth-wise comparison of the source terms further confirms the theoretical $G^\ell$9 source systematically overestimates the empirical value at all depths.

Figure 4: Comparison of the $G^\ell$0 source at all depths isolates the systematic error in the field-theoretic one-loop source model.

Diagrammatic Interpretation and Field Theory Structure

The core recursions for $G^\ell$1, $G^\ell$2, and $G^\ell$3 are interpreted diagrammatically as a loop expansion in a background field theory. The leading-order fluctuation—propagated and sourced by the quadratic MSRJD action—generates covariance $G^\ell$4 through effective noise. The mean correction $G^\ell$5 arises from a one-loop drift cubically contracted with $G^\ell$6. The cubic noise vertex is shown to vanish under the Itō convention for the response field, highlighting the structural sufficiency of the hierarchy for the observables under consideration.

Implications for Theory and Practice

Theoretical Significance: The work establishes that an EFT using only the empirical kernel $G^\ell$7 has a finite validity window in depth, with failures in the predicted fluctuation and mean corrections that are both quantitatively and mechanistically localized. The breakdown occurs where higher-order moments and nonlinear functionals of the preactivations (in particular, the sigma-kernel $G^\ell$8 and its hierarchy) begin to impact drift and covariance. The core implication is that $G^\ell$9-only closure is insufficient for consistent, deep predictions at finite width.

Practical Implications: For theoretical tools describing finite-width initialization in deep ResNets, these results delineate the limits of presently popular kernel-based closures. Extensions capturing sigma-kernel dynamics or higher polynomial and non-Gaussian corrections are required to sustain accuracy at large depths. The explicit identification of where closure fails (transport for $$G^{\ell+1} = G^\ell + H^\ell + \varepsilon^2 J^\ell,$$0; source for $$G^{\ell+1} = G^\ell + H^\ell + \varepsilon^2 J^\ell,$$1) provides concrete targets for future finite-width, field-theoretic approaches.

Future Directions: The natural extension is towards a coupled $$G^{\ell+1} = G^\ell + H^\ell + \varepsilon^2 J^\ell,$$2 observable hierarchy, enabling non-Gaussian collective variables and closure schemes that track the joint drift of kernels and nonlinear statistics. Truncation and stochastic analysis for these coupled systems, including higher-order fluctuation corrections and their diagrammatic structure, remain open and crucial problems.

Conclusion

This work delivers a comprehensive, field-theoretic framework for the finite-width dynamics of pre-activation ResNets at initialization, centered on a $$G^{\ell+1} = G^\ell + H^\ell + \varepsilon^2 J^\ell,$$3-only collective kernel EFT. The analysis rigorously establishes the range of validity of Gaussian closure-based hierarchies, identifies the precise mechanism and onset of their failures, and motivates systematic extensions incorporating additional collective variables. For both theory and practice, these findings clarify the essential directions for beyond-kernel approaches in the finite-width analysis of deep neural networks.