Back-Propagating Kernel Renormalization (BPKR)

• BPKR is an analytical framework that exactly computes generalization error, representation statistics, and phase diagrams in deep linear networks and approximates behavior in nonlinear networks.
• It employs a layerwise backward integration to renormalize weight parameters, capturing the effects of architectural, training, and regularization choices on learning outcomes.
• The framework bridges Gaussian Process and NTK approaches, offering insights into phenomena such as double descent, feature alignment, and phase transitions in model capacity.

Back-Propagating Kernel Renormalization (BPKR) is an analytical framework designed to enable the exact calculation of generalization error, hidden-layer representation statistics, and phase diagrams of deep linear neural networks (DLNNs), as well as a robust approximation for certain nonlinear deep neural networks (DNNs). BPKR leverages ideas from statistical mechanics and kernel methods, implementing a layerwise backward integration (renormalization) of the neural network’s weight degrees of freedom to characterize information flow and the effect of architectural, training, and regularization choices on learning outcomes.

1. Motivation and Theoretical Context

The motivation for BPKR stems from the need for rigorous theoretical understanding of deep learning generalization and representation in overparameterized regimes, where traditional theories—especially for single-layer or shallow models—prove inadequate. In DLNNs, the mapping is globally linear but the parameterization induces a nonlinear learning problem due to the product structure of weight matrices across layers. Classical statistical mechanics approaches were limited either to the infinite-width or infinite-depth regime, or to shallow architectures. BPKR extends this by providing a tractable solution for DLNNs of arbitrary depth and width, irrespective of the input or label distribution, and enables calculation of both the average and higher moments of network predictions after training (Li et al., 2020).

2. Mathematical Structure and Renormalization Procedure

BPKR analyzes a supervised regression task with $P$ datapoints, where each datapoint $(x^\mu, y^\mu)$ is mapped to an output through an $L$-layer linear network. The learning objective is MSE with $L_2$ regularization. The central technical element is the partition function over the posterior weight distribution: $$P(\Theta) = \frac{1}{Z} e^{-E(\Theta)/T},\quad E(\Theta) = \frac{1}{2} \sum_{\mu=1}^P [f(x^\mu, \Theta) - y^\mu]^2 + \frac{T}{2\sigma^2} \Vert \Theta \Vert^2,$$ where $T$ is temperature and $\sigma^2$ governs regularization.

BPKR proceeds by systematically integrating out weight matrices in reverse order (from output layer to input), updating the effective Hamiltonian at each step. Upon integrating out the weights for layer $l$, the covariance (“kernel”) of representations at layer $l-1$ is renormalized: $K_{l-1} \to u_{l-1} K_{l-1},$ where $u_{l-1}$ is a scalar renormalization parameter, determined via a self-consistent mean-field equation: $$1 - \sigma^{-2} u_{l} = \alpha (1 - u_l^{-l} r_{l}),$$ with $\alpha = P/N$ (training set to width ratio), $N$ being layer width, and $r_{l}$ the normalized quadratic output at layer $l$. The process repeats until the input layer is reached, at which point all weight degrees of freedom have been integrated out and the recursive relation for $u_0$ encapsulates the learning statistics of the network.

3. Analytical Results: Generalization and Phase Diagrams

BPKR provides exact analytical expressions for the mean and variance of network predictions: $$\left\langle f(x) \right\rangle = k_0^\top(x) K_0^{-1} Y, \qquad \left\langle \delta f(x)^2 \right\rangle = u_0^L \left[ K_0(x,x) - k_0^\top(x) K_0^{-1} k_0(x) \right],$$ where $k_0(x)$ is the vector of input-layer covariances between $x$ and the training set. This separation yields the following insights:

• The predictor mean matches the infinite-width, infinite-depth “neural tangent kernel” (NTK) regime, being independent of depth and width.
• The variance, and thus the generalization error, depend critically on network depth and width via $u_0^L$.
• Analytical phase diagrams for bias/variance as a function of width ($N$), depth ($L$), data complexity, regularization ($\sigma^2$), and stochasticity emerge, and permit identification of settings (e.g., the “interpolation threshold”) where generalization can benefit or deteriorate with increasing model capacity.

4. Hidden Layer Representations and Feature Learning

By halting the integration at an intermediate layer $l$, BPKR characterizes the emergent kernel at that layer as: $$\langle K_l \rangle = \sigma^{2l} (1-1/N)^l K_0 + \frac{m_l}{N} YY^\top,$$ with $K_0$ the input kernel and $m_l$ an order parameter. This quantifies how label information (“task structure”)—captured by the rank-one $YY^\top$ term—steadily becomes more prominent as one moves deeper into the network, even though the mapping is linear. The result demonstrates that deep representations in DLNNs become increasingly “aligned” with the task. For networks with multiple outputs, the contribution from $YY^\top$ generalizes to a sum over task eigenvectors, showing that depth can modulate feature selectivity.

5. Extension to Nonlinear Networks and Universality

While BPKR provides exact results for deep linear networks, it can be heuristically extended to nonlinear networks with ReLU activation. The extension employs Gaussian Process (GP) kernel recursion for finite depth ReLU networks and modifies all GP kernels by a single scalar renormalization parameter $u_0$ analogous to the linear case: $$1 - \sigma^{-2} u_0 = \alpha(1 - u_0^{-L} r_0), \qquad r_0 = \frac{\sigma^{2L}}{P} Y^\top \langle K_L^{GP} \rangle^{-1} Y.$$ Predictions for the mean and variance closely track empirical results for moderately wide, moderately deep ReLU networks, capturing double-descent phenomena and delineating the impact of under- and overparameterization on learning outcomes. Deviations arise in very deep or extremely narrow regimes, attributable to higher-order (non-scalar) kernel distortions.

6. Comparison to Related Kernel Renormalization Approaches

BPKR’s global kernel renormalization is especially pertinent for fully connected architectures, in which the entire kernel is multiplicatively adjusted by a scalar at each layer. This is in marked contrast to convolutional architectures, where local kernel renormalization—parametrized by matrices rather than scalars—allows fine-grained, spatially dependent feature learning at finite width. In overparameterized CNNs, the ability to locally renormalize patch-to-patch interactions endows the model with superior feature selectivity compared to FCNs, which, according to recent theoretical results, are limited to trivial global adjustments (Aiudi et al., 2023).

The BPKR framework can be seen as a bridge between the analytical tractability of Gaussian Process and NTK approaches (valid in the infinite-width limit) and the nuanced, finite-width-dependent behavior observable in practical deep networks.

7. Broader Implications, Limitations, and Future Directions

BPKR establishes, for the first time, an exact analytical handle on the post-training statistics of deep (linear) neural networks and suggests a unifying path between statistical mechanics, kernel theory, and deep learning. It demonstrates that network depth primarily amplifies variance (with the mean controlled by shallower statistics), and that feature representations deepen alignment with task structure layerwise even in absence of nonlinearity.

A plausible implication is that many empirically observed phenomena in overparameterized linear and shallow nonlinear networks—such as double descent, task-dependent representation sharpening, and resilience to overfitting—can be anticipated and quantified via closed-form expressions derived from BPKR. Limitations persist for highly nonlinear, highly non-Gaussian or strongly localized kernel scenarios, where only numerical or diagrammatic (QFT-inspired) approaches can capture the full range of phenomena.

Ongoing work focuses on extending BPKR to nonlinear, non-translation-invariant kernels, nontrivial architectures, and leveraging more advanced renormalization tools (e.g., functional RG, 2PI formalisms) to capture higher-order interactions and function space flows. These theoretical developments are expected to yield further insight into the universality and limitations of kernel-based learning in modern neural architectures.

---

Aspect	BPKR (DLNNs/FCNs)	Local Kernel Renormalization (CNNs)
Kernel update per layer	Scalar (global renormalization)	Matrix (local/patch-wise renorm.)
Feature learning at finite width	Absent (trivial scaling)	Present (data-dependent)
Analytical tractability	Exact (DLNNs); heuristic (ReLU NNs)	Requires richer description