Renormalized NNGP Kernel in Bayesian Neural Networks

• The paper introduces that the renormalized NNGP kernel modifies the infinite-width baseline to capture finite-width, data-dependent output correlations via a saddle-point derived order parameter.
• It details how the proportional limit retains 1/N effects by rescaling the kernel shape with a nontrivial Q*, thereby accounting for inter-output couplings absent in lazy-training regimes.
• The study compares renormalization in Bayesian MLPs, NN–QFT, and practical GP regression, highlighting its impact on generalization performance and predictive mean-variance behavior.

The renormalized NNGP kernel is a modification of the neural network Gaussian process kernel that incorporates effects absent in the strict infinite-width limit. In Bayesian one-hidden-layer networks with multiple readout neurons, the central finite-width effect is a data-dependent deformation of the infinite-width NNGP covariance by an output-space order parameter, which explains output-output correlations that vanish in the lazy-training regime (Baglioni et al., 2024). In related parts of the literature, the same phrase also refers to renormalization-group corrections in the neural network–quantum field theory correspondence, and to variance normalization procedures required to make practical NNGP kernels valid for Gaussian-process regression (Erbin et al., 2022, Muyskens et al., 2024). In deeper Bayesian architectures, an analogous proportional-regime construction rescales the large-width kernel through low-dimensional Wishart order parameters (Baglioni et al., 28 May 2026).

1. Infinite-width baseline and proportional-width setting

For a one-hidden-layer fully connected network with input dimension $N_0$, hidden width $N_1$, output dimension $D$, activation $\phi$, hidden weights $w\in\mathbb R^{N_1\times N_0}$, and readout weights $v\in\mathbb R^{D\times N_1}$ drawn i.i.d. Gaussian, the output is

$$f_a(x)=\sum_{i=1}^{N_1} v_{a,i}\,\phi\!\left(\frac{w_i\cdot x}{\sqrt{N_0}}\right)\frac{1}{\sqrt{N_1}}.$$

In the $N_1\to\infty$ limit, any two outputs become independent Gaussian processes with covariance

$$K^{(\infty)}(x,x') \equiv \mathbb E_w\!\left[\phi\!\left(\frac{w\cdot x}{\sqrt{N_0}}\right)\phi\!\left(\frac{w\cdot x'}{\sqrt{N_0}}\right)\right].$$

This is the standard NNGP kernel for the one-hidden-layer model (Baglioni et al., 2024).

The proportional limit changes the asymptotic regime. Instead of sending width to infinity at fixed dataset size, one takes the number of training examples $P$ and the hidden width $N_1$0 to infinity together while keeping

$N_1$1

finite. This regime retains the leading $N_1$2 effects that survive when $N_1$3 grows proportionally to $N_1$4. In the manuscript on kernel shape renormalization, these surviving terms are precisely what account for the non-trivial output-output correlations seen in finite Bayesian one-hidden-layer networks (Baglioni et al., 2024).

A basic consequence is that the infinite-width statement “different outputs are uncorrelated Gaussians” ceases to be exact once $N_1$5 is finite. The renormalized NNGP kernel is the object that re-expresses this finite-width, finite-sample coupling in kernel form.

2. Saddle-point derivation of kernel shape renormalization

The Bayesian formulation starts from the partition function

$N_1$6

with $N_1$7 and $N_1$8 the MSE over $N_1$9 examples. In the joint $D$0 limit at fixed $D$1, one introduces a $D$2 positive-definite order parameter $D$3 and obtains, by saddle-point methods together with a Gaussian-equivalence theorem for the pre-activations,

$D$4

The action is

$D$5

where $D$6 is the $D$7 input matrix, $D$8 the $D$9 label vector, and

$\phi$0

In components,

$\phi$1

This form of the action is stated in the manuscript as following Pacelli et al., Nature Machine Intelligence (2023) (Baglioni et al., 2024).

The renormalized kernel is obtained by evaluating the integral at its minimizer $\phi$2. At that saddle,

$\phi$3

The renormalization is therefore not a change in the scalar input-space covariance alone; it is a deformation of the kernel’s shape in output space. At $\phi$4, one recovers the standard lazy-training result because the saddle is $\phi$5. At finite $\phi$6, the minimizer is nontrivial, $\phi$7, and the effective kernel acquires inter-output structure (Baglioni et al., 2024).

3. Perturbative structure, assumptions, and asymptotic limits

The order parameter can be expanded perturbatively as

$\phi$8

so that the renormalized kernel becomes

$\phi$9

The manuscript provides a first-order closed expression for $w\in\mathbb R^{N_1\times N_0}$0 in terms of $w\in\mathbb R^{N_1\times N_0}$1, $w\in\mathbb R^{N_1\times N_0}$2, $w\in\mathbb R^{N_1\times N_0}$3, the labels $w\in\mathbb R^{N_1\times N_0}$4, and the matrices $w\in\mathbb R^{N_1\times N_0}$5; the resulting expansion is explicitly described as “1-loop in $w\in\mathbb R^{N_1\times N_0}$6” (Baglioni et al., 2024).

Two asymptotic limits organize the interpretation. First, as $w\in\mathbb R^{N_1\times N_0}$7, one has $w\in\mathbb R^{N_1\times N_0}$8, hence $w\in\mathbb R^{N_1\times N_0}$9, and the shape renormalization vanishes. Second, as $v\in\mathbb R^{D\times N_1}$0, the prior becomes negligible and $v\in\mathbb R^{D\times N_1}$1 is dominated by the $v\in\mathbb R^{D\times N_1}$2 term; in practice, off-diagonal entries of $v\in\mathbb R^{D\times N_1}$3 can become $v\in\mathbb R^{D\times N_1}$4, leading to strong inter-output coupling (Baglioni et al., 2024).

The derivation rests on two explicit approximations. The first is Gaussian equivalence of the pre-activations in the $v\in\mathbb R^{D\times N_1}$5 limit, identified in the manuscript with Breuer–Major type CLTs. The second is the saddle-point evaluation of the $v\in\mathbb R^{D\times N_1}$6 integral, which neglects $v\in\mathbb R^{D\times N_1}$7 fluctuations around $v\in\mathbb R^{D\times N_1}$8 (Baglioni et al., 2024). These assumptions are important because they delimit the regime in which the renormalized kernel is expected to be quantitatively accurate.

4. Output-output correlations, weight overlaps, and predictive consequences

In the proportional-limit formulation, the off-diagonal entries of $v\in\mathbb R^{D\times N_1}$9 encode the correlations between different outputs. In the infinite-width regime, $$f_a(x)=\sum_{i=1}^{N_1} v_{a,i}\,\phi\!\left(\frac{w_i\cdot x}{\sqrt{N_0}}\right)\frac{1}{\sqrt{N_1}}.$$0, so off-diagonals vanish and different outputs are uncorrelated Gaussians. At finite $$f_a(x)=\sum_{i=1}^{N_1} v_{a,i}\,\phi\!\left(\frac{w_i\cdot x}{\sqrt{N_0}}\right)\frac{1}{\sqrt{N_1}}.$$1, nonzero $$f_a(x)=\sum_{i=1}^{N_1} v_{a,i}\,\phi\!\left(\frac{w_i\cdot x}{\sqrt{N_0}}\right)\frac{1}{\sqrt{N_1}}.$$2 quantify data-induced correlations between outputs $$f_a(x)=\sum_{i=1}^{N_1} v_{a,i}\,\phi\!\left(\frac{w_i\cdot x}{\sqrt{N_0}}\right)\frac{1}{\sqrt{N_1}}.$$3 and $$f_a(x)=\sum_{i=1}^{N_1} v_{a,i}\,\phi\!\left(\frac{w_i\cdot x}{\sqrt{N_0}}\right)\frac{1}{\sqrt{N_1}}.$$4 (Baglioni et al., 2024).

The same information can be read directly in the geometry of the last-layer weights:

$$f_a(x)=\sum_{i=1}^{N_1} v_{a,i}\,\phi\!\left(\frac{w_i\cdot x}{\sqrt{N_0}}\right)\frac{1}{\sqrt{N_1}}.$$5

Thus the renormalized kernel does not merely reproduce output covariance phenomenology; it also measures overlaps between the readout vectors of distinct classes. In this sense, kernel shape renormalization restores feature-learning effects that are absent in the pure lazy limit (Baglioni et al., 2024).

The predictive role is equally direct. The manuscript states that the same renormalized kernel appears in the predictive mean and variance of the GP posterior. Consequently, nontrivial $$f_a(x)=\sum_{i=1}^{N_1} v_{a,i}\,\phi\!\left(\frac{w_i\cdot x}{\sqrt{N_0}}\right)\frac{1}{\sqrt{N_1}}.$$6 can improve or worsen generalization depending on whether the induced inter-output structure matches true label correlations (Baglioni et al., 2024).

The reported numerical experiments test both generalization and correlations. For synthetic, MNIST, and CIFAR10 inputs, Figure 1 compares the predicted generalization loss

$$f_a(x)=\sum_{i=1}^{N_1} v_{a,i}\,\phi\!\left(\frac{w_i\cdot x}{\sqrt{N_0}}\right)\frac{1}{\sqrt{N_1}}.$$7

against Langevin-sampled Bayesian one-hidden-layer networks and finds excellent agreement for hidden widths up to a few thousand. Relative errors on $$f_a(x)=\sum_{i=1}^{N_1} v_{a,i}\,\phi\!\left(\frac{w_i\cdot x}{\sqrt{N_0}}\right)\frac{1}{\sqrt{N_1}}.$$8 are typically below a few percent for $$f_a(x)=\sum_{i=1}^{N_1} v_{a,i}\,\phi\!\left(\frac{w_i\cdot x}{\sqrt{N_0}}\right)\frac{1}{\sqrt{N_1}}.$$9. Figure 2 compares the predicted $N_1\to\infty$0 against the Monte Carlo average overlaps $N_1\to\infty$1 for CIFAR10 with $N_1\to\infty$2 at $N_1\to\infty$3 and $N_1\to\infty$4; both diagonal and off-diagonal elements match quantitatively, including negative overlaps among confusable classes. For $N_1\to\infty$5, off-diagonal $N_1\to\infty$6 for visually similar CIFAR classes such as “cars” versus “trucks” (Baglioni et al., 2024).

5. Distinct meanings of “renormalized NNGP kernel” in the literature

The literature uses the term in several technically distinct senses.

Setting	Renormalized object	Mechanism
Proportional Bayesian one-hidden-layer network (Baglioni et al., 2024)	$N_1\to\infty$7	Output-space shape renormalization by saddle-point matrix
NN–QFT correspondence (Erbin et al., 2022)	$N_1\to\infty$8	One-loop finite-$N_1\to\infty$9 interaction, counterterms, RG flow
Practical GP use of infinite-width ReLU kernel (Muyskens et al., 2024)	$$K^{(\infty)}(x,x') \equiv \mathbb E_w\!\left[\phi\!\left(\frac{w\cdot x}{\sqrt{N_0}}\right)\phi\!\left(\frac{w\cdot x'}{\sqrt{N_0}}\right)\right].$$0	Variance normalization or unit-sphere embedding

In the neural network–quantum field theory correspondence, Erbin et al. describe the infinite-width limit as a free field theory and finite-$$K^{(\infty)}(x,x') \equiv \mathbb E_w\!\left[\phi\!\left(\frac{w\cdot x}{\sqrt{N_0}}\right)\phi\!\left(\frac{w\cdot x'}{\sqrt{N_0}}\right)\right].$$1 corrections as interactions. For a one-dimensional translation-invariant Gaussian-activation kernel with $$K^{(\infty)}(x,x') \equiv \mathbb E_w\!\left[\phi\!\left(\frac{w\cdot x}{\sqrt{N_0}}\right)\phi\!\left(\frac{w\cdot x'}{\sqrt{N_0}}\right)\right].$$2, the bare momentum-space kernel is

$$K^{(\infty)}(x,x') \equiv \mathbb E_w\!\left[\phi\!\left(\frac{w\cdot x}{\sqrt{N_0}}\right)\phi\!\left(\frac{w\cdot x'}{\sqrt{N_0}}\right)\right].$$3

and the leading finite-width correction is a local $$K^{(\infty)}(x,x') \equiv \mathbb E_w\!\left[\phi\!\left(\frac{w\cdot x}{\sqrt{N_0}}\right)\phi\!\left(\frac{w\cdot x'}{\sqrt{N_0}}\right)\right].$$4 interaction with bare coupling $$K^{(\infty)}(x,x') \equiv \mathbb E_w\!\left[\phi\!\left(\frac{w\cdot x}{\sqrt{N_0}}\right)\phi\!\left(\frac{w\cdot x'}{\sqrt{N_0}}\right)\right].$$5. The one-loop self-energy is

$$K^{(\infty)}(x,x') \equiv \mathbb E_w\!\left[\phi\!\left(\frac{w\cdot x}{\sqrt{N_0}}\right)\phi\!\left(\frac{w\cdot x'}{\sqrt{N_0}}\right)\right].$$6

and after subtraction the renormalized kernel at scale $$K^{(\infty)}(x,x') \equiv \mathbb E_w\!\left[\phi\!\left(\frac{w\cdot x}{\sqrt{N_0}}\right)\phi\!\left(\frac{w\cdot x'}{\sqrt{N_0}}\right)\right].$$7 is written as

$$K^{(\infty)}(x,x') \equiv \mathbb E_w\!\left[\phi\!\left(\frac{w\cdot x}{\sqrt{N_0}}\right)\phi\!\left(\frac{w\cdot x'}{\sqrt{N_0}}\right)\right].$$8

Holding the bare variance fixed yields the beta function

$$K^{(\infty)}(x,x') \equiv \mathbb E_w\!\left[\phi\!\left(\frac{w\cdot x}{\sqrt{N_0}}\right)\phi\!\left(\frac{w\cdot x'}{\sqrt{N_0}}\right)\right].$$9

In this usage, renormalization means scale dependence of the kernel induced by finite-width interactions (Erbin et al., 2022).

In practical Gaussian-process regression with infinite-width ReLU kernels, “renormalization” instead denotes variance normalization. The unnormalized covariance is $P$0, and the normalized kernel is

$P$1

The paper recommends embedding each coordinate by

$P$2

and then applying $P$3 normalization, which enforces $P$4 constant over all $P$5 and keeps every coordinate in $P$6. The same work emphasizes several numerical pathologies: positive-definiteness failures, a valid hyperparameter region that shrinks rapidly with depth, and near-singularity even inside the valid region. Its recommended mitigations are unit-hypersphere normalization, fine-grid verification of positive definiteness via Cholesky decomposition, restriction to small depth such as $P$7, and addition of a modest nugget $P$8 (Muyskens et al., 2024).

That practical study also argues that the renormalized NNGP correlation behaves very much like a Matérn-$P$9 correlation. The small-distance expansion of $N_1$00 is

$N_1$01

which matches the Matérn-$N_1$02 derivative at the origin when $N_1$03. On Friedman, MODIS, and Borehole benchmarks, each run $N_1$04 times with $N_1$05 training and $N_1$06 test points, the minimum RMSEs reported for NNGP were $N_1$07, $N_1$08, and $N_1$09; for Matérn $N_1$10 with fixed $N_1$11, $N_1$12, $N_1$13, and $N_1$14; and for optimized Matérn $N_1$15, $N_1$16, $N_1$17, and $N_1$18. The maximum RMSE for NNGP on MODIS was approximately $N_1$19 because of near-singularity episodes, while weight-difference statistics indicated that Matérn-$N_1$20 reproduced NNGP kriging weights almost exactly (Muyskens et al., 2024).

6. Deep proportional-regime extensions and local renormalization in CNNs

A recent extension generalizes proportional-regime kernel renormalization from shallow networks to Bayesian MLPs of fixed depth $N_1$21 through the Equivalent Wishart Ansatz (EWA). For a fully connected Bayesian DNN with Gaussian priors of precision $N_1$22, squared-error likelihood at inverse temperature $N_1$23, data Gram matrix

$N_1$24

and one-layer NNGP map

$N_1$25

the proportional limit $N_1$26 with fixed $N_1$27 yields an effective GP posterior with renormalized kernel

$N_1$28

The order parameters $N_1$29 are obtained by minimizing an effective action $N_1$30 (Baglioni et al., 28 May 2026).

The EWA replaces the full hierarchy of empirical kernel fluctuations by Wishart variables with the same mean. This leads to an effective action of the form

$N_1$31

with $N_1$32 and

$N_1$33

The saddle-point equations are

$N_1$34

When all hidden layers have the same width, symmetry gives $N_1$35 for all $N_1$36, so $N_1$37. At zero temperature, defining

$N_1$38

the shallow case $N_1$39 has the explicit solution

$N_1$40

while for general $N_1$41 one obtains

$N_1$42

The Bayes-optimal predictor then has exactly the mean and variance of GP regression with kernel $N_1$43 (Baglioni et al., 28 May 2026).

The same paper extends the construction to convolutional architectures. For CNNs, the renormalization becomes local rather than global: instead of a single scalar or a single output-space matrix, one obtains a patch-patch matrix structure. In the one-hidden-layer CNN derivation, the prior characteristic function reduces to an integral over a Wishart order-parameter matrix $N_1$44, and for deep CNNs the effective kernel takes the form

$N_1$45

with $N_1$46 built from layerwise Wishart factors. Empirically, the EWA theory is tested against posterior sampling in finite deep networks with depths $N_1$47 and $N_1$48 on classic benchmark datasets, where it shows overall very good agreement together with two distinct types of systematic deviations (Baglioni et al., 28 May 2026).

Taken together, these results define the renormalized NNGP kernel as the principal mechanism by which finite width, finite sample size, or explicit normalization modifies the kernel inherited from infinite-width theory. In shallow Bayesian networks this modification is a nontrivial output-space deformation; in the NN–QFT correspondence it is an RG flow of kernel parameters; in practical GP implementations it is a variance normalization required for positive-definite covariance matrices; and in deep proportional-width models it becomes a low-dimensional or local renormalization of the hierarchical kernel itself.