---
title: Infinite-Width Bayesian Neural Networks
url: https://www.emergentmind.com/topics/infinite-width-bayesian-neural-networks
type: topic
---

# Infinite-Width Bayesian Neural Networks

Infinite-width Bayesian neural networks (BNNs) are a fundamental limiting regime in probabilistic deep learning, where the hidden-layer widths of neural architectures tend to infinity. In this setting, the random-function prior over outputs typically converges to a Gaussian process (GP), allowing closed-form Bayesian inference for regression and classification tasks. This correspondence relies on the central limit theorem under suitable scaling of weight variances, and has been extended to various architectures including convolutional networks, tensor networks, and deep linear networks. Recent work explores non-Gaussian generalizations, especially via heavy-tailed priors or more sophisticated hierarchical variance models, yielding richer stable or Student-t process limits. Uncertainty quantification, posterior inference, and MCMC sampling techniques all benefit from the mathematical tractability of infinite-width BNNs, but practical generalization differences remain between finite-width and infinite-width models, especially under model misspecification.

## 1. Mathematical Foundations: GP and Stable Process Limits

For fully-connected feedforward networks with $L$ layers, input dimension $d_{in}$, output dimension $d_{out}$, and layer widths $n_1,...,n_{L-1}$, the pre-activations at layer $\ell$ take the form:
$$
f^{(0)}(x)=x,\qquad
f^{(\ell)}(x) = W^{(\ell)}\,\varphi_\ell\bigl(f^{(\ell-1)}(x)\bigr) + b^{(\ell)},\quad \ell=1,\dots,L,
$$
with Gaussian priors on weights $W^{(\ell)}_{ij}\sim\mathcal{N}(0,\sigma_W^2/n_{\ell-1})$ and biases $b^{(\ell)}_i\sim\mathcal{N}(0,\sigma_b^2)$ [2502.04247][1912.02803][2201.04746]. As $n_1,...,n_{L-1}\to\infty$, finite-dimensional distributions of $f(x)$ converge to a GP with kernel $K$ recursively constructed via integrals over (joint) Gaussian random variables, for instance:
$$
K^{(\ell)}(x,x') = \sigma_W^2\,\mathbb{E}_{(u,v)\sim\mathcal{N}_2(0,\,K^{(\ell-1)})}[\varphi_\ell(u)\varphi_\ell(v)] + \sigma_b^2.
$$
This extends to deep architectures, convolutional networks, tensor-network models, and can be implemented at scale (with GPU or TPU acceleration) leveraging libraries like Neural Tangents [1912.02803].

However, when instead prior weights are drawn from symmetric $\alpha$-stable laws with infinite variance, the limiting function is a multivariate stable process, not a Gaussian process. In such cases, the process is specified via its characteristic function (no covariance), and conditionally Gaussian representations—where the stable law is realized as a scale mixture of Gaussians—enable tractable inference and feature learning [2305.10664][2410.01284].

## 2. Posterior Inference and Hierarchical Variance Models

BNNs with Gaussian weight priors allow posterior inference via closed-form GP regression. When a hierarchical prior is placed on the last-layer weights and the likelihood variance—for instance, sharing a random variance $\tau\sim\mathrm{InvGamma}(\alpha,\beta)$—the infinite-width limit of the posterior over outputs is a Student-t process [2502.04247]. Marginalizing $\tau$ yields a multivariate Student-t density:
$$
p(f(X)|y_D) = \mathrm{StudentT}\left(\nu=2\alpha',\,\mu_n,\,\frac{\beta'}{\alpha'}\Sigma_n\right),
$$
with heavier polynomial tails than a GP, more robust uncertainty quantification, and tunable degrees of freedom $\nu$.

For networks with unbounded (infinite variance) priors, posterior inference leverages a conditionally Gaussian representation via a positive stable mixing variable:
$$
z_j^{(\ell)}(x) \sim \mathcal{N}(0,\,s_+^{(\ell)}\Sigma^{(\ell)}),
\quad s_+^{(\ell)}\sim S^+_{\alpha/2},
$$
and inference proceeds by sampling latent scales $s_+$ via Metropolis-Hastings, updating layer-wise random kernels recursively [2410.01284].

## 3. Uncertainty Quantification and Calibration

Infinite-width BNNs yield GPs with deterministic kernels, affording exact closed-form uncertainty quantification via GP formulas for posterior mean and covariance [2201.04746][1912.02803]. The NNGP view treats network outputs as draws from a GP specified by the infinite-width kernel; the NTK construction—for gradient-descent-trained networks—yields an alternative GP kernel expressing training-time dynamics.

With Student-t or stable process limits, uncertainty bands become heavier-tailed, offering greater robustness to outliers and small-sample effects. In practice, marginalizing hierarchical variances or stable mixing variables yields uncertainty intervals with closer-to-nominal coverage in challenging non-smooth or discontinuous regression tasks [2502.04247][2305.10664][2410.01284].

Overconfidence in standard finite-width ReLU BNNs—manifested as output variance growing only quadratically in distance from the training set—is corrected in the infinite-feature limit, where cubic variance growth ensures softmax probabilities converge to $1/C$ far away from the data, thereby “maximal uncertainty" [2010.02709].

Empirical and theoretical calibration metrics such as NLL, Brier score, and Expected Calibration Error (ECE) confirm the superior out-of-distribution calibration of infinite-width GP-based BNNs under both regression and classification, especially for OOD data and distributional shift [2010.07355].

## 4. Comparison to Finite-Width Bayesian Neural Networks

Finite-width BNNs define non-Gaussian, heavier-tailed priors and exhibit subtle dependence effects between units that vanish only in the infinite-width limit [2211.09184][2111.14397][2303.02859]. Empirical studies demonstrate finite BNNs have more flexible frequency spectra and may generalize better than NNGP models under model mismatch, especially for processes with significant high-frequency structure. However, inference in finite-width BNNs is intractable for large architectures, requiring expensive MCMC or variational techniques.

Depth in finite linear BNNs induces data-dependent scale averaging—each output channel is a scale mixture of GPs—which collapses to a single GP as width increases [2111.11954]. Bottleneck architectures (finite intermediate layer widths) retain discriminative, input-dependent posteriors and output dependence even in deep limits, contrasting with total independence in fully infinite-width networks [2001.00921].

## 5. Stochastic Sampling: Function-space MCMC and Efficient Scaling

High-dimensional parameter-space MCMC methods often degenerate as network width increases; acceptance rates and mixing collapse unless proposal step sizes shrink with dimension. Function-space MCMC algorithms such as preconditioned Crank-Nicolson (pCN/pCNL)—formulated to preserve Gaussian measures in Hilbert space—circumvent this issue: as layer widths $\to\infty$, acceptance rates approach unity and effective sample sizes per step scale favorably [2408.14325]. Empirical results show that for wide networks, pCN/pCNL samplers offer dramatically improved mixing over standard Langevin MCMC, and are well posed in both function and parameter space.

Libraries such as Neural Tangents automate the practical implementation of infinite-width inference, supporting both GPU/TPU acceleration and batched computation for posterior mean and covariance extraction [1912.02803].

## 6. Non-Gaussianity and Advanced Kernel Processes

Recent advances extend the classical infinite-width Gaussian regime to non-Gaussian processes driven by heavy-tailed priors, nontrivial variance hierarchies, or finite-feature corrections. Edgeworth expansions yield analytic quantification of kurtosis and non-Gaussian corrections at order $1/N$ in finite-width networks [2303.02859], while scale-mixture representations (via stable laws) generalize standard kernel learning to stochastic kernel processes that encode representation learning in the infinite-width limit [2410.01284].

Deep kernel processes under infinite variance priors admit recursive kernel construction analogous to Cho & Saul (2009), but with layer-wise conditional random covariance matrices linked by stable mixing variables and learned directly from data, enabling scalable and adaptive posterior inference well beyond conventional GP regression.

## 7. Practical Applications and Limitations

Infinite-width Bayesian neural networks underpin tractable uncertainty quantification methods, favor scalable function-space inference, and serve as a principled basis for transfer learning via GP posteriors on pre-trained embeddings [2010.07355]. Student-t and stable process generalizations provide heavier-tailed, robust models for discontinuous, outlier-prone data, and bottleneck architectures improve multi-output dependence and discriminative feature learning.

Limitations include the deterministic nature of GP/NNGP kernels in the classical Gaussian regime—precluding representation learning unless heavy-tailed or hierarchical priors are used—and limited scalability of fully Bayesian inference for finite-width architectures. Open directions involve quantifying optimal width-depth trade-offs under data/noise constraints, devising scalable non-Gaussian inference for large networks, and integrating adaptive kernel learning into function-space modeling for high-dimensional tasks.

---

**Key References:**  
- Student-t processes in infinite-width BNN posteriors [2502.04247]  
- Bottleneck and composition phenomena [2001.00921]  
- Stable process limits and conditionally Gaussian kernels [2410.01284][2305.10664]  
- Infinite-feature corrections for ReLU BNNs [2010.02709]  
- Function-space MCMC in wide BNNs [2408.14325]  
- Uncertainty quantification and calibration [2201.04746][2010.07355]  
- Neural Tangents implementation [1912.02803]  
- Empirical spectrum and generalization advantages [2211.09184][2111.14397][2303.02859]  
- Scale-averaging in deep linear BNNs [2111.11954]

Source: https://www.emergentmind.com/topics/infinite-width-bayesian-neural-networks