Bayesian Representation Learning Limit

• Bayesian representation learning limit is the regime where Bayesian models balance prior and data likelihood to enable adaptive feature learning instead of fixed kernel responses.
• The framework unifies analysis across deep linear models, Gaussian processes, and transformers, showing how changes in width, depth, and prior trigger feature adaptation or collapse.
• This precise characterization has led to new regularization techniques and optimality results, informing the bias–variance trade-off in state-of-the-art Bayesian models.

The Bayesian representation learning limit describes the boundary between regimes in which Bayesian neural networks and related models can and cannot perform data-dependent feature learning, particularly as a function of architectural parameters such as width, depth, and prior. This concept unifies analysis across deep linear models, neural networks with nonlinearities, deep Gaussian processes (DGPs), and even position-aware transformers, precisely quantifying when and how internal representations adapt to the data under Bayesian inference, and when they collapse onto non-adaptive kernel methods. The characterization of this limit has led to new theoretical frameworks, optimality results, and practical regularization schemes.

1. Defining the Bayesian Representation Learning Limit

In standard infinite-width limits of Bayesian neural networks (NNGP; e.g., Neal's theorem), the hidden-layer widths tend to infinity at fixed data size, yielding models whose predictive distributions are governed by fixed, data-independent kernels. In these kernel limits, the model posterior depends on the data only through the last readout layer; all hidden representations are effectively "frozen," precluding genuine feature learning. This is most striking in fully Bayesian deep linear and deep Gaussian process models: for any fixed finite data set, the feature map converges to a deterministic function under the prior, and the posterior over hidden features collapses (Zavatone-Veth et al., 2022, Yang et al., 2021).

The "Bayesian representation learning limit" refers to a theoretically and practically meaningful scaling regime in which the trade-off between prior and data-likelihood is maintained even as width increases, allowing non-trivial adaptation of internal representations. This can be achieved either by (i) targeted scaling of both hidden and output dimensions (Yang et al., 2021), or (ii) recognizing that the first-order corrections to the infinite-width (kernel) limit encode small but quantifiable, often universal, representation-learning effects (Zavatone-Veth et al., 2022, Zavatone-Veth et al., 2021). This nuanced boundary has been addressed in both linear and nonlinear Bayesian models.

2. Feature Learning in Deep Linear Models: Large-Width and Finite-Width Regimes

For deep Bayesian linear neural networks trained on Gaussian data, a comprehensive asymptotic analysis reveals the precise scaling of feature learning (Zavatone-Veth et al., 2022). In the proportional high-dimensional limit (hidden widths $n_l$, data dimension $d$, and sample size $p$ all scaling linearly), the Bayes error can be decomposed as:

$$\epsilon = \epsilon_{\mathrm{LR}} + \Delta\epsilon,$$

where $\epsilon_{\mathrm{LR}}$ is the generalization error of linear regression (the kernel limit) and $\Delta\epsilon$ encodes finite-width corrections. The leading-order correction is universal:

$$\Delta\epsilon = \mathcal{C} \sum_{\ell=1}^L \frac{\alpha}{\gamma_\ell} + \mathcal{O}\left(\sum \frac{\alpha^2}{\gamma_\ell^2}\right), \quad \gamma_\ell = n_\ell/d,$$

and is identical for random-feature (RF) models and fully trained deep networks (NN), implying that representation learning is "invisible" at this order. Only at subleading order ($1/\gamma^2$) does a difference arise:

$$\epsilon_{\mathrm{RF}} - \epsilon_{\mathrm{NN}} = \frac{L(L+1)}{2\,\tilde\sigma^2} \frac{\alpha^2}{\gamma^2} + \mathcal{O}(\gamma^{-3}),$$

quantifying that representation learning is a vanishing $1/n^2$ effect in wide networks, and largely negligible in practical high-dimensional settings (Zavatone-Veth et al., 2022, Zavatone-Veth et al., 2021). This rigorously establishes the Bayesian representation learning limit in deep linear settings.

3. Nonlinear, Deep, and Gaussian Process Models: Beyond the Kernel Regime

The collapse of representation learning in standard NNGP and DGP infinite-width limits has been rigorously characterized (Yang et al., 2021, Zavatone-Veth et al., 2021). For a DGP with $d$0 layers and widths $d$1, if only the hidden widths $d$2 tend to infinity while the output width remains fixed, the posterior over Gram matrices collapses to the prior chain of kernels, eliminating data flow to the features.

The Bayesian representation learning limit can be restored, even in the infinite-width setting, by simultaneously scaling the output width to infinity. This modifies the Bayesian optimization to retain a nontrivial data likelihood term at all layers. The resulting DKM (Deep Kernel Machine) objective is:

$d$3

and has a unique maximizer that depends on the data $d$4, re-enabling representation learning in the limit (Yang et al., 2021). The posterior remains exactly Gaussian for DGPs under this construction. This formulation generalizes shallow kernel methods and allows for scalable inducing-point approximations.

For networks with nonlinearities and finite but large width, the leading $d$5 correction to the posterior kernel is universal and encodes explicit, model-independent feature adaptation (Zavatone-Veth et al., 2021). The limiting behavior captures how learning signals modulate feature covariances, and these corrections have been computed analytically for fully connected, convolutional, and shallow nonlinear architectures.

4. Statistical and Algorithmic Implications

The prevalence or absence of representation learning has direct implications for generalization performance, optimization, and the double descent phenomenon. In large-width (kernel) limits, risk is minimized by maximizing or minimizing widths, depending on prior–target scale alignment, and no interior minimum exists. For random-feature models, a narrow bottleneck can induce "model-wise" double descent, while fully trained (NN) networks avoid this pathology by effectively re-routing information, albeit with vanishingly small feature-learning advantage (Zavatone-Veth et al., 2022).

In Bayesian optimization with neural-network-based latent representations, the capacity for representation learning is constrained by collision phenomena: pairs of inputs with different outputs may map close in latent space, reducing effective information and increasing regret. Collision-free regularization enforces a lower bound on latent distances, yielding a formal limit to achievable regret based on the mutual information between the GP and the collision-induced noise, $d$6:

$d$7

where $d$8 is the information gain and $d$9 quantifies irreducible loss from representations failing to separate targets (Zhang et al., 2022).

In position-aware transformers, the deviation from ideal Bayesian learning is formally bounded. Martingale property violations shrink as $p$0, MDL-optimality guarantees excess risk $p$1 in expectation over orderings, and the implicit posterior converges to the Beta posterior with $p$2 gap. The practical optimality of chain-of-thought reasoning length is quantified as $p$3, reflecting algorithmic consequences of the representation learning boundary (Chlon et al., 15 Jul 2025).

5. Architecture-Dependent Phase Transitions and New Representation Learning Regimes

Novel architectural modifications can shift or even invert the effect of width on representation learning. The narrow-width regime of parallel-branching networks (BPB-NN), including residual and graph-based models, exhibits a strong data-dependent feature learning phase absent in infinite-width kernels (Zhang et al., 2024). In the narrow limit ($p$4), "branch symmetry breaking" occurs, with each branch adaptively specializing according to the target function. Readout norms become reflective of data content and are essentially independent of hyperparameters. This leads to the emergent principle that, in bias-limited scenarios, narrower networks can outperform their wide (kernel) counterparts in terms of bias–variance trade-off, thus redefining feature learning limits as a function of width in complex architectures. The symmetry breaking is absent in the kernel regime.

Width regime	Feature learning strength	Key effect	Representative model
Infinite-width / GP	Vanishing ($p$5 or $p$6)	Data-independent kernel	Deep linear, NNGP, DGP
Finite-width	Small, analytic corrections	Universal $p$7 adaptation	FC/conv/Bayes NN, DGP
Branch-narrow (BPB)	Large, symmetry breaking	Data-dependent branch kernels	Graph NN, residual MLP (BPB)

6. Extensions, Open Questions, and Summary of Theoretical Limits

Representation learning limits extend to settings with heavy-tailed priors, where the infinite-width scaling yields non-Gaussian, $p$8-stable process limits. In these, the model retains non-degenerate, data-dependent kernel randomness even as width grows, with exact posterior inference attainable via normal-scale mixture representations (Loría et al., 2023). This mechanism fundamentally differentiates the representation learning limit from standard GP limits.

Multiple open questions remain: optimal selection of latent space dimension to minimize regret bounds under collision constraints, development of bi-Lipschitz embedding constraints to further tighten theoretical limits, and algorithmic frameworks that achieve oracle-optimal regret in dynamic embedding models (Zhang et al., 2022).

The Bayesian representation learning limit thus constitutes a mathematically precise boundary between regimes of non-adaptive and adaptive feature learning in Bayesian models. Its quantitative properties are now well-characterized across linear, nonlinear, GP, and transformer-based architectures, providing essential insight into both the capabilities and fundamental limitations of deep Bayesian representation learning (Yang et al., 2021, Zavatone-Veth et al., 2021, Zavatone-Veth et al., 2022, Zhang et al., 2022, Loría et al., 2023, Zhang et al., 2024, Chlon et al., 15 Jul 2025).