Bayesian Effective Dimension: A Mutual Information Perspective (2512.23047v1)

Abstract: High-dimensional Bayesian procedures often exhibit behavior that is effectively low dimensional, even when the ambient parameter space is large or infinite-dimensional. This phenomenon underlies the success of shrinkage priors, regularization, and approximate Bayesian methods, yet it is typically described only informally through notions such as sparsity, intrinsic dimension, or degrees of freedom. In this paper we introduce the \emph{Bayesian effective dimension}, a model- and prior-dependent quantity defined through the mutual information between parameters and data. This notion quantifies the expected information gain from prior to posterior and provides a coordinate-free measure of how many directions in parameter space are statistically learnable at a given sample size. In regular parametric models the effective dimension coincides with the usual parameter dimension, while in high-dimensional, ill-posed, or strongly regularized settings it can be substantially smaller. We develop basic properties of the effective dimension and present explicit calculations for Gaussian location models and linear models with general design, revealing close connections with spectral complexity and effective rank. These examples illustrate how shrinkage and regularization mechanisms directly control the growth of effective dimension. The framework offers a unifying perspective on dimension reduction in Bayesian inference and provides insight into uncertainty quantification and the behavior of approximate posterior distributions.

• The paper introduces a novel mutual information-based measure of Bayesian effective dimension that quantifies model complexity in high-dimensional contexts.
• It employs explicit calculations in Gaussian models to demonstrate how shrinkage and regularization impact the number of learnable parameter directions.
• Implications include enhanced diagnostics for overparameterization, improved uncertainty quantification, and refined criteria for model selection.

Bayesian Effective Dimension from a Mutual Information Perspective

Introduction and Motivation

"Bayesian Effective Dimension: A Mutual Information Perspective" (2512.23047) presents a unified framework for quantifying intrinsic dimensionality in Bayesian inference, emphasizing global and coordinate-free characterization rather than model-specific or estimator-dependent measures. The central motivation is to formalize the widely observed phenomenon that high-dimensional Bayesian models, often coupled with shrinkage or regularization priors, effectively behave in a low-dimensional manner even as the nominal parameter dimension is large or infinite. Classical notions—degrees of freedom, metric entropy, and effective rank—fall short of capturing the Bayesian information-theoretic structure, particularly across a range of priors and in the presence of regularization or model misspecification.

The paper advances the notion of Bayesian effective dimension, defined through the mutual information between the parameters and the observed data, at fixed sample size, with respect to the prior and the likelihood specified by the modeling assumptions. This measure serves as a coordinate-free quantification of the number of learnable, statistically distinguishable directions in parameter space—effectively, a loss-agnostic intrinsic complexity metric for Bayesian learning.

Definition and Formal Properties

The mutual information between parameters $\Theta$ and data $X^{(n)}$ is given by

$$I(\Theta; X^{(n)}) = E\left[ \log \frac{p(X^{(n)} \mid \Theta)}{p(X^{(n)})} \right] = E\left[ KL(\Pi(\cdot|X^{(n)}) \,\|\, \Pi) \right],$$

representing the expected Kullback–Leibler divergence from prior to posterior. This quantity is always nonnegative, zero only in cases of no learning, and increases monotonically with information content.

The Bayesian effective dimension at sample size $n$ is defined as

$$d_{\mathrm{eff}}(n) := \frac{2I(\Theta; X^{(n)})}{\log n}.$$

This normalization, motivated by the parametric scaling $I(\Theta; X^{(n)}) = \frac{d}{2} \log n + O(1)$, ensures that effective dimension asymptotically coincides with parameter dimension for regular models, but may differ dramatically in high-dimensional, ill-posed, or strongly regularized regimes.

Key properties include:

• Invariance: $d_{\mathrm{eff}}(n)$ is invariant under one-to-one reparameterizations of $\Theta$.
• Monotonicity: $d_{\mathrm{eff}}(n)$ is non-increasing under coarsening or summarization of data.
• Loss-independence: It is independent of specific estimators or loss functions, depending purely on the joint prior–model structure.

Explicit Calculation in Gaussian Models

Explicit non-asymptotic calculations are provided for conjugate Gaussian models, elucidating dependence on spectral properties of the design and the shrinkage regime.

For the multivariate location model,

$$\theta \sim N(0, \tau^2 I_d), \quad X_i \sim N_d(\theta, \sigma^2 I_d),$$

the effective dimension simplifies to:

$$d_{\mathrm{eff}}(n) = d \cdot \frac{\log (1 + n\tau^2/\sigma^2)}{\log n},$$

which converges to $d$ as $n \rightarrow \infty$ for fixed $\tau^2, \sigma^2$, but can be arbitrarily smaller for strong shrinkage ($\tau^2 \ll 1$).

In linear regression with Gaussian prior,

$$Y \sim N(X\beta, \sigma^2 I_n), \quad \beta \sim N(0, \tau^2 I_p),$$

the mutual information and effective dimension are governed by the spectrum of $X^\top X$. Specifically,

$$I(\beta; Y) = \frac{1}{2}\sum_{j=1}^r \log(1 + \frac{\tau^2}{\sigma^2} s_j^2),$$

where $\{s_j\}$ are the singular values of $X$, and $r = \text{rank}(X)$. Therefore, ill-conditioning or strong regularization directly limits effective dimension, potentially keeping it bounded even as $p \to \infty$.

The paper distinguishes between effective dimension and degrees of freedom from frequentist smoothing theory. Whereas the trace-based degrees of freedom may saturate for high signal-to-noise components, effective dimension grows only logarithmically, enforcing strong penalization for modes that are already highly identifiable.

Interpretation and Implications

Theoretical Implications

• Multiplicity Reduction: Effective dimension captures the reduction in uncertainty attributable to observed data, incorporating global aspects such as prior structure, regularization, and information bottlenecks.
• Infinite-dimensional Regimes: For infinite-dimensional problems (e.g., under decaying singular value spectra), effective dimension can remain bounded if $\sum s_j^2 < \infty$, emphasizing that under strong ill-posedness or sufficiently informative prior, global learning saturates and model complexity does not grow with ambient dimension.
• Misspecification Sensitivity: In misspecified (non-Gaussian or otherwise incorrect) models, effective dimension reflectively adapts, quantifying learnability in only those directions where the model is locally identifiable.

Practical Implications

• Diagnostic of Overparameterization: Small effective dimension, in contrast with large nominal parameter count, provides direct evidence of a low degree of meaningful Bayesian learning, suggesting irrelevance of most parameters.
• Comparing Exact vs. Approximate Inference: Approximate posteriors that inflate the covariance structure necessarily imply lower information gain and, hence, lower effective dimension—offering a quantifiable diagnostic for information loss due to variational or other approximations.
• Shrinkage Priors: For global-local priors (e.g., horseshoe, Student-t), the effective dimension is random and data-dependent. Marginalization over latent variances produces adaptivity, concentrating mutual information in a subset of coordinates and inducing a data-driven, random effective dimensionality without imposing hard sparsity.

Effective Dimension and Regularization

Regularization mechanisms are shown to control effective dimension by directly suppressing unstable or data-insensitive directions. In spectral terms, they truncate or downweight contributions from modes with low signal-to-noise, manifest in the mutual information expression through saturation effects in the log-transform, especially for global-local priors.

For approximate Bayesian inference, any covariance inflation relative to the exact posterior (as in mean-field variational inference or Laplace methods) strictly reduces mutual information and effective dimension, elucidating why conservative uncertainty quantification is frequently observed—approximate posteriors fundamentally learn fewer directions.

Open Problems and Future Directions

Several key theoretical and practical directions are highlighted:

• Estimation: Developing empirical or simulation-based estimators for effective dimension in complex models.
• Contraction Rates and Uncertainty Quantification: Relating effective dimension to posterior contraction rates and quantification of uncertainty, particularly in nonparametric and misspecified models.
• Extension to Non-Gaussian, Non-convex Problems: Analysis of effective dimension under non-Gaussian, nonlinear, or heavily misspecified regimes.
• Model Selection and Approximation Diagnostics: Systematics for interpreting or optimizing effective dimension in model selection, approximate inference, and workflow design.
• Random Effective Dimension under Shrinkage: Characterizing the distributional properties of random effective dimension in models with heavy-tailed or adaptive (e.g., horseshoe-type) shrinkage priors.

Conclusion

This work establishes Bayesian effective dimension via mutual information as a foundational, model-intrinsic complexity measure for Bayesian inference. The analysis elucidates both theoretical and practical aspects of effective dimension, connecting information-theoretic and spectral regularization perspectives and providing operational tools for quantification of learnable model complexity. The resulting framework yields insights into posterior contraction, uncertainty quantification, regularization, and the comparative effects of exact and approximate Bayesian inference. Open problems include empirical estimation, analysis in non-Gaussian and misspecified settings, and further specification of the interplay between shrinkage structure and adaptive dimensionality.