Bayesian Effective Dimension in Inference

• Bayesian effective dimension is a measure that quantifies the intrinsic directions in parameter space where statistical learning and posterior contraction occur.
• It employs information-theoretic metrics, spectral diagnostics, and gradient-based methods to assess and reduce model complexity.
• This concept is crucial for applications in deep learning, inverse problems, and cosmological inference by enhancing uncertainty quantification and computational scalability.

The Bayesian effective dimension quantifies the intrinsic or learnable dimensionality in a given Bayesian inference problem, encapsulating the number of directions in parameter space where posterior contraction or statistical learning occurs, as opposed to the ambient parameter count. This concept has emerged across various domains—including linear and nonlinear inverse problems, probabilistic principal component analysis, deep neural network generalization, latent-variable graphical models, nonparametric density and subspace estimation, and cosmological parameter inference—each employing problem-specific definitions and computational strategies but consistently relying on information-theoretic, spectral, or identifiability-based arguments.

1. Foundational Definitions and Information-Theoretic Formulations

The contemporary formulation of Bayesian effective dimension is grounded in expected information gain between prior and posterior distributions. In “Bayesian Effective Dimension: A Mutual Information Perspective” (Banerjee, 28 Dec 2025), the effective dimension is defined as

$$d_\text{eff}(n) := \frac{2\,I(\Theta; X^{(n)})}{\log n}$$

where $I(\Theta; X^{(n)})$ is the mutual information between parameters and data. For regular parametric models, this coincides asymptotically with the parameter dimension. In high-dimensional, ill-posed, or regularized regimes, $d_\text{eff}(n)$ may be dramatically smaller, providing a coordinate-free and prior-dependent measure of how many directions are identifiable at a given sample size.

A related measure, Bayesian model dimensionality (BMD), is defined as the variance of Shannon information (surprisal) under the posterior:

$$\tilde{d} = \operatorname{Var}_P[\mathcal{I}] = \langle \mathcal{I}^2 \rangle - \langle \mathcal{I} \rangle^2$$

where $$\mathcal{I}(\theta) = \log \left[ P(\theta|D) / \pi(\theta) \right]$$ (Handley et al., 2019). In multivariate Gaussian inference, $I(\Theta; X^{(n)})$ and $\tilde{d}$ exactly recover $d$.

Both mutual information and BMD exhibit invariance under reparameterization, additivity over independent subblocks, and direct sample-based computation through MCMC or nested sampling algorithms (Handley et al., 2019).

2. Spectral Criteria and Gradient-Based Dimension Reduction

A large family of Bayesian effective dimension estimators exploits the spectrum of a diagnostic matrix: typically the Fisher information matrix, the gradient covariance of the log-likelihood, or the expected Hessian over the prior or posterior (Banerjee, 28 Dec 2025, Cui et al., 2021, Ehre et al., 2022, Lan, 2018, König et al., 30 Jun 2025, Baptista et al., 2024). Consider the generalized eigenproblem:

$E[H(Y)] v_i = \lambda_i\,\Gamma\,v_i$

where $E[H(Y)]$ is the prior- or posterior-averaged Fisher information, and $\Gamma$ a prior-derived weight matrix (Cui et al., 2021). The dominant eigenvectors $\{ v_1, \ldots, v_r \}$ span the likelihood-informed or "active" subspace. The smallest $r$ such that $\sum_{i>r} \lambda_i$ falls below a prescribed KL-tolerance $\epsilon$ sets the Bayesian effective dimension.

In linear Gaussian inverse problems with prior covariance $\Gamma_0$ and data-misfit Hessian $H = G^\top \Gamma_{\text{obs}}^{-1} G$, the effective dimension is (König et al., 30 Jun 2025):

$$d_{\text{eff}} = \operatorname{tr}\left( H(\Gamma_0^{-1} + H)^{-1} \right) = \sum_{i=1}^d \frac{\lambda_i}{1 + \lambda_i}$$

where $\lambda_i$ are the generalized eigenvalues of $(H, \Gamma_0^{-1})$. This dimension sets the minimal subspace in which posterior contraction occurs and controls optimal dimension-reduced posterior approximations.

Gradient-based estimates are extended to simulation-based or data-driven scenarios through score-ratio matching and score-based networks (Baptista et al., 2024), yielding analogous eigenvalue certificate bounds for subspace truncation.

3. Model Selection, Identifiability, and Hierarchical Structure

In latent-variable models and Bayesian networks, the effective dimension addresses parameter nonidentifiability and singularity. Allowing $\theta$ to denote all standard model parameters, one defines (Kocka et al., 2011):

$$d_e(M) = \operatorname{rank}_\text{regular} J_M(\theta)$$

where $J_M(\theta)$ is the Jacobian from parameter space to the observable distribution. In tree-structured graphical models (HLCs), recursive decomposition yields (Kocka et al., 2011):

$d_e(M) = d_e(M_1) + d_e(M_2) - k_0,$

with $k_0$ the overlap in parameterizations. Effective dimension serves as the penalty in Bayesian Information Criterion variants (BIC$_e$), often outperforming standard dimension-based criteria in structure learning when latent redundancy is present.

Nonparametric Bayesian subspace estimation models place a prior directly on subspace dimension $k$ (Bhattacharya et al., 2011). The posterior on $k$ quantifies uncertainty regarding the true effective dimension and provides identifiability guarantees under mild regularity assumptions.

4. Computation, Error Bounds, and Reduction Algorithms

Determination of Bayesian effective dimension in practice involves:

• Spectral decomposition (Lanczos, Krylov) of Fisher/gradient/Hessian matrices for leading eigenvalues and vectors (König et al., 30 Jun 2025, Cui et al., 2021).
• Empirical subspace learning through score-matching networks or MAVE estimators (Baptista et al., 2024, Hu et al., 2024).
• Marginal likelihood curve analysis in probabilistic PCA and model selection, e.g., maximizing the discrete second derivative at the peak for normal-gamma PPCA (Bouveyron et al., 2017).
• Posterior contraction and trace/covariance comparisons between prior and posterior Gaussians yields $N_{\text{eff}}$ in deep learning (Maddox et al., 2020).

Theoretical guarantees, typically in terms of KL-divergence, are explicit: dimension reduction incurs a maximum error of $\frac{\kappa}{2}\sum_{i>r}\lambda_i$ in KL if only the first $r$ eigen-directions are retained, with $\kappa$ a log-Sobolev constant (Cui et al., 2021, Ehre et al., 2022, Baptista et al., 2024). In variance-minimization or F\"orstner distance, the dimension-reduced posterior is the unique minimizer among all rank-$r$ approximations (König et al., 30 Jun 2025).

5. Non-Asymptotic and High-Dimensional Behaviors

Non-asymptotic analysis reveals that Bayesian effective dimension can be dynamically small for finite data or low signal-to-noise ratios, enabling adaptive model selection and uncertainty quantification well below ambient parameter counts (Bouveyron et al., 2017, Banerjee, 28 Dec 2025). In deep networks, $N_{\text{eff}}$ mirrors generalization error and double descent curves, in contrast to parameter count (Maddox et al., 2020). In inverse problems, most posterior uncertainty concentrates in a handful of dominant modes even at high $d$ (König et al., 30 Jun 2025, Lan, 2018, Ehre et al., 2022).

In infinite-dimensional white-noise models, the local effective dimension is tied to the minimizer (oracle index) of approximating truncation error versus noise variance (Belitser, 2024). Minimax impossibility results constrain uniform two-sided inference but allow for sharp concentration under head/tail regularity assumptions on the signal.

6. Practical Applications and Model-Specific Adaptations

Bayesian effective dimension critically enables efficient posterior computation, adaptive model reduction, and scalable sampling in high-dimensional settings:

• Dimension and model reduction in linear dynamical systems and PDE-constrained problems (1711.02475, König et al., 30 Jun 2025).
• Certified dimension reduction for importance sampling and cross-entropy methods in Bayesian updating (Ehre et al., 2022).
• Score-ratio and gradient-free approaches in simulation-based inference and generative modeling, providing spectral certificates of truncation (Baptista et al., 2024).
• Bayesian optimization in high-$D$ using effective dimension reduction subspaces (EDR), e.g., via MAVE estimators and subspace-adaptive GP models (Hu et al., 2024).

In cosmology, BMD (\texttildelow Var$[\mathcal{I}]$) measures dimensionality of constraint across competing probes and tension metrics (Handley et al., 2019). In random media, sparse encoders and ARD priors automatically extract the set of truly predictive features, quantifying effective latent dimension (1711.02475).

7. Connections to Regularization, Shrinkage, and Approximate Methods

Regularization and shrinkage mechanisms modulate effective dimension by explicit reduction of learnable directions. Prior constraints, penalty terms, or global-local shrinkage priors (horseshoe, Gaussian mixtures) randomize which directions are learned, maintaining a finite effective dimension even in otherwise ill-posed or infinite-dimensional problems (Banerjee, 28 Dec 2025). Approximate posteriors that inflate covariance yield lower mutual information and truncate $d_{\text{eff}}$ (Banerjee, 28 Dec 2025, Maddox et al., 2020).

In summary, Bayesian effective dimension grounds model complexity, uncertainty quantification, and computational scalability in an analytic framework, encompassing mutual information, spectral diagnostics, and identifiability under prior and data structure. Its rigorous definitions, computational protocols, and error controls are central to modern Bayesian analysis in high-dimensional statistical inference across applied and theoretical domains.