Estimating the Bayesian effective dimension from data

Develop data-driven estimators or computable bounds for the Bayesian effective dimension d_eff(n) = 2 I(Θ; X^{(n)}) / log n, where I(Θ; X^{(n)}) denotes the mutual information between parameters Θ and data X^{(n)} under a specified prior Π and sampling model {P_θ^{(n)}}, and ascertain their finite-sample properties, including cases such as Gaussian linear regression where plug-in estimators of log det(Σ_0^{-1} Σ) are employed.

Background

The paper defines the Bayesian effective dimension d_eff(n) via mutual information as a coordinate-free measure of how many directions in parameter space are statistically learnable at a given sample size. While it is defined in terms of the joint prior–model distribution, practitioners need methods to estimate or bound this quantity from observed data.

The authors highlight that even in Gaussian settings where analytical forms exist for mutual information, practical estimation (e.g., via plug-in or empirical log-determinant approximations) lacks established finite-sample guarantees, motivating a concrete estimation problem and associated theoretical analysis.