Fully-Learned Bayesian Estimation

• Fully-Learned Bayesian Estimation is a framework where traditional fixed Bayesian components like priors and hyperparameters are learned directly from data.
• It employs diverse methods such as score matching, simulation-trained estimators, and hierarchical inference to propagate uncertainty and improve prediction accuracy.
• Applications range from Gaussian process models to survey sampling, offering enhanced uncertainty quantification compared to plug-in approaches.

Searching arXiv for the papers on arXiv and closely related work on fully-learned Bayesian estimation. Fully-learned Bayesian estimation designates a broad class of methods in which quantities that are usually fixed, plugged in, or only partially modeled are instead learned or integrated within a Bayesian procedure. In recent work, this includes learning both the prior and the measurement distributions from data, learning direct estimators of posterior-optimal quantities from synthetic simulations, treating latent variables, hyperparameters, inducing inputs, and regularization strengths as random variables, and constructing posterior distributions directly over decision objects such as the optimal action-value function. The unifying theme is the replacement of point estimation, hand-tuned hyperparameters, or incomplete uncertainty propagation by posterior inference, posterior sampling, or learned surrogates for Bayesian quantities (Habi et al., 2 Feb 2025, Ghosh et al., 2022, Yerramilli et al., 2022, Zhou et al., 2022, Tran et al., 2023, Guo et al., 3 May 2025, Lalchand et al., 2019).

1. Conceptual scope

The contemporary literature uses several distinct constructions that fall under this umbrella. Some methods learn Bayesian objects directly from data. The learned Bayesian Cramér–Rao bound learns both the prior and the measurement distributions using score matching and neural networks, with a Posterior Approach and a Measurement-Prior Approach (Habi et al., 2 Feb 2025). DeepBayes trains recurrent neural networks offline on synthetically generated data and, in the limit of large synthetic datasets and sufficient network capacity, recovers the conditional mean estimator, which is the Bayes estimator under squared loss (Ghosh et al., 2022). In Bayesian optimization, VBO-MI replaces posterior sampling and acquisition-function inner loops with a fully gradient-based actor-critic framework that uses variational mutual information estimation (Mirkarimi, 13 Jan 2026).

Other methods are fully Bayesian in a hierarchical sense. In latent variable Gaussian process models, the latent variables for qualitative inputs and the GP hyperparameters are inferred jointly rather than estimated by maximum likelihood and plugged into prediction formulas (Yerramilli et al., 2022). Fully Bayesian Gaussian process regression similarly replaces ML-II with a posterior over hyperparameters and forms predictions by marginalizing over that posterior (Lalchand et al., 2019). ProxMCMC extends constrained and regularized estimation from fixed variance and regularization strength parameters to a hierarchical Bayesian model in which all parameters, including regularization strength, are inferred from the data (Zhou et al., 2022). In Bayesian autoencoders and shape models, decoder weights, latent variables, sparse Gaussian Process priors, and VIB parameters are all treated as random variables and estimated with scalable approximate Bayesian procedures (Tran et al., 2023, Adams et al., 2023).

A third cluster of work focuses on Bayesian inference under sampling or design complications. Fully Bayesian estimation under informative sampling jointly models the response and inclusion probability, and fully Bayesian estimation under dependent and informative cluster sampling extends this construction with cluster-indexed random effects in both the response model and the model for published unit-level sampling weights (Leon-Novelo et al., 2017, Leon-Novelo et al., 2021). These models are fully Bayesian not because they use neural networks, but because they integrate the sampling mechanism into the posterior rather than treating weights as fixed plug-in adjustments.

Modality	What is learned or integrated	Representative papers
Score-based estimation	Prior score, measurement score, posterior score	(Habi et al., 2 Feb 2025)
Closed-form Bayesian neural computation	Weight posteriors and output moments	(Huber, 2020, Yang et al., 2023)
Simulation-trained Bayes estimators	Direct mapping from observations to parameters	(Ghosh et al., 2022)
Hierarchical latent-variable Bayes	Latent variables, hyperparameters, inducing inputs	(Yerramilli et al., 2022, Lalchand et al., 2019, Tran et al., 2023)
Constrained and regularized Bayes	Regularization strength and variance parameters	(Zhou et al., 2022)
Bayesian decision objects	Posterior over $Q^*$, variational MI acquisition	(Guo et al., 3 May 2025, Mirkarimi, 13 Jan 2026)

2. Statistical constructions

A recurring construction is explicit posterior marginalization over previously fixed quantities. In the LVGP model, qualitative levels are mapped to latent variables $\mathbf{z}(t) \in \mathbb{R}^d$, and inference targets both the usual GP hyperparameters and the collection of latent variables: $$p(\boldsymbol{\theta}, \mathbf{Z} \mid \mathbf{W}, \mathbf{Y}) \propto p(\mathbf{Y} \mid \mathbf{W}, \boldsymbol{\theta}, \mathbf{Z}) \, p(\boldsymbol{\theta}) \, p(\mathbf{Z}).$$ Prediction is then obtained by integrating over $$p(\boldsymbol{\theta},\mathbf{Z}\mid \mathbf{W},\mathbf{Y})$$ rather than plugging in maximum-likelihood or MAP estimates (Yerramilli et al., 2022). Fully Bayesian Gaussian process regression uses the same logic at the hyperparameter level: $$p(\mathbf{f}^* \mid \mathbf{y}) = \int p(\mathbf{f}^* \mid \mathbf{y},\bm{\theta})\, p(\bm{\theta}\mid \mathbf{y})\, d\bm{\theta},$$ so the predictive distribution becomes a mixture of GP predictive posteriors indexed by hyperparameter draws (Lalchand et al., 2019).

A second construction learns the ingredients of Bayesian information bounds when the prior and measurement models are unknown. The LBCRB work assumes i.i.d. parameter-measurement pairs $\mathcal{D}=\{(\theta_n,\mathcal{X}_n)\}_{n=1}^N$ and learns either the posterior score $s(\theta,\mathcal{X};\Omega)$ or the prior score $s_P(\theta)$ and Fisher score $s_F(x;\theta)$. In the Measurement-Prior Approach, the Bayesian Fisher information decomposes as

$B = k \cdot M + P,$

and the learned bound is formed from empirical estimates $\mathbf{z}(t) \in \mathbb{R}^d$0, $\mathbf{z}(t) \in \mathbb{R}^d$1, and $\mathbf{z}(t) \in \mathbb{R}^d$2 (Habi et al., 2 Feb 2025). This formulation allows the number of i.i.d. measurements $\mathbf{z}(t) \in \mathbb{R}^d$3 to vary without retraining.

A third construction learns the Bayes estimator directly by simulation. DeepBayes generates synthetic data by drawing parameters from a prior, simulating output sequences, and training an LSTM or GRU estimator with mean squared error. As $\mathbf{z}(t) \in \mathbb{R}^d$4 and the network becomes sufficiently expressive, the estimator converges to

$\mathbf{z}(t) \in \mathbb{R}^d$5

the conditional mean estimator (Ghosh et al., 2022). This is distinct from explicit posterior sampling, but it still targets a Bayesian-optimal quantity.

In Bayesian neural computation, one line of work derives closed-form Bayesian updates for Gaussian weight distributions. In the Bayesian perceptron, $\mathbf{z}(t) \in \mathbb{R}^d$6, pre-activations are Gaussian, and predictive moments for sigmoid and piecewise-linear activations are computed analytically or by highly accurate analytic surrogates (Huber, 2020). The multiclass extension approximates softmax with a multivariate probit function and propagates posterior means and covariances through multilayer networks in closed form, yielding sequential Bayesian updates for the weights without Monte Carlo sampling (Yang et al., 2023).

3. Inference mechanisms and computational strategies

The inferential mechanisms used in fully-learned Bayesian estimation vary widely. For hierarchical GP models with latent variables, the No-U-Turn Sampler is used through the NumPyro implementation, producing posterior samples $\mathbf{z}(t) \in \mathbb{R}^d$7 and predictive mixtures over GP means and variances (Yerramilli et al., 2022). Approximate inference for fully Bayesian GPR compares NUTS-based HMC with variational inference using a factorized Gaussian or a full-rank Gaussian on log-hyperparameters (Lalchand et al., 2019). Bayesian density estimation via inference engines similarly relies on NUTS, expectation propagation, semiparametric mean field variational Bayes, or slice sampling after binning the data and fitting a Poisson additive mixed model (Wand et al., 2020).

When posteriors involve non-smooth penalties or constraints, smoothing is introduced deliberately. ProxMCMC replaces the non-differentiable regularization term with its Moreau-Yosida envelope,

$\mathbf{z}(t) \in \mathbb{R}^d$8

which enables gradient-based MCMC, including Langevin dynamics and Hamiltonian Monte Carlo, while also allowing priors on regularization strength and variance parameters (Zhou et al., 2022). In fully Bayesian autoencoders with latent sparse Gaussian Processes, posterior estimation is performed via stochastic gradient Hamiltonian Monte Carlo over decoder weights, latent variables, inducing inputs, inducing variables, and kernel hyperparameters, with an amortized inference network trained to mimic SGHMC samples for efficient test-time inference (Tran et al., 2023).

In score-based settings, the training criterion is itself a Bayesian object. The LBCRB Posterior Approach uses conditional score matching to approximate $\mathbf{z}(t) \in \mathbb{R}^d$9, while the Measurement-Prior Approach uses score matching for $$p(\boldsymbol{\theta}, \mathbf{Z} \mid \mathbf{W}, \mathbf{Y}) \propto p(\mathbf{Y} \mid \mathbf{W}, \boldsymbol{\theta}, \mathbf{Z}) \, p(\boldsymbol{\theta}) \, p(\mathbf{Z}).$$0 and a Fisher score matching objective for $$p(\boldsymbol{\theta}, \mathbf{Z} \mid \mathbf{W}, \mathbf{Y}) \propto p(\mathbf{Y} \mid \mathbf{W}, \boldsymbol{\theta}, \mathbf{Z}) \, p(\boldsymbol{\theta}) \, p(\mathbf{Z}).$$1. The Physics-encoded score neural network is introduced to incorporate structural knowledge such as $$p(\boldsymbol{\theta}, \mathbf{Z} \mid \mathbf{W}, \mathbf{Y}) \propto p(\mathbf{Y} \mid \mathbf{W}, \boldsymbol{\theta}, \mathbf{Z}) \, p(\boldsymbol{\theta}) \, p(\mathbf{Z}).$$2 into the Fisher score network, thereby improving sample complexity and interpretability (Habi et al., 2 Feb 2025).

Some methods avoid iterative optimization during inference altogether. The Bayesian perceptron and its multiclass generalization update posterior means and covariances analytically after each datum, using Kalman-like or moment-matching recursions (Huber, 2020, Yang et al., 2023). DeepBayes also separates training from inference: the expensive stage is completed offline, and application to new data requires only a forward pass through the trained recurrent network (Ghosh et al., 2022).

Decision-theoretic formulations add another computational pattern. In Bayesian learning of the optimal action-value function, the posterior over $$p(\boldsymbol{\theta}, \mathbf{Z} \mid \mathbf{W}, \mathbf{Y}) \propto p(\mathbf{Y} \mid \mathbf{W}, \boldsymbol{\theta}, \mathbf{Z}) \, p(\boldsymbol{\theta}) \, p(\mathbf{Z}).$$3 is sampled with an adaptive sequential Monte Carlo algorithm that traverses a sequence of relaxed posterior distributions, particularly important when the Bellman likelihood is degenerate under deterministic rewards (Guo et al., 3 May 2025). VBO-MI uses a variational critic to estimate mutual information through the Donsker–Varadhan bound and an action-net to propose queries directly, eliminating the traditional inner-loop acquisition optimization bottleneck (Mirkarimi, 13 Jan 2026).

4. Uncertainty quantification and calibration

A central motivation for fully-learned Bayesian estimation is that plug-in approaches often ignore important sources of uncertainty. In the LVGP setting, uncertainty in the latent variables can be significant especially with limited training data, and the fully Bayesian treatment is reported to offer significant improvements in prediction accuracy and uncertainty quantification over the plug-in approach. The paper states that fully Bayesian LVGP consistently outperforms plug-in MAP/ML approaches in Relative Root Mean Square Error, and yields properly calibrated uncertainty estimates with coverage close to nominal and sharper intervals than overconfident plug-in intervals (Yerramilli et al., 2022).

The same contrast appears in survey sampling. Under dependent and informative cluster sampling, the fully Bayesian method jointly models $$p(\boldsymbol{\theta}, \mathbf{Z} \mid \mathbf{W}, \mathbf{Y}) \propto p(\mathbf{Y} \mid \mathbf{W}, \boldsymbol{\theta}, \mathbf{Z}) \, p(\boldsymbol{\theta}) \, p(\mathbf{Z}).$$4 with cluster random effects in both the response and inclusion models. In simulations, “FULL.both” most closely achieves correct uncertainty quantification for model parameters, including the generating variances for cluster-indexed random effects, and maintains nominal coverage around $$p(\boldsymbol{\theta}, \mathbf{Z} \mid \mathbf{W}, \mathbf{Y}) \propto p(\mathbf{Y} \mid \mathbf{W}, \boldsymbol{\theta}, \mathbf{Z}) \, p(\boldsymbol{\theta}) \, p(\mathbf{Z}).$$5, whereas plug-in Bayesian and frequentist alternatives underestimate uncertainty (Leon-Novelo et al., 2021). Under informative sampling without explicit clustering, the fully Bayesian likelihood correction produces credible intervals with nominal coverage in synthetic experiments and avoids the need to calibrate normalization of the sampling weights (Leon-Novelo et al., 2017).

Several deep models separate aleatoric and epistemic components explicitly. Fully Bayesian VIB-DeepSSM treats both the network weights $$p(\boldsymbol{\theta}, \mathbf{Z} \mid \mathbf{W}, \mathbf{Y}) \propto p(\mathbf{Y} \mid \mathbf{W}, \boldsymbol{\theta}, \mathbf{Z}) \, p(\boldsymbol{\theta}) \, p(\mathbf{Z}).$$6 and the latent representation $$p(\boldsymbol{\theta}, \mathbf{Z} \mid \mathbf{W}, \mathbf{Y}) \propto p(\mathbf{Y} \mid \mathbf{W}, \boldsymbol{\theta}, \mathbf{Z}) \, p(\boldsymbol{\theta}) \, p(\mathbf{Z}).$$7 as random variables, and estimates total predictive uncertainty as the sum of a sample variance term and a mean predicted variance term: $$p(\boldsymbol{\theta}, \mathbf{Z} \mid \mathbf{W}, \mathbf{Y}) \propto p(\mathbf{Y} \mid \mathbf{W}, \boldsymbol{\theta}, \mathbf{Z}) \, p(\boldsymbol{\theta}) \, p(\mathbf{Z}).$$8 Experiments on synthetic shapes and left atrium data show improved uncertainty reasoning without sacrificing accuracy, and the combined Dropout + Ensembling models have the highest Pearson $$p(\boldsymbol{\theta}, \mathbf{Z} \mid \mathbf{W}, \mathbf{Y}) \propto p(\mathbf{Y} \mid \mathbf{W}, \boldsymbol{\theta}, \mathbf{Z}) \, p(\boldsymbol{\theta}) \, p(\mathbf{Z}).$$9 between uncertainty and error (Adams et al., 2023).

Other fully Bayesian estimators use posterior distributions to attach uncertainty to functionals rather than parameters. Bayesian density estimation via inference engines produces pointwise credible intervals and, in simulation studies, the pointwise $$p(\boldsymbol{\theta},\mathbf{Z}\mid \mathbf{W},\mathbf{Y})$$0 credible intervals are described as “honest,” with actual coverages typically around $$p(\boldsymbol{\theta},\mathbf{Z}\mid \mathbf{W},\mathbf{Y})$$1 to $$p(\boldsymbol{\theta},\mathbf{Z}\mid \mathbf{W},\mathbf{Y})$$2 (Wand et al., 2020). Fully Bayesian estimation of probabilistic sensitivity measures uses posterior draws of sensitivity indices obtained either from partition-based Dirichlet process constructions or partition-free Bayesian nonparametric density estimators, thereby quantifying uncertainty without requiring additional simulator evaluations (Antoniano-Villalobos et al., 2019).

This body of work also clarifies a common misconception. Previous studies on standard GP modeling had largely concluded that a fully Bayesian treatment offers limited improvements, but the LVGP results show that this conclusion is not uniform across model classes (Yerramilli et al., 2022). A plausible implication is that the magnitude of the gain depends on where the dominant uncertainty resides: in hyperparameters alone, in latent embeddings for qualitative levels, in inclusion probabilities under informative sampling, or in epistemic uncertainty over deep network weights.

5. Representative application domains

The applications are unusually diverse. In signal processing, the learned BCRB is demonstrated on a linear measurement problem with unknown mixing and Gaussian noise covariance matrices, frequency estimation, quantized measurement, and a nonlinear frequency-estimation problem with real-world underwater ambient noise (Habi et al., 2 Feb 2025). DeepBayes addresses stochastic nonlinear dynamical models and is evaluated on linear and nonlinear state space models and on a real-world electric drive system (Ghosh et al., 2022).

Engineering and materials design are a major setting for fully Bayesian latent-variable GPs. The LVGP work studies borehole, piston, and OTL circuit examples with discretized numerical variables, as well as materials tasks including lacunar spinels, $$p(\boldsymbol{\theta},\mathbf{Z}\mid \mathbf{W},\mathbf{Y})$$3 phases, and a large-scale $$p(\boldsymbol{\theta},\mathbf{Z}\mid \mathbf{W},\mathbf{Y})$$4 perovskite database (Yerramilli et al., 2022). These are prototypical problems with qualitative inputs, many levels, and sparse observations per level.

Medical imaging and representation learning provide a different set of use cases. Fully Bayesian VIB-DeepSSM predicts correspondence-based statistical shape models of anatomy from unsegmented 3D images, with experiments on synthetic supershapes and left atrium data (Adams et al., 2023). Fully Bayesian autoencoders with latent sparse Gaussian Processes handle synthetic moving ball trajectories, Rotated MNIST, EEG, and Jura missing-data imputation, where correlations among latent codes are modeled through sparse GP priors rather than i.i.d. latent assumptions (Tran et al., 2023).

Regularized and constrained estimation extends the scope to classical statistical learning problems. Fully Bayesian ProxMCMC is illustrated on lasso regression, constrained lasso for simulated microbiome data, graphical lasso for cytometry data, matrix completion, and sparse low-rank matrix regression (Zhou et al., 2022). Bayesian density estimation via inference engines reformulates density estimation as Poisson nonparametric regression on binned counts, allowing large-sample scalability while remaining fully Bayesian (Wand et al., 2020). Fully Bayesian partially linear wavelet models address simultaneous estimation and variable selection in parametric and nonparametric components (Remenyi, 2016).

Sequential decision-making provides yet another interpretation. Bayesian learning of $$p(\boldsymbol{\theta},\mathbf{Z}\mid \mathbf{W},\mathbf{Y})$$5 in an MDP introduces a likelihood based on Bellman’s optimality equations and evaluates posterior-sampling decision rules on the Deep Sea benchmark, emphasizing exploration benefits of posterior sampling in MDPs (Guo et al., 3 May 2025). VBO-MI tackles high-dimensional synthetic functions, PDE optimization, Lunar Lander control, and categorical Pest Control, reporting up to a $$p(\boldsymbol{\theta},\mathbf{Z}\mid \mathbf{W},\mathbf{Y})$$6 reduction in FLOPs compared to BNN-BO baselines (Mirkarimi, 13 Jan 2026).

6. Relations to plug-in, empirical Bayes, and approximate Bayes

The sharpest contrast in this literature is with plug-in and empirical Bayes procedures. ML-II in Gaussian process regression fixes hyperparameters at a maximizer of the marginal likelihood, whereas fully Bayesian GPR integrates over $$p(\boldsymbol{\theta},\mathbf{Z}\mid \mathbf{W},\mathbf{Y})$$7 and forms a predictive mixture over hyperparameter samples (Lalchand et al., 2019). The classical LVGP procedure estimates latent variables similarly to other GP hyperparameters through maximum likelihood estimation and plugs them into the prediction expressions, while the fully Bayesian alternative propagates their posterior uncertainty (Yerramilli et al., 2022). In informative sampling, the pseudo-posterior raises likelihood terms to the power of sampling weights, while the fully Bayesian approach jointly models the response and inclusion mechanism and thereby smooths the weights (Leon-Novelo et al., 2017).

At the same time, “fully Bayesian” does not imply that every method is exact in the same sense. The Bayesian perceptron is closed-form for Gaussian weights with analytic moment propagation, but uses a scaled probit approximation for the sigmoid case (Huber, 2020). The multiclass extension provides closed-form Bayesian updates using a multivariate probit approximation to the softmax output (Yang et al., 2023). ProxMCMC relies on Moreau-Yosida smoothing, BVIB-DeepSSM uses concrete dropout and batch ensemble as scalable approximate Bayesian implementations, and fully Bayesian GPR may rely on variational approximations rather than HMC (Zhou et al., 2022, Adams et al., 2023, Lalchand et al., 2019). In the MDP setting, deterministic rewards induce a degenerate likelihood, so artificial observation noise is introduced in a controlled manner to facilitate more efficient Monte Carlo-based inference (Guo et al., 3 May 2025).

The trade-off between flexibility and dependence on modeling choices remains explicit. DeepBayes is asymptotically as good as the Bayes estimator, but the quality of the learned estimator hinges on the relevance of the prior and the richness of the simulated data, and the trained estimator generalizes only to the specific model structure and fixed input signals used during training (Ghosh et al., 2022). Fully Bayesian survey methods require explicit models for inclusion probabilities or published weights (Leon-Novelo et al., 2017, Leon-Novelo et al., 2021). Score-based LBCRB methods provide strong consistency and theoretical upper bounds on relative error, but the Posterior Approach can be difficult when the dimension of $$p(\boldsymbol{\theta},\mathbf{Z}\mid \mathbf{W},\mathbf{Y})$$8 is large, whereas the Measurement-Prior Approach is designed to be more sample efficient and more amenable to domain knowledge through the Physics-encoded score neural network (Habi et al., 2 Feb 2025).

Taken together, these results suggest that fully-learned Bayesian estimation is not a single algorithmic family but a recurrent design principle. It appears whenever posterior uncertainty over quantities that materially affect inference or decisions—latent variables, hyperparameters, priors, measurement models, regularization levels, sampling mechanisms, or value functions—is treated as part of the estimation problem rather than as a preprocessing step or a plug-in correction.