Bayesian Approximation Error (BAE)

• BAE is a statistical framework that models surrogate errors as random variables to correct bias and recover true posterior uncertainty.
• It empirically estimates error statistics through offline and adaptive sampling, enhancing the fidelity of coarse or surrogate models in complex inverse problems.
• By re-centering the misfit and inflating variance in likelihoods, BAE improves confidence in surrogate-based inference while reducing computational cost.

The Bayesian Approximation Error (BAE) methodology is a statistical framework for quantifying and propagating errors that result from substituting an expensive or high-fidelity forward model with a computationally cheaper but approximate surrogate in Bayesian inverse problems and uncertainty quantification. Rather than neglecting model or emulator errors, BAE systematically models the approximation error as a random variable, estimates its distribution empirically, and rigorously corrects the statistical inference to account for the resulting bias and loss of coverage. This approach underlies a broad range of techniques in PDE-constrained inversion, surrogate-based inference, optimal experimental design, and neural network emulation, providing both feasibility and increased interpretability for high-dimensional and computationally intensive scientific problems.

1. Formulation and Statistical Structure

The BAE framework starts from a hierarchical Bayesian formulation in which the observed data $y \in \mathbb{R}^p$ are related to unknown parameters $\theta \in \mathbb{R}^n$ and a latent process $y_p = f(\theta)$ via measurement noise $\delta \sim \mathcal{N}(0, \Gamma)$, with prior $\pi(\theta)$. The measurement model is:

$$y = f(\theta) + \delta\,, \quad \delta \sim \mathcal{N}(0, \Gamma)$$

where $f(\theta)$ is the "fine" or "high-fidelity" forward model. To accelerate inference, this is replaced by a surrogate $g(\theta)$, introducing a model discrepancy:

$\varepsilon(\theta) = f(\theta) - g(\theta)$

In the augmented statistical model, the surrogate error $\varepsilon(\theta)$ is treated as a random variable, generally approximated (in the basic form) as $$\varepsilon \sim \mathcal{N}(\mu_\varepsilon, \Sigma_\varepsilon)$$, possibly estimated from empirical samples. The total error is the sum of surrogate and measurement errors: $\nu = \varepsilon + \delta$, leading to a joint likelihood:

$$p(y \mid \theta) = \mathcal{N}(y; g(\theta) + \mu_\varepsilon,\; \Gamma + \Sigma_\varepsilon)$$

This formalism generalizes to scenarios with nuisance or secondary parameters, black-box forward maps, and also underpins corrective treatment for neural-network-based emulators (Maclaren et al., 2018, Grandón et al., 2022, Nicholson et al., 5 Dec 2025, Koval et al., 2024).

2. Construction and Estimation of Approximation Error Distributions

The crucial step in BAE is the estimation of the mean and covariance of the approximation error $\varepsilon$:

• Offline estimation involves drawing samples $\theta^{(\ell)}$ (either from the prior, marginal, or a naive posterior), computing both $f(\theta^{(\ell)})$ and $g(\theta^{(\ell)})$, and calculating

$$\mu_\varepsilon \approx \frac{1}{q} \sum_{\ell=1}^q \varepsilon^{(\ell)},\qquad \Sigma_\varepsilon \approx \frac{1}{q-1} \sum_{\ell=1}^q (\varepsilon^{(\ell)} - \mu_\varepsilon)(\varepsilon^{(\ell)} - \mu_\varepsilon)^T$$

• Posterior-informed error sampling improves estimation by focusing sampling on regions of highest posterior density under the naive surrogate, especially when the surrogate is accurate far from the bulk of the prior (Maclaren et al., 2018).
• Adaptive refinement iteratively adds new training samples in parameter regions of high posterior probability, which allows surrogate error to be adaptively reduced over multiple rounds (Zhang et al., 2018).
• Taylor approximation variance reduction leverages Taylor expansions (first or second order) of both $f$ and $g$ to provide low-variance control variates for Monte Carlo estimation of error statistics in very high-dimensional or PDE-governed settings (Nicholson et al., 5 Dec 2025).

3. Impact on Likelihoods, Posterior Distributions, and Uncertainty Quantification

The effect of the BAE correction is two-fold:

• Shift in the mean: The data-model misfit is re-centered by the estimated $\mu_\varepsilon$, correcting for systematic bias in the surrogate.
• Inflation of variance: The uncertainty is inflated by $\Sigma_\varepsilon$, avoiding the underestimation of uncertainty and producing credible intervals that recover appropriate frequentist coverage.

The resulting "BAE-corrected" posterior has the form:

$$\pi(\theta|y) \propto \pi(\theta)\,\mathcal{N}(y; g(\theta) + \mu_\varepsilon,\,\Gamma + \Sigma_\varepsilon)$$

This structure allows the use of standard inference algorithms (MCMC, optimization, variational methods) with minimal or no modification to the computational pipeline (Maclaren et al., 2018, Grandón et al., 2022, Nicholson et al., 2018, Koval et al., 2024).

Empirical and theoretical results demonstrate that:

• Naive surrogate-based inference yields biased and overconfident posteriors whose credible intervals often exclude the true parameter (Maclaren et al., 2018, Grandón et al., 2022, Zhang et al., 2018).
• BAE correction restores the accuracy and reliability of uncertainty quantification and aligns approximate inference with the behavior of the full-model posterior, often at vastly reduced computational cost.

4. Methodological Variants and Applications

The generic structure of BAE admits several methodological variants tailored to specific applications:

Variant	Surrogate	Error Modeling	Key Use Case
Standard BAE	Coarse model	Gaussian $\varepsilon$	PDE-constrained inversion, MCMC acceleration
Posterior-informed	Coarse model	Conditioned on naive posterior	Geothermal and hydrological inference
Surrogate BAE	Emulator (NN, PCE, GP)	Empirical error from validation/test set	Cosmology, surrogate-based UQ
Error-corrected	PCE+GP, GP+PCE	Secondary surrogate for residual	High-dimensional, multimodal posteriors
Taylor-variance	Linear/Quadratic approximations	Control variate corrections	Large-scale PDEs, variance reduction

Applications include Bayesian inverse modeling in geothermal and hydrological systems, estimation of boundary conditions and parameters in PDEs with model uncertainty, cosmological parameter inference with neural nets, optimal design of experiments in high-dimensional settings, and linearized or surrogate-based infinite-dimensional Bayesian inversion (Maclaren et al., 2018, Zhang et al., 2018, Grandón et al., 2022, Nicholson et al., 5 Dec 2025, Alexanderian et al., 2022, Koval et al., 2024).

5. Error Bounds, Theoretical Properties, and Limitations

Recent work has established non-asymptotic and asymptotic bounds on the effect of surrogate/model error in Bayesian inference:

• Hellinger and Wasserstein metrics quantify the distance between exact and approximate posteriors, with explicit bounds in terms of the $L^p$-norms of the surrogate/model errors and integrability assumptions (Lie et al., 2019, Huggins et al., 2018).
• Strong convexity and smoothness assumptions enable tractable control of the posterior approximation error using gradients (Fisher distance), facilitating practical diagnostics and control of error in Laplace approximation, variational Bayes, and Hilbert coreset settings (Huggins et al., 2018).
• Algebraic-geometric analysis in singular or non-regular models, such as non-negative matrix factorization, shows that the separation in negative log-evidence (free energy) between variational and fully Bayesian posteriors scales as the gap between their respective real log canonical thresholds (RLCT), with explicit lower bounds on BAE as a function of model rank and hyperparameters (Hayashi, 2018).
• Corollaries guarantee that with sufficient sampling, the bias and loss of coverage induced by surrogates can be made arbitrarily small, but in practice, the Gaussianity and independence assumptions on $\varepsilon$ are limiting in extreme regimes (Lie et al., 2019, Maclaren et al., 2018).

6. Integration with Optimal Experimental Design (OED)

The BAE methodology has become integral to modern OED for PDE-based inverse problems and sensor placement:

• The inclusion of the BAE-corrected error covariance in the design criterion (e.g., A-optimality) ensures that designs do not exploit spurious confidence in an inaccurate model and are robust to model uncertainty (Alexanderian et al., 2022).
• Recent advances prove that, under the BAE correction, the design objective is independent of the choice of surrogate or linearization, permitting sensor selection using only sample-based error statistics and a trivial surrogate (the zero map), which is practical for black-box forward models (Koval et al., 2024).
• Marginal designs accounting for both primary and nuisance parameters can be constructed, again using the BAE-corrected moments, to yield designs that maintain feasibility and accuracy across model misspecification regimes.

7. Practical Implementation Considerations

• Offline computational cost: The dominant cost lies in evaluating the high-fidelity model at sufficiently many sample points to estimate $\mu_\varepsilon$ and $\Sigma_\varepsilon$. Techniques such as Taylor expansion control variates, local corrections, adaptive sampling, and low-rank matrix approximations dramatically reduce sample requirements in high-dimensional settings (Nicholson et al., 5 Dec 2025).
• Online inference: Once error statistics are estimated, BAE-corrected inference incurs minimal additional computational cost over standard surrogate-based inference.
• Surrogate selection: The quality of the BAE correction depends strongly on the global fidelity of the surrogate. If the surrogate fails to reproduce core physical features, the correction may be too pessimistic or unreliable.
• Diagnostics: Feasibility and coverage of credible intervals should always be assessed by validation against ground truth or additional high-fidelity samples from the posterior region (Maclaren et al., 2018, Zhang et al., 2018).
• Limitations: The basic BAE approach assumes approximately Gaussian surrogate errors, stationarity, and (for computational simplicity) independence from parameters. Substantial non-Gaussian or strongly parameter-dependent errors require methodological extensions, such as nonparametric error models or local Gaussianizations (Maclaren et al., 2018, Nicholson et al., 5 Dec 2025).

• (Maclaren et al., 2018) Incorporating Posterior-Informed Approximation Errors into a Hierarchical Framework to Facilitate Out-of-the-Box MCMC Sampling for Geothermal Inverse Problems and Uncertainty Quantification
• (Grandón et al., 2022) Bayesian error propagation for neural-net based parameter inference
• (Nicholson et al., 5 Dec 2025) Taylor Approximation Variance Reduction for Approximation Errors in PDE-constrained Bayesian Inverse Problems
• (Zhang et al., 2018) Surrogate-Based Bayesian Inverse Modeling of the Hydrological System: An Adaptive Approach Considering Surrogate Approximation Error
• (Huggins et al., 2018) Practical bounds on the error of Bayesian posterior approximations: A nonasymptotic approach
• (Alexanderian et al., 2022) Optimal design of large-scale nonlinear Bayesian inverse problems under model uncertainty
• (Koval et al., 2024) Non-intrusive optimal experimental design for large-scale nonlinear Bayesian inverse problems using a Bayesian approximation error approach
• (Hayashi, 2018) Variational Approximation Error in Bayesian Non-negative Matrix Factorization
• (Lie et al., 2019) Error bounds for some approximate posterior measures in Bayesian inference
• (Nicholson et al., 2018) Estimation of the Robin coefficient field in a Poisson problem with uncertain conductivity field
• (Kaipio et al., 2019) A Bayesian approach to improving the Born approximation for inverse scattering with high contrast materials