Bayesian Neural Network Surrogates

• Bayesian Neural Network Surrogates are probabilistic models that place a prior on neural network weights to provide calibrated uncertainty and approximate high-cost simulators.
• They employ scalable inference strategies including variational approximations and MCMC to efficiently predict outcomes in scientific and engineering applications.
• Surrogate architectures integrate multi-fidelity and multi-modal data through GP hybrids, tensor decompositions, and conjugate last-layer methods to enhance performance and reduce computational cost.

Bayesian neural network (BNN) surrogates are probabilistic machine learning models that approximate expensive or intractable forward models, functions, or posteriors while providing calibrated uncertainty quantification. The Bayesian treatment of neural network parameters enables direct estimation of epistemic uncertainty and offers a structured framework for exploitation in scientific computing, engineering design, simulation-based inference, and optimization. BNN surrogates are increasingly used to replace high-cost simulators, accelerate uncertainty-aware decision-making, provide emulators for inverse problems, and support active and multi-fidelity learning.

1. Mathematical Foundations and Inference Strategies

Bayesian neural network surrogates are constructed by placing a prior distribution over neural network weights $w$ and performing Bayesian inference conditioned on observed data $D = \{x_i, y_i\}_{i=1}^N$. The predictive distribution is expressed as

$p(y^*|x^*, D) = \int p(y^*|x^*, w) p(w|D) dw.$

Inference typically resorts to variational approximations (e.g., mean-field Gaussian, low-rank priors), stochastic gradient MCMC (e.g., SGHMC, HMC, NUTS), or ensemble techniques. Mean-field variational Bayes is standard for scalable inference when $w$ is high-dimensional, typically maximizing the Evidence Lower Bound (ELBO): $$\mathrm{ELBO}(q) = \mathbb{E}_{q(w)}[\log p(D|w)] - \mathrm{KL}[q(w)\|p(w)].$$ Alternatively, fixed-weight priors can be inferred from functional priors via pre-training and low-rank factorization, inducing correlations between weight parameters and facilitating the injection of prior knowledge into the parameter space (Ghorbanian et al., 2024).

In high-dimensional or infinite-width limits, BNNs approach Gaussian process priors with explicitly computable kernels, enabling tractable exact Bayesian inference for predictive means and variances (Hirt et al., 12 Dec 2025, Li et al., 2023). For multi-modal data, surrogates can exploit conditionally conjugate last-layer estimation to perform analytic Bayesian linear regression in the final layer, reducing variance in variational inference (Taylor et al., 26 Sep 2025).

2. Surrogate Architectures and Training Workflows

BNN surrogates adopt network architectures matched to the structure of the task. For time-series inversion, lightweight 1D convolutional residual encoders (1D-ResNet) have been used to map stochastic Petri Net (SPN) trajectories $x\in \mathbb{R}^{T \times d_{in}}$ to rate coefficients $\theta\in\mathbb{R}^{k}$ in likelihood-free settings (Manu et al., 14 Jul 2025). Structural engineering and mechanics surrogates commonly employ fully connected multilayer perceptrons (MLPs) or convolutional encoder–decoder U-nets for scalar, vector, or field outputs (Kuhn et al., 29 Sep 2025, Zhang et al., 2021).

Multi-fidelity workflows integrate low-fidelity evaluations using Gaussian processes, whose predictive means and variances are injected into the BNN surrogate via Gauss-Hermite quadrature nodes, resulting in a two-stage GP-BNN hybrid (GPBNN) for hierarchical uncertainty propagation (Kerleguer et al., 2023). Multi-modal surrogates may fuse auxiliary data streams with either joint neural encoders or layered (hierarchical) BNNs that predict each modality and then aggregate outputs nonlinearly for the main target variable (Taylor et al., 26 Sep 2025).

For high-dimensional regression, tensorized architectures—such as the Bayesian Interpolating Neural Network (B-INN)—employ canonical polyadic (CP) or tensor decomposition alongside alternating closed-form Bayesian linear regression updates to reduce computational complexity to $\mathcal{O}(N)$ while retaining analytic posteriors per block (Park et al., 30 Jan 2026).

Training typically involves simulated data generation (e.g., Gillespie simulations for SPNs, parametric finite element analysis for structural surrogates), batch optimization (e.g., Adam optimizer), and uncertainty-aware validation or calibration phases. Depending on computational constraints and data regimes, variational, MCMC, or MC-dropout inference may be favored (Fotias et al., 29 Jul 2025, Manu et al., 14 Jul 2025).

3. Uncertainty Quantification and Calibration

The fundamental benefit of BNN surrogates is robust quantification of epistemic and, when needed, aleatoric uncertainty. The posterior over weights $w$ translates directly into predictive mean and variance estimates. For variational inference, prediction at $x$ is via averaging over $T'$ sampled weights $\{w^{(t)}\}$: $$\mu(x) = \frac{1}{T'} \sum_{t=1}^{T'} f(x ; w^{(t)}),\quad \sigma^2(x) = \frac{1}{T'}\sum_{t=1}^{T'} [f(x ; w^{(t)}) - \mu(x)]^2.$$ Aleatoric uncertainty arises from additional noise heads or is inferred via joint residual modeling (e.g., in physics-constrained weak-form BNN solvers (Zhang et al., 2021)).

Empirical coverage is assessed by comparing predicted intervals to nominal coverage. In practice, surrogates often exhibit over- or under-confidence, which is addressed by post-hoc calibration: scaling the predictive standard deviation by a factor $\kappa$ to align observed and nominal interval coverage (Kuhn et al., 29 Sep 2025). Calibration is tracked by metrics such as total calibration error, bias (signed error), miscalibration area, and negative log predictive density.

Dropout-based inference and ensembles offer scalable uncertainty quantification, though MC-dropout may underperform exact MCMC posterior sampling in coverage fidelity. Infinite-width BNNs inherit the uncertainty structure of Gaussian processes and are especially well-calibrated in high-dimensional, data-scarce regimes (Li et al., 2023).

4. Application Domains and Performance

BNN surrogates have been deployed across a range of applications:

• Bayesian Optimization (BO) and active learning: BNNs are used as scalable surrogates for guiding exploration via acquisition functions such as Expected Improvement (EI), Upper Confidence Bound (UCB), and their MC variants. They outperform GPs in high-dimensional controller tuning and non-stationary objectives, while in low-dimensional or stationary settings GPs remain competitive (Fotias et al., 29 Jul 2025, Hirt et al., 12 Dec 2025, Li et al., 2023, Makrygiorgos et al., 14 Apr 2025).
• Physics-constrained simulation and surrogate PDE solvers: BNNs trained on weak-form PDE residuals offer label-free surrogates for steady-state diffusion, elasticity, and nonlinear PDEs, supporting calibrated field-level uncertainty and extrapolation to unseen boundary conditions (Zhang et al., 2021).
• Multi-fidelity/multi-modal emulation: GP-BNN hybrids propagate GP uncertainty into the BNN emulator for high-fidelity codes, yielding sharp, calibrated predictions even when data coverage is sparse or non-uniform across fidelity levels (Kerleguer et al., 2023, Taylor et al., 26 Sep 2025).
• Mechanics and infrastructure assessment: Anchored BNN ensembles with functional priors enable the integration of a priori knowledge, robust out-of-distribution detection, and uncertainty-aware triage in mechanics surrogates and large-scale engineering asset management (Ghorbanian et al., 2024, Kuhn et al., 29 Sep 2025).
• Large-scale physical systems: Tensorized B-INNs achieve scalable, reliable uncertainty quantification in parametric PDEs and aerodynamics with up to millions of training points, at 20–10,000x speedup over BNN-MCMC or GP surrogates (Park et al., 30 Jan 2026).
• Parallel surrogate MCMC: Surrogate-assisted parallel tempering combines lightweight neural surrogates for likelihood evaluation with robust exploration of complex BNN posteriors, supporting large-scale Bayesian inversion and uncertainty quantification (Chandra et al., 2018).

Benchmarks consistently confirm that in low dimensions and when smoothness/stationarity holds, GP surrogates may match or outperform BNN surrogates. However, BNN surrogates (especially infinite-width/NNGP) become crucial as problem dimension, data complexity, or non-stationarity increases (Li et al., 2023, Hirt et al., 12 Dec 2025).

5. Algorithmic and Computational Considerations

Scalability is central in surrogate modeling. Classic GP inference scales cubically in the number of samples $N$, limiting its practical use beyond $N \sim 10^4$. Finite-width BNNs with MCMC or variational inference exhibit linear scaling in $N$ for fixed size but substantial cost in network size or when computing exact Bayesian posteriors in high-dimensional parameterizations (Park et al., 30 Jan 2026, Fotias et al., 29 Jul 2025).

High-dimensional BNN surrogates leverage infinite-width GP limits or explicit tensor decompositions. For example, the B-INN alternates closed-form Bayesian linear regression updates per dimension, achieving $\mathcal{O}(N)$ overall complexity and exact analytic block-wise posteriors (Park et al., 30 Jan 2026). Dropout, deep ensembles, and conjugate last-layer inference further reduce computational cost and deliver scalable uncertainty quantification for dense data and multi-modal emulation (Taylor et al., 26 Sep 2025).

Surrogate-based Bayesian optimization, active learning, and uncertainty-driven exploration rely on efficient retraining or warm-start strategies to maintain computational tractability while guiding the acquisition of new data (Hirt et al., 12 Dec 2025, Park et al., 30 Jan 2026).

6. Methodological Extensions and Limitations

Key extensions of the BNN surrogate paradigm include:

• Anchored ensembles using low-rank prior covariance learned from functional realizations, facilitating principled prior specification and transfer learning (Ghorbanian et al., 2024).
• Conjugate last-layer estimation for fast, low-variance Bayesian update in multi-modal and multi-fidelity NNs (Taylor et al., 26 Sep 2025).
• Gradient-informed BNN surrogates for BO, where automatic differentiation provides surrogate and gradient observations, improving convergence in high dimensions (Makrygiorgos et al., 14 Apr 2025).
• Physics-constrained BNN surrogates, where weak-form PDE residuals define the likelihood, supporting label-free training on diverse domains and BCs but requiring bespoke network design and careful variational inference (Zhang et al., 2021).

Limitations include imperfect uncertainty coverage for parameters with weak identifiability, dependence on coverage of covariate regimes in training data, and challenges in scaling MCMC-based inference for very large parameter spaces or high-fidelity data (Manu et al., 14 Jul 2025, Kerleguer et al., 2023). For high-dimensional or operator-valued inputs/outputs, tensor or graph-based surrogates, deep operator networks, or correlated weight priors are ongoing areas of exploration (Park et al., 30 Jan 2026, Ghorbanian et al., 2024).

7. Comparative Performance and Practical Recommendations

Extensive cross-domain benchmarking finds that BNN surrogate performance is highly problem-dependent:

• Infinite-width BNNs (NNGP) excel in high-dimensional, data-scarce, and non-stationary settings, often matching or exceeding GP surrogates (Li et al., 2023, Hirt et al., 12 Dec 2025).
• Full Bayesian inference (HMC) delivers the most credible uncertainty, but stochastic gradient schemes (SGHMC, MC-dropout) are practical for large-scale or time-constrained settings (Fotias et al., 29 Jul 2025).
• Deep ensembles, while computationally efficient, may lack exploration diversity in low-data regimes (Li et al., 2023).
• For multi-modality and missing data, structured SVI with conjugate Bayesian last-layer inference yields improved prediction and calibration (Taylor et al., 26 Sep 2025).

Recommendations are to default to infinite-width BNNs or kernel-based surrogates in high-dimensional, non-stationary, or sparse-data regimes; use mean-field or hybrid variational inference for moderate-scale problems requiring computational tractability; and leverage anchored or multi-modal approaches where prior information or heterogeneous data are present (Ghorbanian et al., 2024). For simulation-based Bayesian inference in discrete-event or physically-informed settings, amortized BNN surrogates with MC-dropout achieve likelihood-free, calibrated posterior approximation orders-of-magnitude faster than rejection or MCMC samplers (Manu et al., 14 Jul 2025).

Empirical studies emphasize the importance of matching surrogate model inductive biases, training procedures, and uncertainty quantification strategy to the specific application characteristics and data constraints.