Papers
Topics
Authors
Recent
Search
2000 character limit reached

Efficient Bayesian Deep Ensembles via Analytic Predictive Inference

Published 7 Jul 2026 in cs.LG and stat.ME | (2607.06776v1)

Abstract: We introduce an efficient Bayesian deep ensemble method for predictive regression designed to enhance interpretability while maintaining competitive predictive performance and computational efficiency. Our method combines the statistical rigor of Bayesian inference with the scalability of deep ensembles, providing calibrated uncertainty estimates that enable its use not only for standalone prediction but also as a component within broader learning systems. To achieve these goals, our work relies on three key design components: (i) low-dimensional ensemble representation: predictions are expressed as a combination of a small number of trained neural predictors, enabling scalable inference whose cost depends on ensemble size rather than dataset size; (ii) closed-form Bayesian aggregation: ensemble predictions are combined using Bayesian linear regression, yielding interpretable posterior weights and calibrated uncertainty without approximate inference; and (iii) Independent ensemble training: multiple neural networks are trained separately, producing diverse predictive representations that improve robustness and uncertainty calibration. Empirical results on standard regression benchmarks demonstrate that the proposed approach achieves competitive predictive performance while maintaining reliable uncertainty estimates across settings.

Summary

  • The paper introduces Bayesian Deep Kernel Networks (BDKN), which integrate deep ensembles with closed‐form Bayesian aggregation to produce analytic predictive distributions.
  • It leverages ensemble-induced feature maps and Bayesian linear regression to automatically downweight poor ensemble members, ensuring calibrated uncertainty.
  • Empirical evaluations on UCI benchmarks show that BDKN achieves robust performance and scalability compared to GP, DE, and other Bayesian methods under varied optimization regimes.

Efficient Bayesian Deep Ensembles via Analytic Predictive Inference


Motivation and Contributions

The paper "Efficient Bayesian Deep Ensembles via Analytic Predictive Inference" (2607.06776) addresses the challenge of achieving simultaneously scalable inference, robust uncertainty quantification, and interpretability in regression models using deep neural networks. Conventional approaches such as Gaussian Processes (GPs), Bayesian Neural Networks (BNNs), Bayesian Last-Layer (BLL) networks, and Deep Ensembles (DE) expose fundamental trade-offs in scalability, uncertainty calibration, and probabilistic interpretability. GPs afford principled uncertainty estimation but are computationally onerous, especially for large datasets or high-dimensional feature spaces. BNNs provide theoretical rigor but rely on intractable approximate inference, often resulting in miscalibrated uncertainty. BLLs offer practical tractability but restrict uncertainty to a fixed latent representation. DEs achieve empirical success via simple model averaging, but lack a diagnostic probabilistic structure and are susceptible to degradation when ensemble member quality varies.

The proposed framework, Bayesian Deep Kernel Networks (BDKN), unites the scalability of deep ensembles with a closed-form Bayesian aggregation mechanism, delivering analytic posterior predictive distributions and interpretable probabilistic weights. The construction leverages:

  • Ensemble-induced feature maps: Multiple independently trained neural networks induce a heterogeneous feature space representation.
  • Bayesian linear aggregation: Ensemble outputs are combined via Bayesian linear regression, yielding calibrated uncertainty estimates and posterior-driven aggregation.
  • Finite-rank GP interpretation: The resultant model is interpretable as a finite-rank GP whose basis functions are learned and automatically weighted.

This approach robustly downweights poorly supported ensemble members without manual heuristics, supports scalable training (as feature maps are trained independently), and delivers principled uncertainty estimates with minimal computational overhead relative to standard deep ensembles. Figure 1

Figure 1: Ensemble-induced feature representation, Bayesian linear aggregation, and analytic posterior predictive inference in BDKN.


Methodology

The BDKN procedure begins with HH independently trained neural networks, each producing a nonlinear feature map ϕh(xi;θh)\phi_h(x_i; \theta_h). These feature maps collectively constitute the columns of the design matrix Φ∈RN×H\boldsymbol{\Phi} \in \mathbb{R}^{N \times H} for NN samples. Ensemble members are trained using a heteroscedastic objective, outputting both mean and variance to encourage diversity in smoothness and robustness across the ensemble.

Upon completion of ensemble training, Bayesian inference is performed exclusively on the ensemble outputs in function space. The aggregation is achieved via Bayesian linear regression, placing a zero-mean isotropic Gaussian prior (variance σβ2\sigma^2_\beta) on linear coefficients β\boldsymbol{\beta} and a Jeffreys prior on observation noise variance σε2\sigma^2_{\varepsilon}. The conditional posterior over coefficients is Gaussian; the noise variance marginal posterior (under the Jeffreys prior) is inverse-gamma. For a new input, predictive inference—integrating out both coefficients and noise variance—yields a closed-form Student-tt distribution. Hyperparameters are set via empirical Bayes, maximizing the log marginal likelihood in closed form.

The induced model is interpretable as a finite-rank GP with a kernel

k(x,x′)=σε2σβ2∑h=1Hϕh(x)ϕh(x′)k(x, x') = \sigma^2_{\varepsilon} \sigma^2_\beta \sum_{h=1}^H \phi_h(x) \phi_h(x')

where the hypothesis space is confined to the linear span of ensemble-induced feature maps. The complexity and expressivity are controlled directly by ensemble size HH.


Computational Complexity

BDKN maintains a computational profile nearly identical to standard deep ensembles. The main costs are ensemble training, which is fully parallelizable. Bayesian aggregation introduces an overhead of ϕh(xi;θh)\phi_h(x_i; \theta_h)0 for forming ϕh(xi;θh)\phi_h(x_i; \theta_h)1 and solving the ϕh(xi;θh)\phi_h(x_i; \theta_h)2 linear system, but this cost is negligible for moderate ϕh(xi;θh)\phi_h(x_i; \theta_h)3. The aggregation step is independent of dataset size ϕh(xi;θh)\phi_h(x_i; \theta_h)4 in memory and only linear in ϕh(xi;θh)\phi_h(x_i; \theta_h)5 during aggregation. This allows for efficient scaling to large data and ensemble sizes.


Empirical Results

BDKN is evaluated on standard UCI regression benchmarks and compared to GP, DE, DKL, BLL, LD-BLL, MFVI-BNN, and VBLL. Two training regimes are considered: aggressive (high learning rate, short epochs) and conservative (low learning rate, longer schedules).

Key empirical findings:

  • BDKN achieves competitive RMSE and NLL across datasets under both settings.
  • Under aggressive optimization, BDKN demonstrates robust behavior, with DE and LDBLL exhibiting instability and greater variance—most notably, DE shows substantial variance and catastrophic failures on Power and Sarcos (see table in paper).
  • Under conservative optimization, most models stabilize and BDKN maintains performance parity with DE, MFVI-BNN, and GP.
  • Increasing ensemble size (e.g., from 5 to 50) monotonically improves both RMSE and NLL, with training time scaling efficiently given parallelization. Figure 2

Figure 2

Figure 2

Figure 2: Test RMSE and NLL dynamics for Boston dataset under high learning rate training, highlighting BDKN's stability relative to DE, BLL, LDBLL, and MFVI-BNN across training epochs.

BDKN also demonstrates improved uncertainty calibration and robustness relative to DE in settings with degraded or heterogeneous member quality, as Bayesian aggregation adaptively downweights poorly performing ensemble members. Consistency and convergence are observed across both high and low learning rate regimes on datasets such as Boston and Yacht.


Theoretical and Practical Implications

BDKN provides a bridge between deep ensemble methodologies and the theoretical guarantees of Bayesian inference, offering:

  • Diagnostic posterior-driven aggregation that mitigates overconfidence, especially under distribution shift or variable optimization quality.
  • Interpretability via posterior weights, permitting analysis of individual ensemble member utility and robustness.
  • Closed-form predictive uncertainty, facilitating integration as a component in hierarchical learning systems such as reinforcement learning or physics-informed regression.
  • Scalability to large datasets without incurring the cubic cost associated with GPs.

Finite-rank GP interpretation and analytic aggregation, together with empirical evidence, indicate that BDKN serves as an effective surrogate for full GP regression in large-scale settings, provided sufficient ensemble diversity and size.


Future Directions

Extensions to classification, structured prediction, and other non-Gaussian likelihoods are natural next steps, with analytic tractability of posterior predictive distributions an open challenge. Enhancing uncertainty calibration by incorporating principled feature map regularization or latent derivative modeling may further improve robustness. Applying BDKN to reinforcement learning and scientific machine learning domains is promising, where scalable uncertainty-aware regression is operationally critical.


Conclusion

BDKN unifies the scalability and empirical strengths of deep ensembles with Bayesian aggregation, yielding interpretable, calibrated, and robust predictive inference in regression. The framework achieves principled uncertainty quantification without relying on computationally intensive approximate inference or restrictive single-representation architectures. Empirical analysis substantiates BDKN's robustness under varying optimization, ensemble size, and dataset scale, positioning it as a viable, scalable surrogate for GP and other Bayesian models in practical regression tasks.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Tweets

Sign up for free to view the 1 tweet with 5 likes about this paper.