- The paper introduces Bayesian Deep Kernel Networks (BDKN), which integrate deep ensembles with closed‐form Bayesian aggregation to produce analytic predictive distributions.
- It leverages ensemble-induced feature maps and Bayesian linear regression to automatically downweight poor ensemble members, ensuring calibrated uncertainty.
- Empirical evaluations on UCI benchmarks show that BDKN achieves robust performance and scalability compared to GP, DE, and other Bayesian methods under varied optimization regimes.
Efficient Bayesian Deep Ensembles via Analytic Predictive Inference
Motivation and Contributions
The paper "Efficient Bayesian Deep Ensembles via Analytic Predictive Inference" (2607.06776) addresses the challenge of achieving simultaneously scalable inference, robust uncertainty quantification, and interpretability in regression models using deep neural networks. Conventional approaches such as Gaussian Processes (GPs), Bayesian Neural Networks (BNNs), Bayesian Last-Layer (BLL) networks, and Deep Ensembles (DE) expose fundamental trade-offs in scalability, uncertainty calibration, and probabilistic interpretability. GPs afford principled uncertainty estimation but are computationally onerous, especially for large datasets or high-dimensional feature spaces. BNNs provide theoretical rigor but rely on intractable approximate inference, often resulting in miscalibrated uncertainty. BLLs offer practical tractability but restrict uncertainty to a fixed latent representation. DEs achieve empirical success via simple model averaging, but lack a diagnostic probabilistic structure and are susceptible to degradation when ensemble member quality varies.
The proposed framework, Bayesian Deep Kernel Networks (BDKN), unites the scalability of deep ensembles with a closed-form Bayesian aggregation mechanism, delivering analytic posterior predictive distributions and interpretable probabilistic weights. The construction leverages:
- Ensemble-induced feature maps: Multiple independently trained neural networks induce a heterogeneous feature space representation.
- Bayesian linear aggregation: Ensemble outputs are combined via Bayesian linear regression, yielding calibrated uncertainty estimates and posterior-driven aggregation.
- Finite-rank GP interpretation: The resultant model is interpretable as a finite-rank GP whose basis functions are learned and automatically weighted.
This approach robustly downweights poorly supported ensemble members without manual heuristics, supports scalable training (as feature maps are trained independently), and delivers principled uncertainty estimates with minimal computational overhead relative to standard deep ensembles.
Figure 1: Ensemble-induced feature representation, Bayesian linear aggregation, and analytic posterior predictive inference in BDKN.
Methodology
The BDKN procedure begins with H independently trained neural networks, each producing a nonlinear feature map ϕh(xi;θh). These feature maps collectively constitute the columns of the design matrix Φ∈RN×H for N samples. Ensemble members are trained using a heteroscedastic objective, outputting both mean and variance to encourage diversity in smoothness and robustness across the ensemble.
Upon completion of ensemble training, Bayesian inference is performed exclusively on the ensemble outputs in function space. The aggregation is achieved via Bayesian linear regression, placing a zero-mean isotropic Gaussian prior (variance σβ2) on linear coefficients β and a Jeffreys prior on observation noise variance σε2. The conditional posterior over coefficients is Gaussian; the noise variance marginal posterior (under the Jeffreys prior) is inverse-gamma. For a new input, predictive inference—integrating out both coefficients and noise variance—yields a closed-form Student-t distribution. Hyperparameters are set via empirical Bayes, maximizing the log marginal likelihood in closed form.
The induced model is interpretable as a finite-rank GP with a kernel
k(x,x′)=σε2σβ2h=1∑Hϕh(x)ϕh(x′)
where the hypothesis space is confined to the linear span of ensemble-induced feature maps. The complexity and expressivity are controlled directly by ensemble size H.
Computational Complexity
BDKN maintains a computational profile nearly identical to standard deep ensembles. The main costs are ensemble training, which is fully parallelizable. Bayesian aggregation introduces an overhead of ϕh(xi;θh)0 for forming ϕh(xi;θh)1 and solving the ϕh(xi;θh)2 linear system, but this cost is negligible for moderate ϕh(xi;θh)3. The aggregation step is independent of dataset size ϕh(xi;θh)4 in memory and only linear in ϕh(xi;θh)5 during aggregation. This allows for efficient scaling to large data and ensemble sizes.
Empirical Results
BDKN is evaluated on standard UCI regression benchmarks and compared to GP, DE, DKL, BLL, LD-BLL, MFVI-BNN, and VBLL. Two training regimes are considered: aggressive (high learning rate, short epochs) and conservative (low learning rate, longer schedules).
Key empirical findings:
- BDKN achieves competitive RMSE and NLL across datasets under both settings.
- Under aggressive optimization, BDKN demonstrates robust behavior, with DE and LDBLL exhibiting instability and greater variance—most notably, DE shows substantial variance and catastrophic failures on Power and Sarcos (see table in paper).
- Under conservative optimization, most models stabilize and BDKN maintains performance parity with DE, MFVI-BNN, and GP.
- Increasing ensemble size (e.g., from 5 to 50) monotonically improves both RMSE and NLL, with training time scaling efficiently given parallelization.


Figure 2: Test RMSE and NLL dynamics for Boston dataset under high learning rate training, highlighting BDKN's stability relative to DE, BLL, LDBLL, and MFVI-BNN across training epochs.
BDKN also demonstrates improved uncertainty calibration and robustness relative to DE in settings with degraded or heterogeneous member quality, as Bayesian aggregation adaptively downweights poorly performing ensemble members. Consistency and convergence are observed across both high and low learning rate regimes on datasets such as Boston and Yacht.
Theoretical and Practical Implications
BDKN provides a bridge between deep ensemble methodologies and the theoretical guarantees of Bayesian inference, offering:
- Diagnostic posterior-driven aggregation that mitigates overconfidence, especially under distribution shift or variable optimization quality.
- Interpretability via posterior weights, permitting analysis of individual ensemble member utility and robustness.
- Closed-form predictive uncertainty, facilitating integration as a component in hierarchical learning systems such as reinforcement learning or physics-informed regression.
- Scalability to large datasets without incurring the cubic cost associated with GPs.
Finite-rank GP interpretation and analytic aggregation, together with empirical evidence, indicate that BDKN serves as an effective surrogate for full GP regression in large-scale settings, provided sufficient ensemble diversity and size.
Future Directions
Extensions to classification, structured prediction, and other non-Gaussian likelihoods are natural next steps, with analytic tractability of posterior predictive distributions an open challenge. Enhancing uncertainty calibration by incorporating principled feature map regularization or latent derivative modeling may further improve robustness. Applying BDKN to reinforcement learning and scientific machine learning domains is promising, where scalable uncertainty-aware regression is operationally critical.
Conclusion
BDKN unifies the scalability and empirical strengths of deep ensembles with Bayesian aggregation, yielding interpretable, calibrated, and robust predictive inference in regression. The framework achieves principled uncertainty quantification without relying on computationally intensive approximate inference or restrictive single-representation architectures. Empirical analysis substantiates BDKN's robustness under varying optimization, ensemble size, and dataset scale, positioning it as a viable, scalable surrogate for GP and other Bayesian models in practical regression tasks.