Deep Neural Network Ensembles

• Deep Ensembles are collections of independently trained neural networks that aggregate predictions to enhance accuracy, calibration, and robustness.
• They employ aggregation techniques such as linear pooling and quantile-based methods to reduce model variance and improve uncertainty estimation.
• Empirical results show that deep ensembles offer significant performance gains across applications, with practical strategies available for efficient deployment.

Deep neural network ensembles ("deep ensembles") are a methodology in which multiple neural network models are independently trained—typically via different random initializations or heterogeneous architectures—and combined at inference time to produce aggregated predictive distributions or class probabilities. This approach aims to enhance predictive accuracy, robustness, calibration, and uncertainty quantification relative to single-model baselines. Deep ensembles have become a de facto state-of-the-art method for uncertainty estimation and model robustness across many domains, including time series forecasting and computer vision.

1. Formulation and Aggregation Schemes

Consider a supervised prediction problem where input $x\in\mathcal{X}$ and target $y\in\mathbb{R}$ (regression) or $y\in\{1,\ldots,C\}$ (classification). A deep ensemble of size $m$ comprises $m$ neural network models, each independently trained, yielding predictive distributions $f_i(y|x)$ or, for probabilistic forecasting, quantile functions $Q_i(\alpha|x)$. The ensemble aims to aggregate these member distributions into a single $F_{\text{agg}}$ or $Q_{\text{agg}}$ that is sharper and reliably calibrated.

Two principal classes of aggregation methods are established (Schulz et al., 2022):

• Probability-based aggregation (Linear Pool):

$$f_{\rm LP}(y|x) = \sum_{i=1}^m w_i f_i(y|x), \qquad F_{\rm LP}(y|x) = \sum_{i=1}^m w_i F_i(y|x)$$

with nonnegative weights $w_i$ summing to unity, chosen by equal assignment, skill-based validation (e.g., $\propto \exp(-\overline{\text{CRPS}_i})$), or direct optimization on validation data.

• Quantile-based aggregation (Vincentization):

$$Q_{\rm VI}(\alpha|x) = \sum_{i=1}^m w_i Q_i(\alpha|x)$$

Optionally, location and scale corrections (intercept $a$, spread $w_0$) can be fit by minimizing the mean CRPS on a held-out set:

$$Q_{\rm agg}(\alpha|x) = a + w_0 \sum_{i=1}^m w'_i Q_i(\alpha|x)$$

All parameters can be fitted jointly for bias/dispersion correction.

For standard classification, the ensemble predictive is the mean of base posterior vectors: $$p_{\text{ens}}(y|x) = \frac{1}{m} \sum_{i=1}^m p_i(y|x)$$ The final predicted label is $\hat y = \arg\max_c p_{\text{ens}}(y=c|x)$.

2. Theoretical Justification and Diversity

Ensembling reduces variance via independent error cancellation across multiple local optima in the non-convex neural loss landscape (Fort et al., 2019). In regression, the ambiguity decomposition formalizes the variance reduction: $$\mathbb{E}\left[(y - \hat y)^2\right] = \text{avg. member error} - \text{diversity term}$$ For classification, diversity can be measured by pairwise disagreement rates, Q-statistic, or correlation metrics (Liu et al., 2019). However, recent work finds that, for high-capacity neural ensembles, artificially increasing diversity (e.g., via bagging or explicit decorrelation penalties) can be detrimental—optimal performance is achieved by training independently and focusing on maximizing single-model accuracy (Abe et al., 2023).

Empirically, independently trained networks (random initializations) occupy markedly different functional modes, yielding maximal functional diversity and improved ensemble performance relative to subspace or dropout-based approaches (Fort et al., 2019).

3. Bayesian and Empirical Bayes Perspectives

Although deep ensembles are not derived from a Bayesian prior, they can be viewed as a Monte Carlo approximation to Bayesian model averaging, where the approximate posterior over parameters $q(\theta)$ is a mixture of point masses at each trained model's parameters (Hoffmann et al., 2021, Loaiza-Ganem et al., 29 Jan 2025): $$q(\theta) = \frac{1}{m} \sum_{i=1}^m \delta(\theta - \theta_i^*)$$ The predictive distribution becomes

$$p_{\text{ens}}(y|x) = \mathbb{E}_{\theta \sim q(\theta)}\big[ p(y|x,\theta)\big]$$

This mixture can be interpreted as an empirical Bayes solution: deep ensembles optimize both the approximating distribution and an implicit, data-dependent prior that is maximally concentrated on high-likelihood solutions (Loaiza-Ganem et al., 29 Jan 2025). This mechanism accounts for superior calibration and robustness to distribution shift.

Enhancing the ensemble with local posterior structure (e.g., equipping each member with a local Laplace or low-rank Gaussian beyond a point estimate) can improve certain out-of-distribution metrics in small ensembles but typically incurs a loss in in-distribution log-likelihood at large $m$ (Jordahn et al., 17 Mar 2025).

PAC-Bayesian approaches provide a formal mechanism for optimizing ensemble weights: $$\min_{w:\sum w_i=1} \mathrm{TandemLoss}(w) + \text{complexity penalty}$$ where the tandem loss penalizes correlated errors between ensemble members and can yield nonvacuous generalization bounds. Optimizing the weights under PAC-Bayesian criteria usually yields slight improvements over uniform weighting, especially when including correlated checkpoints (e.g., from snapshot ensembling) (Hauptvogel et al., 8 Jun 2024).

4. Empirical Performance, Domains, and Efficiency

Deep ensembles yield compelling empirical gains in accuracy, calibration, and robustness across domains. In time series classification, an ensemble of 60 independently trained CNNs exceeds both single deep models and the best non-deep-learning methods (e.g., HIVE-COTE) (Fawaz et al., 2019). For small sample sizes, ensembles of shallow models outperform larger single networks of matching compute budget, due to a regularizing "averaging" that smooths model outputs (Brigato et al., 2021).

The effectiveness of deep ensembles is most pronounced when $m \approx 10$–$20$; returns diminish for larger $m$ (Schulz et al., 2022). Table structures and step-by-step guidelines detail optimal construction processes:

• Use independent initializations and, if possible, architectural diversity for maximizing unbiased variance reduction.
• Employ quantile aggregation with post-hoc intercept and scale correction for probabilistic forecasting.
• Distill large ensembles via pruning techniques—focal diversity-based hierarchical pruning identifies smaller ensembles ($S_d \approx 0.4-0.6\,m$) of equal or superior generalization at reduced inference cost (Wu et al., 2023).

For computational efficiency, several variants have been developed:

• Noisy Deep Ensemble: Train a single "parent" to convergence, then create "children" by noise-injected weight perturbation and short fine-tuning, capturing most of the gain at a fraction of the cost (Sakai et al., 8 Apr 2025).
• Layer Ensembles: Independently sample layer weights to assemble multiple virtual networks from shared layer outputs, enabling up to $19\times$ inference speed-up and quadratic memory reduction compared to standard ensembles (Oleksiienko et al., 2022).
• Ensemble-Spread Over Time (DESOT): In sequence-processing tasks, distribute ensemble evaluation temporally across frames in a sequence, achieving ensemble-level uncertainty at single-model inference cost (Meding et al., 2023).

5. Diversity, Robustness, and Limitations

Robustness to adversarial deception and distribution shift is a hallmark of deep ensembles: diversity in ensemble errors means attacks that fool one member less frequently transfer to the consensus, and model uncertainty is better calibrated under OOD shift (Liu et al., 2019, Wu et al., 2023). However, there are important caveats:

• Large homogeneous single models can match ensemble accuracy, calibration, and OOD performance given similar parameter budgets; ensembling per se does not uniquely confer robustness (Abe et al., 2022).
• In high-capacity regimes, promoting additional predictive diversity can actually harm ensemble accuracy, as most residual errors arise on rare, ambiguous cases, and forced diversity yields more errors on easy predictions (Abe et al., 2023).
• Bootstrapped Deep Ensembles can more faithfully quantify epistemic uncertainty by incorporating parameter uncertainty due to finite dataset size, improving confidence interval coverage over standard ensembles (Sluijterman et al., 2022).

6. Interpretability, Regularization Connections, and Open Questions

Cluster-path analysis and related methods formalize interpretability within ensemble frameworks: by analyzing activation-paths through clustered hidden spaces, one identifies stable generalizable features and separates confidently predicted ("good") from ambiguous ("bad") input regions (Tao, 2019). This perspective relates classical regularization (ridge, dropout, early stopping) to ensemble diversity: each regularizer biases toward coarse partitioning or variance-increasing mechanisms, exploiting the same uncertainty and error-cancellation effects as deep ensembles.

Despite substantial empirical validation, open questions remain:

• Scalability of diversity metrics and ensemble pruning to ultra-large ensembles (Wu et al., 2023).
• The interplay between classical Bayesian methods, empirical Bayes, and ensemble weighting in terms of coverage guarantees and OOD detection (Hauptvogel et al., 8 Jun 2024, Jordahn et al., 17 Mar 2025).
• Algorithmic strategies for maximizing "useful" diversity without sacrificing single-model accuracy.

7. Practical Guidelines and Recommendations

• Train $m=10$–$20$ models with different random seeds and/or architectures; larger $m$ offers diminishing marginal utility (Schulz et al., 2022).
• For probabilistic forecasting, prefer quantile-based aggregation with CRPS-minimized location/scale recalibration.
• When resource-constrained, consider pruning, noise-injection methods, or layer-wise sharing schemes to reduce compute costs while retaining ensemble benefits (Wu et al., 2023, Sakai et al., 8 Apr 2025, Oleksiienko et al., 2022).
• Always validate ensemble calibration, especially under distribution shift; post-hoc temperature scaling may help.
• Prefer independent training over forced diversity-enhancing joint objectives, unless in low-capacity/small-data settings (Abe et al., 2023).

Deep ensembles remain a foundational and empirically robust strategy for accuracy, calibration, and practical uncertainty quantification in deep learning, with ongoing refinements in aggregation, efficiency, and theoretical understanding (Fort et al., 2019, Loaiza-Ganem et al., 29 Jan 2025, Hauptvogel et al., 8 Jun 2024).