Optimal Ensemble Size: Performance & Efficiency

• Optimal ensemble size is the number of component models selected to optimize trade-offs between predictive accuracy, computational cost, and interpretability.
• It is defined through theoretical frameworks like majority and weighted voting, ensuring model diversity and independence for reduced error.
• Empirical and optimization-based studies determine the sweet spot that balances performance gains with resource constraints and energy efficiency.

Optimal ensemble size refers to the selection or determination of the number of component predictors (classifiers, regressors, optimizers, or models) to include in an ensemble, so as to maximize generalization performance, computational efficiency, interpretability, or other relevant system objectives under given constraints. The question of optimal ensemble size arises across diverse machine learning domains and often reflects a trade-off between added predictive power and increased resource or design costs. This topic spans theoretical foundations, empirical heuristics, optimization-based formulations, and emerging concerns such as energy efficiency and explainability.

1. Theoretical Foundations of Ensemble Size

Several theoretical frameworks underpin the analysis of optimal ensemble size. Classical results for majority voting (MV) and weighted majority voting (WMV) show that the aggregation of independent strong models can reduce error, but the rate and conditions for improvement differ by aggregation rule and diversity within the ensemble.

• In a geometric framework, classifier outputs are represented as points in a $p$-dimensional Euclidean space (where $p$ is the number of classes). For MV, adding more strong, independent classifiers can strictly decrease the ensemble’s loss, with Theorem 1 establishing that the Euclidean loss of the centroid (ensemble vote) is never larger than that of the average individual classifier. However, the law of diminishing returns applies: adding redundant or weak classifiers can lead to performance plateaus or even regressions (Bonab et al., 2017).
• For WMV, there exists a unique “sweet spot” when the ensemble contains exactly $p$ linearly independent classifiers, affording an unbiased least squares solution for optimal weights. When the number of classifiers $m \neq p$ or if the score vectors are highly correlated, the coefficient matrix in the linear system for optimal weighting becomes rank-deficient, undermining the ability to achieve the ideal weighted combination (Bonab et al., 2017); (Bektas et al., 2023).
• In advanced deep learning settings, negative log-likelihood (NLL) or calibrated NLL (CNLL) of an ensemble follows a power law with respect to ensemble size $n$, specifically:

$\mathrm{CNLL}(n) = c + b \cdot n^{a}$

where $c$ is the asymptotic CNLL (as $n \to \infty$), $b$ is the difference between single-model and infinite-ensemble CNLL, and $a < 0$ expresses the decay rate of error with ensemble size (Lobacheva et al., 2020). This formulation enables prediction of performance gains and guides budgeted selection of $n$.

2. Optimization-Based Methodologies

Recent advances employ explicit optimization to determine or approximate optimal ensemble size, often via joint weight learning, hyperparameter selection, or combinatorial search:

• Quadratic margin maximization (QMM) prunes large ensembles to a smaller, diverse subensemble by solving:

$$\text{minimize} \quad w^T \hat{\Sigma} w \,,\quad \text{subject to } \Lambda^{\nu} w \geq \varphi^{\nu},\ w^T 1 = 1,\ w \geq 0$$

Here, $\hat{\Sigma}$ is the error covariance, $\Lambda^{\nu}$ encodes a subset of margin constraints, and the solution often yields many zero weights, thus directly reducing ensemble size without loss of generalization (Martinez, 2019).

• Bayesian hyperparameter and ensemble optimization utilizes acquisition functions to select not just model parameters but also ensemble membership dynamically. The ensemble size is fixed a priori (e.g., $m=12$) for tractability, but a post-optimization, greedy forward stepwise construction can reveal whether this size was near-optimal, offering a template for adaptive sizing (Lévesque et al., 2016).
• In exhaustive search frameworks (e.g., for black-box optimizer ensembles), GPU-accelerated combinatorial evaluations over possible pairs (or triplets) are used to determine the best ensemble configuration, where empirical performance rather than a fixed ensemble size drives optimality (Liu et al., 2020).
• Extrapolated cross-validation (ECV) leverages the predictable structure of squared prediction risk in randomized ensembles to extrapolate the risk for large ensemble sizes using one- and two-member estimators. The algorithm selects the smallest $M$ such that the risk is within a tolerance $\delta$ of the infinite-ensemble oracle, with strong consistency guarantees (Du et al., 2023).

3. Empirical and Practical Considerations

Empirical studies often reveal a "diminishing returns" phenomenon. For example, in Ensemble Kalman Filter (EnKF) applications, increasing the ensemble from small numbers (50, 70, 100) to moderate (250) yields pronounced RMSE improvements, but further increases (500, 1000, 2000) result in negligible gains. The optimal size is then typically pegged at around 250, balancing computational cost and estimator reliability (Keller et al., 2018).

In energy-limited or resource-constrained deployments (e.g., embedded devices, real-time streaming), larger ensembles may fragment available resources to the point where each component model is too simple, leading to underfitting. Adaptive algorithms that dynamically tune the number of models (by monitoring differences between prequential and postquential accuracy for on-data points) can track an ensemble size that achieves up to 95% of the optimal fixed performance under memory constraints (Khannouz et al., 2022).

Meta-learning approaches extend this adaptivity, using statistical properties of inputs (e.g., 390 time-series features) to predict not only which models to include but also the ideal ensemble size for each instance, outperforming static benchmarks (Vaiciukynas et al., 2020).

4. Trade-offs and New Metrics: Energy, Interpretability, and Complexity

With increasing awareness of energy efficiency and sustainability ("Green AI"), the cost of additional ensemble members goes beyond computational time to include energy and carbon footprint. Studies show that, for typical ML classification workloads,

• Going from a 2- to a 3-model ensemble increases energy by ~37.5%, though accuracy remains statistically unchanged; similarly, 3-to-4 increases energy by ~27% (Omar et al., 3 Jul 2024).
• Majority voting fusion (as opposed to meta-model fusion) yields both higher accuracy and markedly lower energy consumption.
• Subset-based (data-split) training lowers energy cost further, recommending ensemble sizes of 2 (or at most 3) for resource- and energy-conscious applications (Omar et al., 3 Jul 2024).

For interpretability, optimized rule ensemble methods (e.g., Forest-ORE) extract minimal, high-performing rule sets from random forests via mixed-integer programming, allowing the user to balance the rule set size (i.e., ensemble size) against slight losses in global accuracy while maximizing coverage and readability (Maissae et al., 26 Mar 2024).

5. Diversity, Independence, and Statistical Structure

A recurring theme is the need for diversity among the component models as a precondition for ensemble benefit:

• The geometric and linear independence frameworks provide a basis for determining the minimal ensemble size needed to achieve, with high probability, a set of linearly independent vote vectors. This is typically equivalent to the number of classes $m$, but the actual required number may be higher in settings with dependent or correlated classifiers (Bektas et al., 2023).
• Real datasets may diverge from theoretical predictions, with true optimality determined by additional factors like noise heterogeneity, variance across learners, and data nonstationarity—implying that diversity indices and independence criteria must be empirically verified.

6. Advanced Statistical and Application-Specific Results

Task-specific metrics also play a decisive role. In high-dimensional Kalman inversion for inverse problems, the "subspace property" of classical methods means that ensemble size must be on the order of the parameter space dimensionality for convergence. Dropout regularization has been shown to mitigate this issue, realizing convergence and favorable query complexity scaling with much smaller ensembles (Liu et al., 2023).

In probabilistic (uncertainty-aware) settings, such as multivariate forecast evaluation, the sample logarithmic score is biased by finite ensemble size. Correction formulas are derived (e.g., involving the multivariate digamma function and ensemble size $n$) so that “fair” logarithmic scores are nearly independent of $n$, thus decoupling performance assessment from brute-force scaling and enabling fair comparison between ensembles of very different sizes (Leutbecher et al., 22 May 2024).

In reinforcement learning, particularly offline RL, ensemble size interacts with policy constraints: large ensembles sharpen uncertainty penalties for out-of-distribution actions, but much smaller ensembles suffice when policy deviation is regularized (e.g., via behavioral cloning penalties), leading to strong performance with lower computational load (Beeson et al., 2023).

7. Recent Directions: Adaptive and Fine-Grained Control

Contemporary research increasingly focuses on dynamic, adaptive methods that assign model involvement on a per-instance basis:

• Sequential inference methods learn an optimal halting rule, so that for "easy" samples, only a small number of ensemble members are activated, achieving up to 56% reduction in inference cost relative to fixed, full-ensemble strategies without degrading accuracy (Li et al., 2023).
• Fine-grained margin-maximizing ensembles with learned, category-specific confidence matrices have demonstrated the ability to match or outperform classic ensembles (such as random forests with 100 trees) while using only 10% as many base learners, illustrating that optimal ensemble size can be drastically compressed through targeted optimization and adaptive weighting (Yuan et al., 19 Sep 2024).

---

Optimal ensemble size is thus a dynamic function of theoretical data geometry, interaction between component models, application requirements, computational or energy constraints, and design objectives such as interpretability. While theoretical analyses often highlight minima at the number of classes or where diversity and independence are fully exploited, practical optimality is typically validated through empirical, data-driven, or optimization-based refinement, frequently subject to deployment-specific constraints and objectives.