Ensemble Learning-Based Algorithms

• Ensemble learning-based algorithms are designs that combine predictions from multiple models using various aggregation strategies to improve accuracy, robustness, and uncertainty quantification.
• They leverage both fixed and adaptive weighting schemes—such as bagging, stacking, and probabilistic calibration—to manage diversity and optimize performance.
• These methods are applied in domains like image segmentation, disease diagnosis, and time series forecasting, balancing bias, variance, and model complementarity.

Ensemble learning-based algorithms are learning procedures that build multiple predictive models and combine their outputs into a single predictor, typically to improve predictive performance, robustness, stability, or uncertainty quantification. In the contemporary arXiv literature, the term covers simple averaging, majority voting, weighted voting, stacking, adaptive probabilistic weighting, online prediction with expert advice, federated consensus, and task-specific hybrids for multi-instance learning, segmentation, time series, software testing, and reinforcement learning (Mendes-Moreira et al., 2024, Fazla et al., 2022, Raza et al., 2022).

1. Definition and formal structure

At its most basic, an ensemble combines base predictions $f_1(x),\dots,f_M(x)$ through an aggregation rule. A standard linear form in forecasting is

$$\hat{y}_t^E = \sum_{i=1}^M w_t^{(i)} \hat{y}_t^{(i)},$$

where $\hat{y}_t^{(i)}$ is the $i$-th base prediction and $w_t^{(i)}$ is its weight (Fazla et al., 2022). In regression with simple averaging, the ensemble predictor is

$f_{\text{ens}}(x)=\frac{1}{M}\sum_{m=1}^M f_m(x),$

and the expected risk can be analyzed through bias, variance, and diversity (Mendes-Moreira et al., 2024).

The supplied literature shows that ensemble weights need not be fixed. In adaptive probabilistic ensembles, the weight of model $k$ at input $x$ can be defined as

$$u(f_k,x)=\frac{\exp(g_k(x)/\lambda)}{\sum_{j=1}^K \exp(g_j(x)/\lambda)},$$

with $g_k$ modeled by Gaussian processes and $$\hat{y}_t^E = \sum_{i=1}^M w_t^{(i)} \hat{y}_t^{(i)},$$0 controlling sparsity (Liu et al., 2018, Liu et al., 2019). In this formulation, model selection becomes input-dependent, and uncertainty is attached both to the weights and to the final prediction.

The same general pattern appears in classification, though the aggregation operator may differ. In multi-class group-decision aggregation,

$$\hat{y}_t^E = \sum_{i=1}^M w_t^{(i)} \hat{y}_t^{(i)},$$1

where $$\hat{y}_t^E = \sum_{i=1}^M w_t^{(i)} \hat{y}_t^{(i)},$$2 is the decision matrix or rating vector of learner $$\hat{y}_t^E = \sum_{i=1}^M w_t^{(i)} \hat{y}_t^{(i)},$$3, and $$\hat{y}_t^E = \sum_{i=1}^M w_t^{(i)} \hat{y}_t^{(i)},$$4 is a weight derived from precision, recall, and accuracy (He et al., 2020). In stacked multi-instance learning, first-level predictions $$\hat{y}_t^E = \sum_{i=1}^M w_t^{(i)} \hat{y}_t^{(i)},$$5 are passed to a meta-classifier $$\hat{y}_t^E = \sum_{i=1}^M w_t^{(i)} \hat{y}_t^{(i)},$$6,

$$\hat{y}_t^E = \sum_{i=1}^M w_t^{(i)} \hat{y}_t^{(i)},$$7

so the ensemble is explicitly two-level rather than a direct vote (Sundararajan et al., 2014).

A recurring distinction is therefore between fixed and adaptive ensembles, and between deterministic and probabilistic ensembles. This suggests that “ensemble learning-based algorithm” is best understood as a design space defined by how diversity is generated, how predictions are integrated, and whether uncertainty in the integration itself is modeled.

2. Mechanisms for generating diversity

A central principle in the literature is that ensemble performance depends on obtaining base learners that are not merely accurate, but complementary. The simplest mechanisms are the classical ones: bagging, random subspace, boosting, and stacking (Nugroho et al., 2020, Mendes-Moreira et al., 2024). Yet the surveyed papers show a much broader repertoire.

One route is feature or representation perturbation. In streaming data, RPNB builds an online homogeneous ensemble of Naïve Bayes classifiers by projecting each incoming data chunk into multiple low-dimensional spaces using different random matrices $$\hat{y}_t^E = \sum_{i=1}^M w_t^{(i)} \hat{y}_t^{(i)},$$8, then combining the resulting classifiers with the Sum rule (Nguyen et al., 2017). In coronary heart disease diagnosis, Random Subspace perturbs the feature set seen by each learner, while Bagging perturbs the instances; both are then combined with cascade generalization (Nugroho et al., 2020).

A second route is parameter and hyperparameter diversification. In multi-instance learning with Citation Nearest Neighbour classifiers, diversity is created by optimizing the parameter vector

$$\hat{y}_t^E = \sum_{i=1}^M w_t^{(i)} \hat{y}_t^{(i)},$$9

through NSGA-II, with objectives based on class-wise accuracies obtained by leave-one-out validation (Sundararajan et al., 2014). In regression with neural networks, seven simple strategies—Bagging, Pasting, Random Subspace, Dropout, Snapshot, Negative Correlation Learning, and Stacking—are profiled through a bias–variance–diversity decomposition, then combined pairwise to generate 21 new ensemble algorithms (Mendes-Moreira et al., 2024).

A third route is data partitioning or specialization by regime. In partition-based ensemble learning, the training set is divided into $\hat{y}_t^{(i)}$0 disjoint subsets $\hat{y}_t^{(i)}$1, one SVM is trained per subset, and the partition itself is optimized as a non-binary combinatorial object $\hat{y}_t^{(i)}$2, where $\hat{y}_t^{(i)}$3 assigns instance $\hat{y}_t^{(i)}$4 to a partition (Dushatskiy et al., 2021). In speech dereverberation, multiple HDDAE models are trained for different reverberation conditions, and their outputs are later fused by a CNN (Lee et al., 2018).

A fourth route is uncertainty- or confidence-driven diversity management. For polyp localization, the ensemble does not blindly average all segmentation models; it uses Shannon entropy

$\hat{y}_t^{(i)}$5

to decide whether model $\hat{y}_t^{(i)}$6 should contribute at pixel $\hat{y}_t^{(i)}$7 (Dang et al., 2021). In the tensor-optimization framework, diversity is formalized through a confidence tensor $\hat{y}_t^{(i)}$8, where $\hat{y}_t^{(i)}$9 measures how the $i$0-th classifier predicts class $i$1 when the true class is $i$2 (Yuan et al., 2024).

3. Aggregation architectures and weighting schemes

The integration stage ranges from simple hard voting to highly structured meta-modeling. Majority voting remains a baseline. In AMP prediction, binary outputs from SVM, RF, and GBM are mapped to $i$3, summed as

$i$4

and interpreted as “Strong Positive,” “Positive,” “Negative,” or “Strong Negative” depending on whether $i$5 (Zarayeneh et al., 2020). In CRWM, prediction is also expert-based, but the experts are arranged in a cascade of Randomized Weighted Majority learners specialized for different output regions (Zamani et al., 2014).

Weighted voting introduces explicit competence modeling. In the group-decision-making formulation, each base learner $i$6 receives a per-class performance vector $i$7, with $i$8, $i$9, and $w_t^{(i)}$0 computed through One-vs-Rest confusion matrices, and the final class is selected by weighted summation (He et al., 2020). In context-aware time-series ensembling, the weights are not learned from base predictions themselves, but from the union of the base models’ feature vectors, so the meta learner outputs context-dependent $w_t^{(i)}$1 under unconstrained, affine, or convex constraints (Fazla et al., 2022).

Stacking generalizes these weighted rules by learning a second-level predictor. In the stacked ensemble of lazy learners for multi-instance learning, the first layer is a set of Citation Nearest Neighbour classifiers, while the second layer is an SVM with RBF kernel trained on the vector of leave-one-out predictions from the base learners (Sundararajan et al., 2014). In the integrated deep and ensemble learning algorithm for dereverberation, a CNN takes the concatenated outputs of several HDDAE specialists and produces the final clean log-power spectrum (Lee et al., 2018).

Adaptive gating is another important architecture. For polyp segmentation, the final class probability at each pixel is

$w_t^{(i)}$2

so only models with entropy below their learned threshold contribute (Dang et al., 2021). In federated learning, PoSw forms consensus from distributed predictions $w_t^{(i)}$3 by first counting votes, then resolving ties through confidence sums

$w_t^{(i)}$4

thereby turning ensemble prediction into a distributed consensus process rather than a centralized meta-model (Raza et al., 2022).

Probabilistic aggregation goes further by modeling the entire predictive distribution. In adaptive and calibrated ensemble learning, the final predictive CDF is written as

$w_t^{(i)}$5

where $w_t^{(i)}$6 is the ensemble’s uncalibrated predictive CDF and $w_t^{(i)}$7 is a monotonic Gaussian-process link that calibrates it (Liu et al., 2018, Liu et al., 2019).

4. Optimization objectives and theoretical guarantees

The surveyed algorithms optimize markedly different objectives. In multi-instance learning, NSGA-II searches for Pareto-optimal CNN parameter sets maximizing $w_t^{(i)}$8 and $w_t^{(i)}$9, using leave-one-out cross-validation (Sundararajan et al., 2014). In confidence-based polyp localization, Comprehensive Learning Particle Swarm Optimization searches over threshold vectors $f_{\text{ens}}(x)=\frac{1}{M}\sum_{m=1}^M f_m(x),$0 to maximize the average Dice coefficient under constraints $f_{\text{ens}}(x)=\frac{1}{M}\sum_{m=1}^M f_m(x),$1 (Dang et al., 2021). In partition-based ensembles, surrogate-assisted GOMEA optimizes validation accuracy over the combinatorial search space $f_{\text{ens}}(x)=\frac{1}{M}\sum_{m=1}^M f_m(x),$2 while economizing on expensive SVM training evaluations (Dushatskiy et al., 2021).

Other works formulate explicit structural constraints. In the tensor-optimization method, the strong learner uses a parameter matrix $f_{\text{ens}}(x)=\frac{1}{M}\sum_{m=1}^M f_m(x),$3 subject to

$f_{\text{ens}}(x)=\frac{1}{M}\sum_{m=1}^M f_m(x),$4

where $f_{\text{ens}}(x)=\frac{1}{M}\sum_{m=1}^M f_m(x),$5 repeats each base classifier accuracy $f_{\text{ens}}(x)=\frac{1}{M}\sum_{m=1}^M f_m(x),$6 $f_{\text{ens}}(x)=\frac{1}{M}\sum_{m=1}^M f_m(x),$7 times. A key theorem states that the sum of each gradient column is zero, so gradient descent preserves the constraint automatically (Yuan et al., 2024). In context-aware time-series ensembling, the convex and affine constraint spaces are integrated directly into the learning procedure of the meta learner, which amounts to using the feasible set itself as a form of regularization (Fazla et al., 2022).

Several papers supply explicit performance or convergence guarantees. For CRWM, if $f_{\text{ens}}(x)=\frac{1}{M}\sum_{m=1}^M f_m(x),$8 denotes the number of mistakes of the best expert in region $f_{\text{ens}}(x)=\frac{1}{M}\sum_{m=1}^M f_m(x),$9, the expected number of mistakes satisfies

$k$0

and the paper argues that the bound is better than RWM’s for sufficiently large datasets when the globally best expert is not the best one in each region (Zamani et al., 2014). For PoSw, Theorem 1 states that the consensus procedure always converges after at most $k$1 rounds, and a corollary permits early stopping once a simple majority is obtained (Raza et al., 2022). For hierarchical ensemble reinforcement learning, the multi-step integration rule

$k$2

is shown to be stable under stated conditions on $k$3, and to reduce the spread between base learners and the ensemble in the linear-policy case (Chen et al., 2022).

The systematic-design literature makes the bias–variance–diversity trade-off explicit. For regression ensembles of neural networks, the expected risk is analyzed as average bias plus average variance minus diversity, and the resulting decomposition is used as a practical design criterion rather than a purely descriptive identity (Mendes-Moreira et al., 2024).

5. Representative realizations and empirical behavior

The application range in the supplied literature is unusually broad, which underlines that “ensemble learning-based algorithm” is a methodological category rather than a single architecture.

Domain	Representative mechanism	Reported outcome
Multi-instance learning	Stacked Citation Nearest Neighbour ensemble with SVM meta-learner (Sundararajan et al., 2014)	On Musk1, sample stacked solutions reached 100% Class 0 and 93.61% Class 1, or 97.78% Class 0 and 100% Class 1
Polyp segmentation	Entropy-gated averaging with CLPSO-optimized thresholds (Dang et al., 2021)	Dice 0.724 on MICCAI2015 and 0.894 on Kvasir-SEG
Coronary heart disease diagnosis	Bagging or Random Subspace applied to cascade generalization (Nugroho et al., 2020)	Bagging–Cascade C4.5 achieved 83.58% accuracy
AMP prediction	Hard-voting ensemble of SVM, RF, and GBM (Zarayeneh et al., 2020)	Accuracy 0.87, F1-score 0.86, Recall 0.86
Federated ECG classification	Confidence-aware consensus via PoSw (Raza et al., 2022)	PoSw achieved 89% vs local models at 84%–88% and global model at 86%
Speech dereverberation	Multiple HDDAE specialists fused by CNN (Lee et al., 2018)	IDEA$k$4 outperformed single HDDAE baselines on PESQ, STOI, and SDI
Software testing	ELBT with diversity-driven test selection (Rahman et al., 2024)	All ensemble-based test suites killed far more mutants than random test suites

The same pattern extends to online learning, community detection, and reinforcement learning. RPNB projects streaming data to multiple low-dimensional spaces and updates Naïve Bayes models online only on misclassified observations (Nguyen et al., 2017). GAEL replaces ordinary genetic crossover with an ensemble-learning-based multi-individual crossover built from edge join strength $k$5 (He et al., 2013). HED in continuous-control reinforcement learning uses an ensemble of actor–critic learners plus a global ensemble critic, then performs multi-step parameter integration to promote inter-learner collaboration (Chen et al., 2022).

These examples indicate that the phrase can refer to prediction algorithms, uncertainty-calibrated probabilistic models, evolutionary search procedures, online expert-advice systems, and even learning-based testing methods, provided that multiple learned components are combined to produce a stronger final decision.

6. Trade-offs, misconceptions, and design directions

A persistent misconception is that ensemble learning-based algorithms are synonymous with bagging or static majority vote. The supplied literature contradicts this directly. Some ensembles are stackers (Sundararajan et al., 2014), some are adaptive weighted combinations over context (Fazla et al., 2022), some are confidence-gated selectors (Dang et al., 2021), some are distributed consensus protocols (Raza et al., 2022), and some are fully probabilistic calibration systems with feature-dependent random weights (Liu et al., 2018, Liu et al., 2019).

A second misconception is that maximizing diversity alone is sufficient. The decomposition-based design results show that diversity enters risk with a favorable sign, but only in relation to bias and variance (Mendes-Moreira et al., 2024). The coronary heart disease study makes the same point empirically from another angle: applying cascade generalization increased the accuracy of the classifiers in the ensemble but decreased the diversity, yet the overall ensemble improved (Nugroho et al., 2020). This suggests that diversity is beneficial only when managed together with base accuracy and, in probabilistic settings, calibration.

The principal limitations recur across papers. Computational cost is repeatedly identified: pairwise bag distances and leave-one-out validation in multi-instance learning (Sundararajan et al., 2014), CLPSO over all pixels in segmentation (Dang et al., 2021), surrogate training for expensive partition search (Dushatskiy et al., 2021), and large ensembles or calibration GPs in probabilistic model averaging (Liu et al., 2018, Liu et al., 2019). Scalability, interpretability, and overfitting also recur. Context-aware unconstrained ensembles can overfit in real time-series data, whereas affine and convex constraints act as regularizers (Fazla et al., 2022). Confidence-based gating depends on reliable probability estimates (Dang et al., 2021, Raza et al., 2022). Stacked or cascade architectures often improve accuracy but complicate mechanistic interpretation (Sundararajan et al., 2014, Nugroho et al., 2020).

The dominant forward direction is systematic rather than ad hoc design. The literature points toward a workflow in which one first characterizes candidate strategies by how they affect bias, variance, diversity, specialization, or calibration, and then composes them into hybrids matched to the task (Mendes-Moreira et al., 2024). In that sense, the modern ensemble learning-based algorithm is no longer merely “many models plus a vote”; it is an explicitly engineered system for managing complementarity, uncertainty, and decision structure across heterogeneous predictive components.