What is Ensembling in Machine Learning?

• Ensembling refers to constructing a composite predictor or decision rule from multiple constituent models, leveraging their diversity in initialization, architecture, or data for potentially superior combined performance.
• Ensembles may be evaluated as single operations, and are used in prediction ensembling, parameter representation combining, stacking, policy aggregation, core sampling optimization, and composite ensembling.
• Ensemble methods have been satisfactorily used for various scenarios such as classification, forecasting, generation, representation learning, watermarking and evaluation.

Ensembling is the construction of a predictor, representation, decision rule, forecast, or composite object from multiple constituent models or informational elements. In machine learning, ensemble members may differ through initialization, stochastic optimization, architecture, hyperparameters, training data, pre-training history, vocabulary, or model family; their outputs may be aggregated by voting, averaging, stacking, constrained selection, residual composition, or probabilistic inference. The principal rationale is that partially different errors can cancel, although the benefit depends on the relation between member competence, disagreement, bias, variance, computational cost, and the downstream objective. The term is also used more broadly for assembling datasets, studies, expert judgments, model outputs, or multiple watermark payloads (Bleichrodt et al., 2024, Globus-Harris et al., 2024).

1. Conceptual scope and taxonomy

In predictive machine learning, an ensemble combines several constituent predictors into one operational output. The constituents may be independently trained copies of one architecture, heterogeneous architectures, models fine-tuned from different pre-trained representations, or candidate predictors generated along one optimization trajectory. The resulting system can be evaluated as a single classifier, regressor, forecaster, language generator, representation, or policy.

A useful distinction concerns the object being combined:

• Prediction ensembling combines probabilities, logits, scores, forecasts, or hard decisions after models have produced them.
• Parameter or representation ensembling combines model parameters, learned features, or internal representations.
• Stacking trains a second-level learner on predictions from base models.
• Policy or decision ensembling combines policies or predictions whose outputs are subsequently passed to a constrained optimization problem.
• Composite ensembling combines multiple payloads or outputs without reducing them to one prediction, as in watermark coexistence.

The broader cross-disciplinary usage includes assemblies of data, framework analysis, and statistical or mathematical modeling. Data may be combined into datasets or repositories; studies may be synthesized through systematic reviews, meta-analyses, or Delphi procedures; and multiple statistical, mechanistic, or hybrid models may be combined into forecasts or conclusions. These categories overlap rather than forming a mutually exclusive taxonomy (Globus-Harris et al., 2024).

Ensembling should not be identified solely with averaging. A watermark ensemble, for example, applies multiple encoders to the same image and decodes the resulting messages independently, while an ensemble for constrained optimization may select among policies according to calibrated downstream utility. In language generation, an ensemble may target a distribution over complete strings rather than average next-token distributions. These cases share the combination of multiple elements but differ in their aggregation object and objective.

2. Statistical mechanisms and aggregation rules

For classifiers, a conventional probability ensemble averages member predictions. If model $i$ produces $p_i(y\mid x)$, an arithmetic ensemble has the form

$$p_{\mathrm{ens}}(y\mid x)=\sum_i\lambda_i p_i(y\mid x), \qquad \lambda_i\geq 0,\quad \sum_i\lambda_i=1.$$

The predicted class is the maximum-probability class. Hard voting instead selects the class receiving the largest number of member votes. Intent classification experiments have used a procedural majority-with-confidence rule: a class receiving a majority is selected; otherwise, the prediction with the highest constituent confidence is used (Costello et al., 2018).

Averaging is variance-reducing when member errors are not perfectly correlated. For error variance $\sigma^2$, ensemble size $n$, and pairwise error correlation $\rho$, a simplified variance expression is

$$\operatorname{Var}(\bar e) = \sigma^2\left(\frac{1-\rho}{n}+\rho\right).$$

The expression emphasizes that increasing the number of members is useful only to the extent that their errors are not perfectly correlated. Empirical work on classification formalizes this relationship through the disagreement-error ratio,

$$\operatorname{DER} = \frac{\text{expected pairwise disagreement}} {\text{average individual error}},$$

and the ensemble improvement rate, which measures the relative reduction from average constituent error to majority-vote error. Under a competence condition, the majority vote cannot be worse than the average constituent classifier, and its improvement is bounded in terms of DER (Theisen et al., 2023).

For nonsymmetric classification losses such as cross-entropy, the classical regression statement that averaging reduces variance while preserving bias requires qualification. In the Bregman-divergence formulation, dual averaging preserves bias and reduces variance, whereas ordinary probability averaging can either increase or decrease bias. Empirically, ordinary probability ensembles often reduce bias and obtain lower negative log-likelihood than dual ensembles, although this bias reduction is not guaranteed by the general theory (Gupta et al., 2022).

Aggregation can also be performed in spaces other than probability space. Geometric averaging of softmax predictions is slightly better than arithmetic averaging in some image-classification experiments. For LLMs, global string-level ensembles may use arithmetic, geometric, harmonic, minimum, maximum, or generalized-mean aggregation. Product and minimum operators are consensus-seeking, whereas sum and maximum operators are coverage-seeking (Chan et al., 5 Mar 2026).

The choice of aggregation is therefore task-dependent. A fixed average is computationally simple and robust, but it may dilute a strong model with weaker members. Greedy selection, learned weights, nonlinear stackers, or context-dependent routing can exploit specialization. Conversely, flexible aggregators can overfit validation data or amplify poorly calibrated members.

3. Sources of diversity and ensemble construction

Diversity may be introduced at several stages. Initialization diversity trains multiple copies of the same architecture with different random weights. Optimization diversity arises from different mini-batch orders, augmentation realizations, or stochastic trajectories. Architectural diversity combines CNNs and RNNs, recurrent cell types, bidirectionality, attention, character-level and word-level representations, or different model scales. Upstream diversity uses pre-trained models trained with different seeds, data subsets, architectures, or upstream datasets.

Multi-layer ensembling combines these mechanisms hierarchically. In multilingual intent classification, first-layer ensembles combine three independently initialized copies of each architecture. Second-layer ensembles combine the resulting first-layer ensembles, thereby combining initialization and architectural diversity. The experiments considered 12 model types and $2^{12}-12-1=4083$ second-layer combinations (Costello et al., 2018).

End-to-end approaches avoid maintaining independently trained full models. EnsembleNet converts a network into a shared-backbone, multi-headed architecture. A lower portion is shared, an upper portion is duplicated into narrower branches, and an online ensemble prediction supervises the branches through co-distillation. Under a simple average and squared-error loss, the conventional ensemble objective decomposes into ensemble supervision plus an agreement term. On ImageNet, YouTube-8M, and Kinetics, the two-headed models achieved higher accuracy than single baselines while using slightly fewer or comparable parameters and FLOPs (Li et al., 2019).

Ensembling can also be an alternative scaling axis. Experiments on CIFAR-10 found that eight Wide ResNet 16-2 models achieved approximately $95\%$ accuracy at about 80 million FLOPs, comparable to a single Wide ResNet 16-8 at about 150 million FLOPs. On ImageNet, two EfficientNet-b3 models achieved higher accuracy with fewer total FLOPs than one EfficientNet-b4. The advantage became more apparent as individual models grew larger and marginal gains from further scaling diminished (Kondratyuk et al., 2020).

In low-data transfer learning, diversity is often more effective when introduced before fine-tuning. A practical procedure ranks a library of pre-trained models by leave-one-out $k$-nearest-neighbour accuracy, fine-tunes the top candidates with a small hyperparameter sweep, and greedily selects an ensemble by validation cross-entropy. In the reported VTAB-1K experiments, upstream diversity from pre-training outperformed diversity generated only by random seeds, augmentations, or fine-tuning hyperparameters (Mustafa et al., 2020).

Snapshot and training-time ensembles reduce repeated training cost. A cyclical learning-rate schedule produces candidate models along one optimization path; training-time likelihoods assign nonuniform weights to snapshots. On CIFAR-10, the strongest reported weighted snapshot result reached $90.8\%$ accuracy, compared with $89.9\%$ for equal-weight minimum-learning-rate snapshots and $91.9\%$ for a five-model independently trained ensemble (Proscura et al., 2022).

Ensembling can be applied to learned representations. Sparse autoencoder ensembles either train independently initialized SAEs and average their reconstructions through naive bagging, or train sequentially on reconstruction residuals through boosting. Bagging improves reconstruction and stability but can produce redundant features; boosting produces lower reconstruction error, higher feature-direction diversity, and greater connectivity, while being more computationally expensive and less stable across runs (Gadgil et al., 21 May 2025).

4. Stacking, adaptive selection, and structured ensembles

Stacking trains a second-level learner on out-of-sample predictions from base models. For a family of stackers $A_\alpha$, the second-level model maps base predictions to a new prediction, and validation or cross-validation selects among candidate stacking strategies. A theoretical oracle inequality shows that validation-selected stacking has expected risk controlled by the best candidate in a finite or finite-dimensional family, subject to Bernstein-type moment and Lipschitz assumptions (Hasson et al., 2023).

Time-series stacking requires temporal rather than random validation. Base models are trained on earlier observations and evaluated on subsequent windows, producing leakage-free out-of-fold forecasts. A structured ensemble may allow weights to vary across series, forecast horizon, and quantile, with regularization controlling how much sharing is imposed across these dimensions (Hasson et al., 2023).

A multi-layer stack ensemble extends this construction:

$$\text{base forecasters} \longrightarrow \text{heterogeneous stackers} \longrightarrow \text{stacker aggregator}.$$

The first layer contains heterogeneous forecasters; the second layer contains means, medians, greedy combinations, linear stackers, LightGBM, and RealMLP; the third layer combines the second-layer outputs. Across 50 real-world datasets, the multi-layer method achieved the best representative Elo rating, average rank, champion count, and relative error for both probabilistic and point forecasting. It won 20 of 50 probabilistic tasks and 17 of 50 point tasks in the reported comparisons (Bosch et al., 19 Nov 2025).

Selection can also be adaptive to downstream decisions. In constrained optimization, models predict multidimensional coefficients of a linear objective. A white-box method calibrates constituent predictions conditional on the event that a model wins a self-assessed-payoff comparison, then selects the apparently best model state by state. A black-box method receives only policies and trains a new predictor calibrated on the policies’ action-level sets. The white-box method has a pointwise-maximum self-assessment guarantee, whereas the black-box method has a conditional swap-style utility guarantee (Globus-Harris et al., 2024).

For long-form language generation, dense next-token ensembling can be harmful because tokenizer mismatch exposes models to unnatural intermediate prefixes and allows errors to propagate. SAFE uses a drafter-verifier generate–verify–ensemble procedure: it generates blocks of drafter tokens, checks tokenizer-boundary safety and probability consensus, and ensembles only at positions where aggregation is both safe and necessary. In reported experiments, SAFE often used the ensemble on fewer than $20\%$ of tokens and frequently fewer than $5\%$ on mathematical tasks (Yun et al., 17 Oct 2025).

Agreement-Based Ensembling addresses a different vocabulary problem by requiring models with different tokenizers to agree on detokenized surface strings. Candidate token combinations are explored with cube pruning, and models may temporarily stall while another tokenizer catches up. The method enables token-level combinations of encoder-decoder and decoder-only models without modifying or retraining the models, although performance depends on surface-form compatibility and is primarily demonstrated for machine translation (Wicks et al., 28 Feb 2025).

5. Applications and empirical effects

Ensembling has been applied across classification, forecasting, generation, representation learning, watermarking, and fairness-sensitive evaluation.

In intent classification, multi-layer CNN/RNN ensembles obtained best F1 scores of $97.54\%$ on ATIS, $91.79\%$ on Banking, and $93.55\%$ on SMP2017. The reported total improvements over the best individual models were $0.23\%$, $1.96\%$, and $4.04\%$, respectively. Character representations were particularly beneficial on the Chinese SMP dataset, while the strongest configurations were dataset-dependent (Costello et al., 2018).

In fine-tuned language-model text classification, a metadataset containing predictions from 1,134 configurations of GPT-2, BERT-Large, BART-Large, ALBERT-Large, and T5-Large was used to compare random, top-ranked, model-average, and greedy ensembles. Greedy-50 achieved the best reported negative log-likelihood on every dataset and split, although greedy-5 often provided a more economical classification-error configuration. Random selection was unreliable and could be substantially worse than Single-Best (Arango et al., 2024).

In fairness-oriented evaluation, homogeneous ensembles—identical architecture, training data, objective, and nominal hyperparameters—disproportionately improved bottom-performing groups. On CIFAR-100 and TinyImageNet, bottom-$k$ classes gained more than top-$k$ classes; bottom-group accuracy continued to improve after mean accuracy plateaued. On CelebA, overall blonde-classification accuracy rose from $92.02\%$ to $94.04\%$, while the reported blonde-male subgroup accuracy rose from $9.44\%$ to $21.80\%$. The proposed mechanism is subgroup-specific disagreement: difficult examples exhibit more model churn, allowing averaging to correct errors that are not shared by all members (Ko et al., 2023).

Watermark ensembling composes independently decodable messages rather than combining predictions. If methods with nominal capacities $m_1,\ldots,m_K$ coexist, the nominal capacity is $\sum_k m_k$. Series composition generally gives stronger signals but lower image quality and is order-dependent; parallel residual composition is order-independent and usually has higher PSNR but weaker decoding. Error-correcting codes and strength clipping can trade payload capacity against accuracy, robustness, and image quality (Petrov et al., 29 Jan 2025).

SAE ensembles improve activation reconstruction and broaden feature coverage. Eight-SAE boosting ensembles achieved explained variance of $0.961$, $0.998$, and $0.995$ for the GELU-1L, Pythia-160M, and Gemma 2-2B settings, respectively. Naive bagging generally produced greater stability, while boosting produced higher diversity and connectivity. Downstream performance depended on the feature-selection task: bagging was often better when only one broad concept feature was selected, whereas boosting was stronger for spurious-correlation removal (Gadgil et al., 21 May 2025).

Ensembling can also improve algorithmic stability. A general framework represents the ensemble as a Markov channel $Q$ applied to the input, with the ideal ensemble output

$A_Q(z)=\mathbb E_{W\sim Q(\cdot\mid z)}[A(W)].$

For bounded scalar algorithms, stability under a perturbation distribution $P$ is bounded by one quarter of the $L_\infty\to L_1$ norm of a covariance operator determined by $P$ and $Q$. The resulting bagging bound decreases at order $1/n$ under exchangeability, and the framework also covers additive noise, missingness, bootstrap aggregation, and subbagging (Barber et al., 9 Sep 2026).

6. Limitations, trade-offs, and unresolved issues

Ensembling does not universally improve performance. The strongest theoretical classification guarantees require competence conditions; without them, a majority vote can be worse than the average constituent classifier. Even under competence, disagreement must be interpreted relative to average error. High raw disagreement among inaccurate models is not necessarily useful, while low disagreement among highly accurate models can imply that little remains to be corrected (Theisen et al., 2023).

The principal cost is computational. Independently trained ensembles multiply training, storage, and inference requirements. Parallel execution can reduce wall-clock latency, but total FLOPs, aggregate memory, energy, and orchestration requirements remain approximately additive. Shared-backbone multi-headed models and snapshot methods reduce training or storage costs but do not eliminate all inference overhead (Li et al., 2019, 2005.16077).

Model selection introduces statistical risk. Searching thousands of ensemble combinations, repeatedly using small validation sets, or selecting among many stackers can overfit validation data. Cross-validation, temporal backtesting, held-out evaluation, and validation cross-entropy can reduce but do not eliminate this risk. Theoretical oracle bounds generally require assumptions that may be difficult to verify for modern neural networks, and time-series theorems established under i.i.d. sampling do not directly apply to dependent rolling-origin evaluation (Mustafa et al., 2020, Hasson et al., 2023).

Diversity is not synonymous with complementarity. Architecturally diverse ensembles may fail to outperform repeated copies of a strong architecture, as observed in the reported neural architecture search experiments. Conversely, weak models can be useful in a larger stack if they specialize in regimes where stronger average performers fail. The relevant property is therefore task-conditional error or utility complementarity, not diversity measured in isolation.

Aggregation can introduce representation-specific pathologies. Arithmetic probability averaging can alter classifier bias; token-level language-model averaging can be destabilized by incompatible tokenization; global string-level sampling can be computationally expensive; watermark composition can overwrite earlier payloads; and SAE boosting can produce overly specific residual features. These effects show that the aggregation space—probabilities, logits, bytes, residuals, policies, or decoded strings—is part of the method rather than an implementation detail.

The evidence is also domain-limited. Results from image classification, intent classification, machine translation, structured generation, forecasting, or watermarking do not establish universal behavior in other tasks. Calibration, distribution shift, adversarial robustness, uncertainty quality, energy use, and deployment latency are often incompletely evaluated. Fairness improvements measured through subgroup accuracy do not imply demographic parity, equalized odds, equal opportunity, calibration parity, or disparate-impact guarantees (Ko et al., 2023).

Several recent approaches therefore treat ensembling as a conditional design problem. The practitioner must determine whether to diversify initialization, architecture, pre-training, data, hyperparameters, or decoding; whether to average, stack, select, distill, calibrate, or compose; and whether the objective concerns accuracy, NLL, worst-group performance, robustness, stability, downstream utility, capacity, or computational efficiency. The general supported principle is that ensembles are most effective when their members are individually competent, make sufficiently different errors or decisions, and are aggregated in a space appropriate to the task.