Parallel Ensembling in Machine Learning

• Parallel ensembling is a method that aggregates outputs from multiple models executed independently to enhance performance across diverse tasks.
• It employs various mechanisms such as uniform probability averaging, token-level sampling, and parameter-space composition to balance accuracy and computational cost.
• Its effectiveness depends on error diversity and communication overhead, with proven applications in language modeling, machine translation, and uncertainty quantification.

Searching arXiv for papers on parallel ensembling and related ensemble formulations. arxiv_search.query{"^^^^^^^^5search_query5 ensembling\" OR ^^^^^^^^5all:\5 Ensemble\" OR ^^^^^^^^5all:\5 Model Composition\" OR ^^^^^^^^5all:\5 parallel collaboration\"","start":^^^^5search_query5^^^^,"max_results":^^^^5all:\5search_query5 arxiv_search.query result: { "results": &&&^^^^5search_query5^^^^&&&)", "title": "Rethinking LLM Ensembling from the Perspective of Mixture Models", "authors": [ "Jiale Fu", "Yifan Wang", "Wenhao Huang", "Xiangru Tang", "Zhilin Yang" ], "abstract": "Model ensembling is a well-established technique for improving the performance of machine learning models. Conventionally, this involves averaging the output distributions of multiple models and selecting the most probable label. This idea has been naturally extended to LLMs, yielding improved performance but incurring substantial computational cost. This inefficiency stems from directly applying conventional ensemble implementation to LLMs, which require a separate forward pass for each model to explicitly compute the ensemble distribution. In this paper, we propose the Mixture-model-like Ensemble (ME). By reinterpreting the ensemble as a mixture model, ME stochastically selects a single model at each step to generate the next token, thereby avoiding the need to explicitly compute the full ensemble distribution. ME is mathematically equivalent to sampling from the ensemble distribution, but requires invoking only one model, making it ^^^^5all:\5^^^^.78x-^^^^5 OR all:\5^^^^.68x faster than conventional ensemble. Furthermore, this perspective connects LLM ensembling and token-level routing methods, suggesting that LLM ensembling is a special case of routing methods. Our findings open new avenues for efficient LLM ensembling and motivate further exploration of token-level routing strategies for LLMs. Our code is available at https://github.com/jialefu/Mixture-model-like-Ensemble/." }, { "id": "(&&&^^^^5all:\5^^^^&&&)", "title": "Ensemble Learning for Heterogeneous LLMs with Deep Parallel Collaboration", "authors": [ "Zhongzhi Yu", "Yufan Li", "Bo Li", "Huiping Zhuang", "Wen Li" ], "abstract": "LLMs exhibit complementary strengths in various tasks, motivating the research of LLM ensembling. However, existing work focuses on training an extra reward model or fusion model to select or combine all candidate answers, posing a great challenge to the generalization on unseen data distributions. Besides, prior methods use textual responses as communication media, ignoring the valuable information in the internal representations. In this work, we propose a training-free ensemble framework DeePEn, fusing the informative probability distributions yielded by different LLMs at each decoding step. Unfortunately, the vocabulary discrepancy between heterogeneous LLMs directly makes averaging the distributions unfeasible due to the token misalignment. To address this challenge, DeePEn maps the probability distribution of each model from its own probability space to a universal relative space based on the relative representation theory, and performs aggregation. Next, we devise a search-based inverse transformation to transform the aggregated result back to the probability space of one of the ensembling LLMs (main model), in order to determine the next token. We conduct extensive experiments on ensembles of different number of LLMs, ensembles of LLMs with different architectures, and ensembles between the LLM and the specialist model. Experimental results show that (i) DeePEn achieves consistent improvements across six benchmarks covering subject examination, reasoning, and knowledge, (ii) a well-performing specialist model can benefit from a less effective LLM through distribution fusion, and (iii) DeePEn has complementary strengths with other ensemble methods such as voting." }, { "id": "(&&&^^^^5 OR all:\5^^^^&&&)", "title": "Private Synthetic Data Meets Ensemble Learning", "authors": [ "Yunpeng Li", "Ishaan Pendurthi", "Jiayao Ding", "Nivedita Poojary", "Dingfan Chen", "Carsten Maple", "Florian Kerschbaum" ], "abstract": "When machine learning models are trained on synthetic data and then deployed on real data, there is often a performance drop due to the distribution shift between synthetic and real data. In this paper, we introduce a new ensemble strategy for training downstream models, with the goal of enhancing their performance when used on real data. We generate multiple synthetic datasets by applying a differential privacy (DP) mechanism several times in parallel and then ensemble the downstream models trained on these datasets. While each synthetic dataset might deviate more from the real data distribution, they collectively increase sample diversity. This may enhance the robustness of downstream models against distribution shifts. Our extensive experiments reveal that while ensembling does not enhance downstream performance (compared with training a single model) for models trained on synthetic data generated by marginal-based or workload-based DP mechanisms, our proposed ensemble strategy does improve the performance for models trained using GAN-based DP mechanisms in terms of both accuracy and calibration of downstream models." }, { "id": "(&&&^^^^5 OR all:\5^^^^&&&)", "title": "Tangent Model Composition for Ensembling and Continual Fine-tuning", "authors": [ "Tianyu Chen", "Dingli Yu", "Tongshuang Shou", "Nan Du", "Shan Jagadeesh", "Xinghao Wang", "Junhong Shen", "Daya Khudia", "Jianyu Zhang", "Ying Xiao", "Hui Miao", "Michael Rabbat", "Yingbin Liang" ], "abstract": "Tangent Model Composition (TMC) is a method to combine component models independently fine-tuned around a pre-trained point. Component models are tangent vectors to the pre-trained model that can be added, scaled, or subtracted to support incremental learning, ensembling, or unlearning. Component models are composed at inference time via scalar combination, reducing the cost of ensembling to that of a single model. TMC improves accuracy by ^^^^5 OR all:\5^^^^.^^^^5 OR all:\5^^^^% compared to ensembling non-linearly fine-tuned models at a ^^^^5 OR all:\5^^^^.5x to ^^^^5all:\5search_query5^^^^x reduction of inference cost, growing linearly with the number of component models. Each component model can be forgotten at zero cost, with no residual effect on the resulting inference. When used for continual fine-tuning, TMC is not constrained by sequential bias and can be executed in parallel on federated data. TMC outperforms recently published continual fine-tuning methods almost uniformly on each setting -- task-incremental, class-incremental, and data-incremental -- on a total of ^^^^5all:\5 OR all:\5^^^^ experiments across ^^^^5 OR all:\5^^^^ benchmark datasets, despite not using any replay buffer. TMC is designed for composing models that are local to a pre-trained embedding, but could be extended to more general settings." }, { "id": "(&&&^^^^5 OR all:\5^^^^&&&)", "title": "Token-level Ensembling of Models with Different Vocabularies", "authors": [ "Antonios Anastasopoulos", "Yue Ma", "Jian Wang", "Philipp Koehn" ], "abstract": "Model ensembling is a technique to combine the predicted distributions of two or more models, often leading to improved robustness and performance. For ensembling in text generation, the next token's probability distribution is derived from a weighted sum of the distributions of each individual model. This requires the underlying models to share the same subword vocabulary, limiting the applicability of ensembling, since many open-sourced models have distinct vocabularies. In research settings, experimentation or upgrades to vocabularies may introduce multiple vocabulary sizes. This paper proposes an inference-time only algorithm that allows for ensembling models with different vocabularies, without the need to learn additional parameters or alter the underlying models. Instead, the algorithm ensures that tokens generated by the ensembled models agree in their surface form. We apply this technique to combinations of traditional encoder-decoder models and decoder-only LLMs and evaluate on machine translation. In addition to expanding to model pairs that were previously incapable of token-level ensembling, our algorithm frequently improves translation performance over either model individually." }, { "id": "(Tanna et al., 18 May 2026)", "title": "Ensembling Tabular Foundation Models - A Diversity Ceiling And A Calibration Trap", "authors": [ "Matteo M. Pasin", "Robert A. Kleinman", "Ehsan Soltani", "Merve Emre", "Maxime Riche", "Amar Saini" ], "abstract": "Tabular foundation models (TFMs) now match or beat tuned gradient-boosted trees on a growing fraction of tabular tasks, but no single TFM wins on every dataset. Ensembling is the go to fix here, and it works less well than expected. Six modern TFMs form a near-redundant pool: their mean pairwise Q-statistic is ^^^^5search_query5^^^^.96^^^^5all:\5 close enough to ^^^^5all:\5^^^^ that any convex combination is bounded above. We benchmark six ensemble strategies over six TFMs on ^^^^5all:\55 OR all:\5^^^^ OpenML classification tasks. The best ensemble, two-level cascade stacking, buys +^^^^5search_query5^^^^.^^^^5all:\5 accuracy over the strongest single TFM at ^^^^5 OR all:\55 OR all:\5^^^^x the compute. A Friedman and Nemenyi analysis places three ensembles and the best base TFM in a single equivalence group; three other ensembles are significantly worse than the best base. Stacking with a logistic-regression meta-learner is the most striking case: competitive accuracy and ROC-AUC, the worst log-loss rank among the ensembles. The meta-learner improves accuracy by sharpening class boundaries, which destroys calibration. We recommend greedy selection as the practical default." }, { "id": "(Hasan et al., 2020)", "title": "Not Low-Resource Anymore: Aligner Ensembling, Batch Filtering, and New Datasets for Bengali-English Machine Translation", "authors": [ "Mushfiqur Rahman", "Anik Paul", "Sajedul Karim", "Shafkat Amin Tonmoy", "Abu Nowshed Chy", "M. Saiful Bari", "Rashedur Rahman", "Shafiq Joty" ], "abstract": "Despite being the seventh most widely spoken language in the world, Bengali has received much less attention in machine translation literature due to being low in resources. Most publicly available parallel corpora for Bengali are not large enough; and have rather poor quality, mostly because of incorrect sentence alignments resulting from erroneous sentence segmentation, and also because of a high volume of noise present in them. In this work, we build a customized sentence segmenter for Bengali and propose two novel methods for parallel corpus creation on low-resource setups: aligner ensembling and batch filtering. With the segmenter and the two methods combined, we compile a high-quality Bengali-English parallel corpus comprising of ^^^^5 OR all:\5^^^^.75 million sentence pairs, more than ^^^^5 OR all:\5^^^^ million of which were not available before. Training on neural models, we achieve an improvement of more than 9 BLEU score over previous approaches to Bengali-English machine translation. We also evaluate on a new test set of ^^^^5all:\5search_query5search_query5search_query5^^^^ pairs made with extensive quality control. We release the segmenter, parallel corpus, and the evaluation set, thus elevating Bengali from its low-resource status. To the best of our knowledge, this is the first ever large scale study on Bengali-English machine translation. We believe our study will pave the way for future research on Bengali-English machine translation as well as other low-resource languages." }, { "id": "(Kondratyuk et al., 2020)", "title": "When Ensembling Smaller Models is More Efficient than Single Large Models", "authors": [ "Tom Hennigan", "Trevor Cai", "Tamara Norman", "Igor Babuschkin" ], "abstract": "Ensembling is a simple and popular technique for boosting evaluation performance by training multiple models (e.g., with different initializations) and aggregating their predictions. This approach is commonly reserved for the largest models, as it is commonly held that increasing the model size provides a more substantial reduction in error than ensembling smaller models. However, we show results from experiments on CIFAR-^^^^5all:\5search_query5^^^^ and ImageNet that ensembles can outperform single models with both higher accuracy and requiring fewer total FLOPs to compute, even when those individual models' weights and hyperparameters are highly optimized. Furthermore, this gap in improvement widens as models become large. This presents an interesting observation that output diversity in ensembling can often be more efficient than training larger models, especially when the models approach the size of what their dataset can foster. Instead of using the common practice of tuning a single large model, one can use ensembles as a more flexible trade-off between a model's inference speed and accuracy. This also potentially eases hardware design, e.g., an easier way to parallelize the model across multiple workers for real-time or distributed inference." }, { "id": "(Pearce et al., 2018)", "title": "Uncertainty in Neural Networks: Approximately Bayesian Ensembling", "authors": [ "Pearce, T.", "Zaki, M.", "Brintrup, A.", "Neely, A." ], "abstract": "Understanding the uncertainty of a neural network's (NN) predictions is essential for many purposes. The Bayesian framework provides a principled approach to this, however applying it to NNs is challenging due to large numbers of parameters and data. Ensembling NNs provides an easily implementable, scalable method for uncertainty quantification, however, it has been criticised for not being Bayesian. This work proposes one modification to the usual process that we argue does result in approximate Bayesian inference; regularising parameters about values drawn from a distribution which can be set equal to the prior. A theoretical analysis of the procedure in a simplified setting suggests the recovered posterior is centred correctly but tends to have an underestimated marginal variance, and overestimated correlation. However, two conditions can lead to exact recovery. We argue that these conditions are partially present in NNs. Empirical evaluations demonstrate it has an advantage over standard ensembling, and is competitive with variational methods." } ] } Parallel ensembling denotes a family of ensemble constructions in which multiple predictors, model components, aligners, or data-generation runs are executed independently and combined through a parallelizable aggregation operator rather than through a single monolithic model. Across recent literature, the combined object may be a class-probability vector, a next-token distribution, a detokenized surface-form hypothesis, a relative-representation vector, a tangent parameter offset, or a union of candidate sentence alignments. The common motivation is to exploit complementary errors while preserving throughput, modularity, or privacy guarantees; the common constraint is that gains depend on diversity, communication overhead, and the geometry of the combination space (Fu et al., 1 May 2026, &&&^^^^5 OR all:\5^^^^&&&, Tanna et al., 18 May 2026).

^^^^5all:\5^^^^. Core formulations

The canonical ensemble rule averages predictive distributions. For classification and next-token prediction, a standard form is

PRESERVED_PLACEHOLDER_^^^^5search_query5^^^^

with PRESERVED_PLACEHOLDER_^^^^5all:\5^^^^ and PRESERVED_PLACEHOLDER_^^^^5 OR all:\5^^^^. Closely related forms include arithmetic averaging of softmax probabilities, geometric-mean aggregation, and majority vote over hard labels. In all cases, the ensemble predictor is defined pointwise from independently computed model outputs, so the statistical object being combined is explicit (Fu et al., 1 May 2026, Kondratyuk et al., 2020).

Parallelism enters through the execution model. In conventional LLM ensembling, sampling each token from PRESERVED_PLACEHOLDER_^^^^5 OR all:\5^^^^ requires a full forward pass through every constituent model at every step, which yields PRESERVED_PLACEHOLDER_^^^^5 OR all:\5^^^^ forward passes per token. In classical vision settings, an ensemble of $k$ identical small models costs $k\,C_s$ FLOPs, whereas a single larger model costs $C_L$; under ideal device-level parallelism with $k\le m$ workers, latency is approximately $t_s+t_{\mathrm{comm}}$, not PRESERVED_PLACEHOLDER_^^^^5all:\5search_query5^^^^, so an ensemble can dominate a larger single model in both accuracy and wall-clock latency when communication overhead is small (Fu et al., 1 May 2026, Kondratyuk et al., 2020).

A second formal family replaces output-space aggregation with parameter-space aggregation. Tangent Model Composition defines tangent components PRESERVED_PLACEHOLDER_^^^^5all:\5all:\5^^^^ around a fixed pretrained point PRESERVED_PLACEHOLDER_^^^^5all:\5 OR all:\5^^^^, then constructs

PRESERVED_PLACEHOLDER_^^^^5all:\5 OR all:\5^^^^

Because the tangent model is linear in PRESERVED_PLACEHOLDER_^^^^5all:\5 OR all:\5^^^^, inference can be reduced to a single composed model rather than an explicit multi-model evaluation loop (&&&^^^^5 OR all:\5^^^^&&&).

These formulations already indicate that “parallel ensembling” is not a single algorithm. It is a design pattern whose instances differ in what is aggregated, when aggregation occurs, and whether the implementation materializes all constituent predictions.

^^^^5 OR all:\5^^^^. Combination loci and implementation regimes

Recent work instantiates parallel ensembling at several distinct loci.

Combination locus	Operator	Representative paper
Output probabilities	Uniform or weighted averaging; vote; geometric mean	(Kondratyuk et al., 2020, &&&^^^^5 OR all:\5^^^^&&&, Tanna et al., 18 May 2026)
Token generation path	Sample model index, then sample token	(Fu et al., 1 May 2026)
Surface-form agreement across vocabularies	Search over agreeing token pairs	(&&&^^^^5 OR all:\5^^^^&&&)
Relative representation space	Arithmetic mean in anchor-based relative space	(&&&^^^^5all:\5^^^^&&&)
Parameter offsets	Linear blend of tangent components	(&&&^^^^5 OR all:\5^^^^&&&)
Candidate alignment sets	Set union followed by semantic filtering	(Hasan et al., 2020)

In prediction-space methods, parallelism is operationally straightforward: each base model produces a probability vector and an ensemble rule combines them. In data-generation methods, the independently computed objects are not predictions but synthetic datasets; downstream models are trained separately and ensembled only at test time. In corpus-construction methods, the combined objects are alignment candidates rather than model scores (&&&^^^^5 OR all:\5^^^^&&&, Hasan et al., 2020).

This heterogeneity has two consequences. First, the same term can describe either parallel training, parallel inference, or both. Second, efficiency claims are not interchangeable: a method that reduces inference-time passes per token is solving a different systems problem from a method that diversifies synthetic data under a fixed privacy budget or a method that unions aligner outputs before semantic filtering. A plausible implication is that comparisons across papers are most meaningful when the aggregation locus is held fixed.

^^^^5 OR all:\5^^^^. Autoregressive language-model ensembling

In autoregressive generation, the main obstacle to parallel ensembling is that the conventional ensemble rule is expensive to materialize token by token. The Mixture-model-like Ensemble reinterprets

PRESERVED_PLACEHOLDER_^^^^5all:\55^^^^

as a two-stage sampler: first sample an index PRESERVED_PLACEHOLDER_^^^^5all:\5^^^^6, then sample PRESERVED_PLACEHOLDER_^^^^5all:\5^^^^7. This is mathematically equivalent to sampling from the ensemble distribution, but it invokes only one model at each step. The method relies on lazy Key-Value cache synchronization: if a dormant model is reactivated after PRESERVED_PLACEHOLDER_^^^^5all:\5^^^^8 steps, it performs a small prefill on the missing tokens to rebuild its cache. On H^^^^5all:\5search_query5search_query5^^^^, reported throughput gains are PRESERVED_PLACEHOLDER_^^^^5all:\5^^^^9 for a two-model sequential baseline and PRESERVED_PLACEHOLDER_^^^^5 OR all:\5search_query5^^^^ for a three-model sequential baseline, with similar-model and heterogeneous-model evaluations on GSM8K, MMLU, BBH, and ARC showing near-parity with conventional ensembling (Fu et al., 1 May 2026).

Parallel ensembling becomes more intricate when constituent LLMs have different vocabularies. Agreement-Based Ensembling addresses this by enforcing agreement in detokenized surface form rather than token identity. Let PRESERVED_PLACEHOLDER_^^^^5 OR all:\5all:\5^^^^ and PRESERVED_PLACEHOLDER_^^^^5 OR all:\5 OR all:\5^^^^ share a common detokenized history PRESERVED_PLACEHOLDER_^^^^5 OR all:\5 OR all:\5^^^^ but have distinct vocabularies PRESERVED_PLACEHOLDER_^^^^5 OR all:\5 OR all:\5^^^^ and PRESERVED_PLACEHOLDER_^^^^5 OR all:\55^^^^. ABE searches the product space PRESERVED_PLACEHOLDER_^^^^5 OR all:\5^^^^6 for token pairs whose detokenized extensions coincide, using cube-pruning over top-PRESERVED_PLACEHOLDER_^^^^5 OR all:\5^^^^7 candidates and a stalling mechanism when one model’s local history is ahead of the other’s. In machine translation, the method yields positive PRESERVED_PLACEHOLDER_^^^^5 OR all:\5^^^^8COMET and PRESERVED_PLACEHOLDER_^^^^5 OR all:\5^^^^9BLEU for same-vocabulary and different-vocabulary pairings, and reported average PRESERVED_PLACEHOLDER_^^^^5 OR all:\5search_query5^^^^BLEU relative to interpolation versus ABE is PRESERVED_PLACEHOLDER_^^^^5 OR all:\5all:\5^^^^ versus PRESERVED_PLACEHOLDER_^^^^5 OR all:\5 OR all:\5^^^^ when both are applicable (&&&^^^^5 OR all:\5^^^^&&&).

DeePEn generalizes heterogeneous LLM ensembling by moving aggregation into a universal relative space. Each model probability vector PRESERVED_PLACEHOLDER_^^^^5 OR all:\5 OR all:\5^^^^ is mapped to PRESERVED_PLACEHOLDER_^^^^5 OR all:\5 OR all:\5^^^^, where PRESERVED_PLACEHOLDER_^^^^5 OR all:\55^^^^ is a normalized relative representation matrix defined over a common anchor set. The ensemble relative vector is the arithmetic mean PRESERVED_PLACEHOLDER_^^^^5 OR all:\5^^^^6, and a search-based inverse transformation then solves

PRESERVED_PLACEHOLDER_^^^^5 OR all:\5^^^^7

within the main model’s simplex. Across six benchmarks, including MMLU, ARC-Challenge, GSM8K, PIQA, TriviaQA, and NaturalQuestions, DEEPEN is reported to outperform the best single model; for example, MMLU rises from approximately PRESERVED_PLACEHOLDER_^^^^5 OR all:\5^^^^8 to approximately PRESERVED_PLACEHOLDER_^^^^5 OR all:\5^^^^9, and ARC-Challenge from approximately PRESERVED_PLACEHOLDER_^^^^5 OR all:\5search_query5^^^^ to approximately PRESERVED_PLACEHOLDER_^^^^5 OR all:\5all:\5^^^^ (&&&^^^^5all:\5^^^^&&&).

Taken together, these results refute two common assumptions. Shared vocabularies are not necessary for token-level collaboration, and explicit evaluation of every constituent distribution at every decoding step is not necessary to reproduce or approximate the behavior of a conventional ensemble.

^^^^5 OR all:\5^^^^. Parallel ensembling beyond LLM decoding

Outside autoregressive generation, parallel ensembling appears in data curation, privacy-preserving synthesis, and conventional supervised prediction.

For Bengali–English machine translation, aligner ensembling combines the outputs of multiple sentence aligners by document-wise set union,

PRESERVED_PLACEHOLDER_^^^^5 OR all:\5 OR all:\5^^^^

and then restores precision via LASER-based semantic filtering,

PRESERVED_PLACEHOLDER_^^^^5 OR all:\5 OR all:\5^^^^

With Hunalign, Bleualign, and Gargantua, the best-performing ensemble is Bleualign PRESERVED_PLACEHOLDER_^^^^5 OR all:\5 OR all:\5^^^^ Hunalign, with optimal margin PRESERVED_PLACEHOLDER_^^^^5 OR all:\55^^^^. On a gold set of PRESERVED_PLACEHOLDER_^^^^5 OR all:\5^^^^6 human-aligned sentences, BH attains PRESERVED_PLACEHOLDER_^^^^5 OR all:\5^^^^7 FPRESERVED_PLACEHOLDER_^^^^5 OR all:\5^^^^8, and BH + LASERPRESERVED_PLACEHOLDER_^^^^5 OR all:\5^^^^9 attains $k$^^^^5search_query5^^^^ F$k$^^^^5all:\5^^^^. Downstream MT trained on BH + LASER uses $k$^^^^5 OR all:\5^^^^M pairs and reaches $k$^^^^5 OR all:\5^^^^ BLEU on SUPara and $k$^^^^5 OR all:\5^^^^ BLEU on SIPC, while the full $k$5M corpus yields $k$6 BLEU, $k$7 over the best previously reported Bengali–English system (Hasan et al., 2020).

In private synthetic data, parallel ensembling operates at the data-generator level. A total privacy budget $k$8 is split across $k$9 independent executions of a DP mechanism $k\,C_s$^^^^5search_query5^^^^, either as a simple DP ensemble using $k\,C_s$^^^^5all:\5^^^^ per run or as a DP ensemble with Poisson subsampling at rate $k\,C_s$^^^^5 OR all:\5^^^^ before applying an amplified budget. The downstream classifiers trained on the resulting synthetic datasets are then ensembled by uniform averaging of logits or probabilities. On Adult with DP-CTGAN, the reported averages across four downstream models are: single synthetic dataset, accuracy $k\,C_s$^^^^5 OR all:\5^^^^ and ECE $k\,C_s$^^^^5 OR all:\5^^^^; DP ensemble $k\,C_s$5, $k\,C_s$6 and $k\,C_s$7; DP ensemble + subsampling $k\,C_s$8, $k\,C_s$9 and $C_L$^^^^5search_query5^^^^ (&&&^^^^5 OR all:\5^^^^&&&).

In image classification, parallel ensembling has been analyzed from a FLOP-efficiency perspective. On CIFAR-^^^^5all:\5search_query5^^^^, a single WRN-^^^^5all:\5^^^^6-8 uses $C_L$^^^^5all:\5^^^^M FLOPs and attains $C_L$^^^^5 OR all:\5^^^^ top-^^^^5all:\5^^^^ accuracy, whereas an ensemble of $C_L$^^^^5 OR all:\5^^^^ WRN-^^^^5all:\5^^^^6-^^^^5 OR all:\5^^^^ uses $C_L$^^^^5 OR all:\5^^^^M FLOPs and attains $C_L$5. On ImageNet, a single EfficientNet-B^^^^5 OR all:\5^^^^ uses $C_L$6 GFLOPs and attains $C_L$7, whereas an ensemble of $C_L$8 B^^^^5 OR all:\5^^^^ uses $C_L$9 GFLOPs and attains $k\le m$^^^^5search_query5^^^^ (Kondratyuk et al., 2020).

These examples show that the ensemble object need not be a set of simultaneously queried predictors. It may instead be a set of candidate alignments or synthetic datasets produced in parallel, with aggregation delayed until filtering or downstream prediction.

5. Diversity, ceilings, and calibration pathologies

The effectiveness of parallel ensembling depends less on the mere number of constituents than on the diversity of their errors or samples. In the DP-synthetic-data setting, the stated rationale is that independently noised datasets increase sample diversity: if synthetic datasets are viewed as draws from distributions $k\le m$^^^^5all:\5^^^^, their mixture $k\le m$^^^^5 OR all:\5^^^^ has support that is strictly larger and empirical covariance

$k\le m$^^^^5 OR all:\5^^^^

that exceeds that of any single $k\le m$^^^^5 OR all:\5^^^^. Empirically, however, this benefit appears mechanism-dependent: gains are reported for GAN-based DP mechanisms, whereas MST, MWEM, and GEM show accuracies within $k\le m$5 of the single-dataset baseline and no significant ECE improvements (&&&^^^^5 OR all:\5^^^^&&&).

A complementary negative result is given for tabular foundation models. Across six TFMs on $k\le m$6 OpenML classification tasks, the mean pairwise Q-statistic is $k\le m$7 with $k\le m$8, indicating near-perfect agreement in successes and failures. Under this high-consensus regime, the best ensemble, two-level cascade stacking, reaches $k\le m$9 accuracy, only $t_s+t_{\mathrm{comm}}$^^^^5search_query5^^^^ over the strongest single TFM, at $t_s+t_{\mathrm{comm}}$^^^^5all:\5^^^^ seconds fit time versus $t_s+t_{\mathrm{comm}}$^^^^5 OR all:\5^^^^ seconds for TabICLv^^^^5 OR all:\5^^^^, a $t_s+t_{\mathrm{comm}}$^^^^5 OR all:\5^^^^ compute increase. A Friedman test gives $t_s+t_{\mathrm{comm}}$^^^^5 OR all:\5^^^^ and Nemenyi critical difference $t_s+t_{\mathrm{comm}}$5, placing Cascade_^^^^5 OR all:\5^^^^level, Stacking_LR, Greedy_Selection, and TabICLv^^^^5 OR all:\5^^^^ in one top equivalence group (Tanna et al., 18 May 2026).

Calibration can also degrade even when ^^^^5search_query5^^^^/^^^^5all:\5^^^^ accuracy remains competitive. In the same TFM study, logistic-regression stacking has competitive accuracy and ROC-AUC but the worst log-loss rank among the ensembles, and its ECE is $t_s+t_{\mathrm{comm}}$6 versus $t_s+t_{\mathrm{comm}}$7 for TabICLv^^^^5 OR all:\5^^^^. Temperature-Scaled Blending exhibits a similar pattern with ECE $t_s+t_{\mathrm{comm}}$8. The reported interpretation is that the meta-learner sharpens class boundaries, improving hard decisions while inflating confidence (Tanna et al., 18 May 2026).

Vision ensembles display a milder version of the same principle through diminishing returns in $t_s+t_{\mathrm{comm}}$9: additional members yield smaller marginal gains, and beyond a certain ensemble size the FLOP cost overtakes the accuracy benefit. This suggests that parallel ensembling is bounded both statistically, by diversity ceilings, and operationally, by communication and compute overhead (Kondratyuk et al., 2020).

6. Parameter-space composition, uncertainty, and operational consequences

A distinct line of work treats parallel ensembling as composition in parameter space rather than aggregation of independently materialized predictions. Tangent Model Composition fine-tunes tangent components PRESERVED_PLACEHOLDER_^^^^5all:\5search_query5search_query5^^^^ independently on disjoint datasets PRESERVED_PLACEHOLDER_^^^^5all:\5search_query5all:\5^^^^ and combines them through PRESERVED_PLACEHOLDER_^^^^5all:\5search_query5 OR all:\5^^^^. Standard ensembling of PRESERVED_PLACEHOLDER_^^^^5all:\5search_query5 OR all:\5^^^^ non-linear fine-tuned models requires PRESERVED_PLACEHOLDER_^^^^5all:\5search_query5 OR all:\5^^^^ inference cost, whereas TMC is reported as PRESERVED_PLACEHOLDER_^^^^5all:\5search_query55^^^^ because all Jacobian-vector products can be batched or computed in one augmented pass. Empirically, this yields a PRESERVED_PLACEHOLDER_^^^^5all:\5search_query5^^^^6–PRESERVED_PLACEHOLDER_^^^^5all:\5search_query5 reduction in inference time for ensembles of PRESERVED_PLACEHOLDER_^^^^5all:\5search_query5^^^^8–PRESERVED_PLACEHOLDER_^^^^5all:\5search_query5 models, with an approximately PRESERVED_PLACEHOLDER_^^^^5all:\5all:\5search_query5^^^^ percentage-point accuracy improvement over non-linear softmax/logit ensembling on PRESERVED_PLACEHOLDER_^^^^5all:\5all:\5all:\5^^^^ class-incremental experiments. Because components live in a vector space, forgetting is implemented by setting PRESERVED_PLACEHOLDER_^^^^5all:\5all:\5 OR all:\5^^^^, leaving no residual effect from the removed task (&&&^^^^5 OR all:\5^^^^&&&).

Anchored ensembling gives another parallel interpretation, this time for approximate Bayesian inference. Instead of regularizing every ensemble member toward the same prior mean, each member draws an anchor PRESERVED_PLACEHOLDER_^^^^5all:\5all:\5 OR all:\5^^^^ and minimizes

PRESERVED_PLACEHOLDER_^^^^5all:\5all:\5 OR all:\5^^^^

In the linear-Gaussian analysis, the anchored MAP solution has the correct posterior mean but covariance PRESERVED_PLACEHOLDER_^^^^5all:\5all:\55^^^^, which underestimates marginal variances and overestimates correlations except in two special cases. Empirically, anchored ensembles are reported to outperform standard ensembles in uncertainty-sensitive settings and to be competitive with variational methods (Pearce et al., 2018).

Operationally, these two approaches expand the meaning of parallel ensembling. TMC emphasizes parallel fine-tuning with single-model inference and zero-cost unlearning; anchored ensembling emphasizes embarrassingly parallel training with uncertainty aggregation at prediction time. This suggests that the field’s central question is not simply how to average outputs, but how to choose an aggregation space in which independence across members can be exploited without erasing the information that makes ensembling useful.

Parallel ensembling therefore occupies a broad methodological spectrum. At one extreme are explicit output-level averages and votes; at another are stochastic or search-based token-level schemes for autoregressive models; at another are parameter-space compositions and privacy-budgeted parallel data generators. Across these settings, the literature is consistent on one point: parallel execution alone does not guarantee improvement. Gains are strongest when the ensemble preserves genuinely complementary structure—across errors, vocabularies, representations, tasks, or synthetic-data modes—and weakest when the constituent pool is already near-redundant (Fu et al., 1 May 2026, Tanna et al., 18 May 2026).