Multi-LLM Ensemble Method

• Multi-LLM ensemble method is a technique that leverages diverse language models during inference to dynamically select, fuse, or refine candidate responses.
• It employs strategies across different inference stages—before, during, and after generation—to optimize performance through routing, token-level fusion, and judge-based selection.
• The method converts model heterogeneity into practical gains by orchestrating complementary strengths, though challenges like high computational cost and correlated errors remain.

A multi-LLM ensemble method is the comprehensive use of multiple LLMs during downstream inference, where each model is treated as a candidate for handling user queries and the system seeks to benefit from their individual strengths (Chen et al., 25 Feb 2025). In current research, the term covers hard routing before generation, token- or span-level coordination during generation, and post-hoc selection, fusion, or cascading after full responses have been produced. Representative formulations range from dynamic logit-level fusion of frozen specialists (Li et al., 3 Jun 2026) to judge-orchestrated candidate selection (Bondarenko et al., 6 May 2026) and semantic-layer aggregation over answer distributions (Mehrabi et al., 29 May 2026). The common premise is that model heterogeneity is useful only when it is converted into a reliable decision rule, a fusion operator, or a structured deferral policy.

1. Conceptual scope and boundaries

The modern literature distinguishes LLM ensembling from several adjacent ideas. The survey literature separates it from parameter merging or fusion, which combines weights into one model, and from collaboration settings in which models are assigned differentiated roles rather than treated as peer candidates facing the same downstream query (Chen et al., 25 Feb 2025). This distinction matters because most ensemble methods operate at inference time and are explicitly compatible with off-the-shelf or frozen experts.

A second boundary concerns the unit of combination. Some methods combine full responses, some combine answer labels, some combine token distributions, and some combine internal states. DLLG, for example, keeps all experts frozen and learns token-wise fusion weights over their logits, with the fused logits defined by

$$\boldsymbol{\ell}_t = \sum_{i=1}^{K} w_{t,i}\,\boldsymbol{\ell}_t^{(i)}.$$

The gate is predicted from the experts’ hidden-state trajectories rather than from handcrafted confidence proxies (Li et al., 3 Jun 2026). By contrast, LLM-PeerReview does not fuse text token-by-token at all; it treats candidate responses as submissions, reuses LLMs as judges, aggregates their scores, and selects one final response (Chen et al., 29 Dec 2025).

A third boundary concerns heterogeneity. Some systems ensemble specialists from the same family because shared tokenizers make logit fusion feasible, as in DLLG (Li et al., 3 Jun 2026). Others deliberately maximize family, scale, and prompting diversity, as in the SemEval MTRAGEval system that used seven LLMs and two prompting variants before selecting the best candidate with a GPT-4o-mini judge (Bondarenko et al., 6 May 2026). This suggests that “multi-LLM ensemble” is not a single architecture class but a design space organized around when experts are invoked, what information is shared, and how disagreement is resolved.

2. Taxonomy by inference stage

The survey taxonomy organizes the field into “ensemble-before-inference,” “ensemble-during-inference,” and “ensemble-after-inference” (Chen et al., 25 Feb 2025). The timing determines both the available information and the dominant technical constraints.

Timing	Core mechanism	Representative systems
Before inference	Router selects one model or a subset from the query alone	RouteLLM, Eagle, HomoRouter (Chen et al., 25 Feb 2025)
During inference	Models are coordinated while decoding tokens, spans, or reasoning steps	DLLG, RLAE, GaC, LE-MCTS (Li et al., 3 Jun 2026)
After inference	Full responses are generated first, then selected, fused, or cascaded	LLM-PeerReview, LLM-TOPLA, judge-based MTRAGEval, DER (Chen et al., 29 Dec 2025)

Before-inference methods emphasize cost-efficiency. They route a query to a model predicted to do well, often under an explicit cost-quality trade-off. The survey places classification-based routers, reward-based routers, assignment-based policies, and non-pretrained routers in this family (Chen et al., 25 Feb 2025). The limitation is structural: these methods have no access to the current query’s candidate responses, so they cannot exploit within-instance complementary evidence.

During-inference methods operate at finer granularity. Token-level systems align or weight next-token distributions, span-level systems generate and assess short fragments, and process-level systems coordinate intermediate reasoning steps (Chen et al., 25 Feb 2025). RLAE is a clear example: it casts ensembling as an MDP over prompt-plus-history states and chooses a span-level weight vector for the next segment of generation, using

$$p(y_h \mid x, y_{<h}) = \sum_{k=1}^{K} w_h^{(k)}\, p_{M_k}(y_h \mid x, y_{<h}, \theta_k).$$

Its main implementation fixes the weights over a span of length $L$, rather than recomputing them per token (Fu et al., 31 May 2025).

After-inference methods defer combination until complete candidate responses exist. This family includes simple consensus selection, learned rerankers, selection-then-regeneration pipelines, and cascades that progressively escalate to stronger models only when the earlier output is judged insufficient (Chen et al., 25 Feb 2025). DER occupies an intermediate position: it models the route through multiple experts as an MDP, but each selected model sees the prior answer and tries to improve it, with stopping governed by a learned Terminator and a cost-aware reward (Hu et al., 2024).

3. Aggregation operators and disagreement resolution

The simplest aggregation rule is voting. In clinical labeling of ECG reports, seven open-source LLMs vote over $\{AF, NotAF, Uncertain\}$, and the final label is accepted only if it is the unique top vote and its vote share exceeds a minimum threshold; otherwise the system abstains to Review (Huang et al., 2024). A related but even simpler design appears in ordinal sentiment classification, where five stochastic runs of the same model are aggregated by the median of valid outputs rather than by majority vote, yielding a robust self-ensemble for labels in $\{1,\dots,5\}$ (Niimi, 26 Apr 2025).

Weighted voting generalizes this idea when models are not equally reliable. DFPE computes subject-specific validation accuracies, filters underperformers by a quantile threshold, clusters models by fingerprint similarity, selects the best representative from each cluster, and then applies exponential accuracy-based weights in a final weighted vote over answer choices (Cohen et al., 29 Jan 2025). MSLEF applies a field-aware version of the same principle to resume parsing: scalar fields such as name, email, phone, and department use weighted majority vote; skills use weighted threshold voting; and nested fields such as education and experience trigger Gemini-based consensus synthesis when the base models disagree (Walid et al., 7 Sep 2025).

Selection-based ensembles replace voting with explicit scoring. In EnsLLM, one candidate program is generated from each model, syntactically invalid programs are removed, and the surviving candidates are ranked by aggregated pairwise similarity:

$$\text{aggregated\_similarity}(P_i)=\sum_{j\neq i}\text{similarity}(P_i,P_j),$$

where the pairwise score combines CodeBLEU-based structural similarity with CrossHair-based behavioral equivalence (Mahmud et al., 20 Mar 2025). LENS learns a different decision rule: each model gets a confidence predictor derived from layer-wise internal states and normalized probabilities, and the ensemble returns the answer of the model with maximal predicted confidence,

$\hat y = y_{\arg\max_i c_i}.$

This is a winner-take-all selector rather than a soft combiner (Guo, 31 Jul 2025).

Some systems use judges rather than validators or learned confidence heads. LLM-PeerReview scores every candidate with every available LLM judge, then aggregates the scores either by simple averaging or by a Dawid–Skene-style truth-inference model with EM, and finally selects the highest-scoring response (Chen et al., 29 Dec 2025). Multilingual KokoroChat uses a stronger generative variant of judge-based fusion: three LLMs produce translation hypotheses, and Gemini 2.5 Pro analyzes the strengths and weaknesses of all candidates before synthesizing a new improved translation, utterance by utterance, while preserving dialogue-level consistency (Suzuki et al., 24 Mar 2026).

A separate line of work resolves disagreement without additional LLM calls. The structured message-passing method for multiple-choice QA represents LLMs as variable nodes in a bipartite factor graph, introduces check nodes for calibration, margin decisiveness, domain expertise, dissent, and relational reliability, and then runs iterative belief propagation over answer distributions (Mehrabi et al., 29 May 2026). Its asymmetric damping mechanism explicitly addresses “anchor corruption,” the case in which weaker models degrade a stronger specialist under naive aggregation.

4. Supervision, optimization, and diversity management

Multi-LLM ensembles differ sharply in how much supervision they require. At one extreme, LLM-PeerReview is described as fully unsupervised: it needs no gold labels, no reference responses for target queries, no task-specific training set, and no calibration set; the weighted variant estimates judge reliabilities from the unlabeled score matrix itself (Chen et al., 29 Dec 2025). At the other extreme, DLLG uses sparse response-level correctness labels broadcast to token positions under teacher forcing and trains its gate by mean squared error,

$$\mathcal{L} = \frac{1}{T}\sum_{t=1}^{T}\sum_{i=1}^{K}(w_{t,i}-s_i)^2,$$

where $s_i\in\{0,1\}$ records whether expert $i$ produced a correct full response on that example (Li et al., 3 Jun 2026).

A middle ground is confidence supervision. LENS trains one lightweight predictor per model with binary cross-entropy on whether that model’s answer is correct, using features obtained from layer-wise hidden states projected through a logit-lens-style transformation (Guo, 31 Jul 2025). This retains frozen base models and adds only small confidence heads.

Reinforcement learning appears when ensemble control itself is treated as a sequential decision problem. RLAE learns span-level mixture weights by PPO or MAPPO, with reward tied directly to final output quality and optional dense process rewards in principle, though the reported experiments primarily use sparse terminal rewards (Fu et al., 31 May 2025). DER formulates sequential expert selection as an MDP over states $$p(y_h \mid x, y_{<h}) = \sum_{k=1}^{K} w_h^{(k)}\, p_{M_k}(y_h \mid x, y_{<h}, \theta_k).$$0, uses PPO to train the controller, and introduces a Knowledge Transfer Prompt so that later experts can revise earlier answers rather than restarting from scratch (Hu et al., 2024). EMORL decomposes multi-objective fine-tuning into independently trained objective-specific models and recombines them afterward through weighted hidden-state fusion, with weights chosen by hierarchical grid search; for three objectives and precision $$p(y_h \mid x, y_{<h}) = \sum_{k=1}^{K} w_h^{(k)}\, p_{M_k}(y_h \mid x, y_{<h}, \theta_k).$$1, this search required 135 evaluations versus 32,768 for standard grid search (Kong et al., 5 May 2025).

Diversity management is often the hidden variable behind empirical gains. DFPE defines per-subject response “fingerprints,” clusters models with DBSCAN and cosine similarity, filters low performers by a subject-specific quantile, and uses $$p(y_h \mid x, y_{<h}) = \sum_{k=1}^{K} w_h^{(k)}\, p_{M_k}(y_h \mid x, y_{<h}, \theta_k).$$2, $$p(y_h \mid x, y_{<h}) = \sum_{k=1}^{K} w_h^{(k)}\, p_{M_k}(y_h \mid x, y_{<h}, \theta_k).$$3, and $$p(y_h \mid x, y_{<h}) = \sum_{k=1}^{K} w_h^{(k)}\, p_{M_k}(y_h \mid x, y_{<h}, \theta_k).$$4 in its best reported configuration (Cohen et al., 29 Jan 2025). LLM-TOPLA goes further by formalizing “focal diversity,” a co-failure metric designed to capture the correlation between diversity and ensemble quality, then uses a genetic algorithm to search the space of sub-ensembles before training a combiner (Tekin et al., 2024). These works support a consistent interpretation: diversity is useful only when it is aligned with non-overlapping error patterns, not merely with surface disagreement.

5. Application domains and empirical patterns

Reported gains span code generation, reasoning, translation, document parsing, grounded dialogue, and clinical labeling.

Domain	Representative finding	Paper
Code generation	MPLE reports up to 17.92% improvement and 96.25% accuracy on HumanEval	(Xue et al., 2024)
Code generation	EnsLLM reaches 90.2% on HumanEval and 50.2% on LiveCodeBench, versus GPT-4o at 83.5% and 43.4%	(Mahmud et al., 20 Mar 2025)
Reasoning and code	DLLG reports the best average score at 1.5B scale, 49.63	(Li et al., 3 Jun 2026)
Broad reasoning, science, and code	RLAE$$p(y_h \mid x, y_{<h}) = \sum_{k=1}^{K} w_h^{(k)}\, p_{M_k}(y_h \mid x, y_{<h}, \theta_k).$$5 reports an average score of 70.1	(Fu et al., 31 May 2025)
Multitask QA	DFPE reports 73.5% overall accuracy and 74.0% discipline-accuracy on MMLU	(Cohen et al., 29 Jan 2025)
Translation in counseling dialogue	Ensemble translations are preferred by humans over any individual LLM	(Suzuki et al., 24 Mar 2026)
Automated data science	SPIO-E improves average predictive performance over SPIO-S under both GPT-4o and Claude 3.5 Haiku backends	(Seo et al., 30 Mar 2025)
Resume parsing	MSLEF reaches RS 91.0 versus 84.0 for the best single model	(Walid et al., 7 Sep 2025)
EHR labeling	The ECG labeling system labels 623,566 reports with an estimated accuracy of 98.2%	(Huang et al., 2024)
Multi-turn grounded dialogue	The SemEval-2026 winning ensemble achieves conditioned HM of 0.7827 versus 0.6390 for the strongest baseline	(Bondarenko et al., 6 May 2026)

Several empirical regularities recur across these applications. First, structured tasks with external validators or task-specific evaluators benefit disproportionately from selection-through-testing. Code generation is the clearest example: executable tests, differential behavior analysis, and behavioral equivalence checks provide unusually strong arbitration signals (Mahmud et al., 20 Mar 2025). Second, long-form or context-sensitive generation often benefits from synthesis rather than selection. Multilingual KokoroChat argues that selection-only ensembles cannot produce an output better than the best candidate already present and can introduce stylistic inconsistency across turns, whereas critique-and-synthesize refinement can improve dialogue-level coherence (Suzuki et al., 24 Mar 2026).

Third, even simple self-ensembles can be competitive in constrained settings. Repeating Llama-3-8B-Instruct five times with different seeds and aggregating by the median reduces RMSE by 18.6% relative to the best non-ensemble baseline, a single-run 70B model, on Yelp 5-point sentiment prediction (Niimi, 26 Apr 2025). This suggests that ensemble benefits do not require heterogeneous models in every task; when output variability is itself a source of error, repeated inference plus robust aggregation may already help.

Fourth, the best-performing ensemble is rarely the largest possible one. LLM-TOPLA reports that many sub-ensembles of size 2–4 outperform the full pool and that many smaller ensembles also outperform the best single model (Tekin et al., 2024). DFPE likewise shows that subject-adaptive filtering and clustering can beat majority voting even when most of the model pool is retained (Cohen et al., 29 Jan 2025). The evidence therefore favors selective heterogeneity over indiscriminate accumulation of experts.

6. Limitations, controversies, and open directions

The most persistent limitation is inference cost. Methods such as DLLG and RLAE require all experts to run at each decoding step; DLLG explicitly notes that all experts must execute at every step and that synchronized multi-expert decoding complicates optimized serving, while RLAE notes that compute still scales linearly with the number of models even when span-level control reduces policy frequency (Li et al., 3 Jun 2026). Judge-based systems pay a different cost: they must generate multiple full candidates and then invoke one or more evaluators, as in LLM-PeerReview and the SemEval judge-orchestrated ensemble (Chen et al., 29 Dec 2025).

A second limitation is representational compatibility. Logit-level fusion typically assumes shared tokenizers or aligned vocabularies; DLLG requires experts from the same model family or at least a shared tokenizer and vocabulary, and the semantic message-passing method is demonstrated only for multiple-choice QA with a shared discrete answer space (Li et al., 3 Jun 2026). Extending such methods to heterogeneous tokenizers or open-ended generation remains unresolved in the cited work.

A third limitation is correlated failure. EnsLLM explicitly notes that the method cannot repair cases where none of the candidates is correct and can fail when multiple models make the same wrong mistake (Mahmud et al., 20 Mar 2025). The MPLE flowchart, although formulated over programming languages rather than LLMs, makes the same structural point: selection-through-validation is only as strong as the validation suite $$p(y_h \mid x, y_{<h}) = \sum_{k=1}^{K} w_h^{(k)}\, p_{M_k}(y_h \mid x, y_{<h}, \theta_k).$$6, and first-pass acceptance can be brittle when the validator is weak or non-representative (Xue et al., 2024). This suggests that ensemble quality is bounded not only by diversity but by the reliability of the arbitration signal.

A fourth issue is evaluation bias. Judge-based methods inherit judge bias, self-preference, verbosity bias, position bias, and prompt sensitivity; LLM-PeerReview introduces its flipped-triple scoring trick specifically to mitigate score-level and position bias, but it still depends on the competence of the judges (Chen et al., 29 Dec 2025). Benchmark design can also distort conclusions. The SemEval MTRAGEval paper notes that all unanswerable Task B instances had empty reference passages, enabling a deterministic “I don't have an answer” shortcut, and Multilingual KokoroChat argues that current reference-free metrics evaluate utterances in isolation and miss dialogue coherence, role appropriateness, and multi-turn naturalness (Bondarenko et al., 6 May 2026).

The survey literature identifies several unresolved research directions: principled span-level ensemble methods, stronger unsupervised non-cascade post-hoc methods, and more general unsupervised cascades for generation (Chen et al., 25 Feb 2025). Taken together with the specialized systems reviewed above, this suggests that no single multi-LLM ensemble method dominates across access regimes and task structures. The practical choice depends on whether logits are available, whether auxiliary supervision exists, whether validators or judges are trustworthy, and whether the task rewards hard selection, soft fusion, iterative refinement, or structured abstention.