When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models

Abstract: Multi-model LLM systems such as routing, voting, cascades, fusion, and mixture-of-agents are used to beat single-model accuracy. We show that their gain is capped by a quantity the field rarely reports. For any policy whose output is one member model answer, accuracy cannot exceed one minus beta, where beta is the rate at which every model is wrong on the same query. In contrast, the usual diagnostic, average pairwise error correlation rho, cannot identify beta: error laws with identical marginals and pairwise correlations can have different all-wrong rates. A Clopper-Pearson bound on beta gives a finite-sample certificate on the largest gain any router, vote, or cascade could deliver before training a router. Across 67 models from 21 providers, a tetrachoric-calibrated single-factor model still underprices the all-wrong tail: on open-ended mathematics, observed beta is 0.052 versus 0.023 under the full 67-model Gaussian copula, about 2.5 times underpricing, with 90 percent CI 1.7 to 3.4 and k equals 17. The effect recurs on execution-graded code, where beta is 0.079. Re-asking the same GPQA-Diamond questions in free-response rather than multiple-choice form reopens the tail, with beta 0.127 and a five-judge panel with kappa 0.73 to 0.92, locating co-failure in answer format rather than subject. At matched quality, low-rho heterogeneous ensembles beat high-rho Self-MoA, but on checkable tasks in our pool, combining models rarely beats the single best model without a strong query-level routing signal. Gains come from models failing on different questions, not from adding more models.

• The paper demonstrates that LLM orchestration gains are capped by the co-failure rate (β), setting a strict upper limit on accuracy improvements.
• The paper employs extensive empirical analysis across 67 models to show that traditional pairwise error correlation (ρ) underestimates co-failure phenomena.
• The paper introduces a cost-zero Clopper–Pearson certificate for β, highlighting the roles of task format and market churn in orchestration strategies.

Co-Failure Ceiling in LLM Orchestration: Routing, Voting, and Mixture-of-Agents Across Frontier Models

Introduction

The proliferation of heterogeneous LLMs from multiple providers has rendered inference orchestration essential for maximizing accuracy while controlling costs and risks. This paper establishes a rigorous theoretical and empirical ceiling on the achievable gains from orchestration strategies—including routing, voting, cascades, fusion, and mixture-of-agents (MoA)—by demonstrating that improvements are strictly capped by the "co-failure" rate ($\beta$): the probability that all models fail on the same query. The widely-used pairwise error correlation statistic ($\rho$), though informative for model diversity, cannot identify or bound $\beta$, and systematically underrepresents orchestration headroom as pool sizes expand. The authors measure $\beta$, $\rho$, and derived ceilings across 67 frontier LLMs from 21 providers and multiple domains, uncovering consistent empirical co-failure phenomena and clarifying when, and to what extent, combining LLMs yields actionable gains.

Orchestration Ceiling: Theoretical Foundations

The orchestration ceiling arises from the fundamental proposition that any policy (router, vote, cascade) outputting member models' responses is bounded in accuracy by $1-\beta$, where $\beta$ is the rate of joint failure (all models err simultaneously). This ceiling is tight: even an oracle router that perfectly picks the correct model per query cannot exceed this limit. The achievable gain above single-best accuracy is the difference $\Pr[\text{single-best wrong}] - \beta$, isolating improvement to resolvable disagreements rather than unresolvable joint failures.

A critical contribution is a practical Clopper–Pearson certificate for $\beta$: a single graded, held-out query set suffices to establish a pre-deployment, cost-zero certificate on the maximal gain any orchestration policy can deliver over single-best, circumventing the need for expensive bespoke routing or fusion training.

Figure 1: The co-failure residual is a common-mode atom, not copula misspecification (MATH-500, 67 models, $k=17/330$). Left: empirical $\rho$0 versus copula-based predictions; right: monotonic growth of underpricing with pool size.

Empirical Analysis: $\rho$1 versus $\rho$2 in Real Pools

The paper investigates 67 models (frontier to cheap), spanning domains such as mathematics, science, and code. Main findings include:

• Measured $\rho$3 far exceeds copula-based predictions from pairwise $\rho$4: On open-ended mathematics (MATH-500), empirical $\rho$5, while the tetrachoric Gaussian copula based on measured $\rho$6 yields $\rho$7, representing an underpricing factor of $\rho$8 (90% CI $\rho$9).
• This underpricing grows with pool size: Resampling model pool size $\beta$0 shows the tetrachoric-underpricing ratio increases monotonically from $\beta$1 (two models) to $\beta$2 at $\beta$3, isolating pool size as the key driver.
• Common-mode atom cannot be encoded via pairwise statistics or exchangeable copulas: Even full-pairwise Gaussian copula or Clayton copulas calibrated to match pairwise correlations underpredict $\beta$4. The empirical $\beta$5 is an irreducible mass indicating non-Gaussian, non-copula tail dependence driven by common failure modes.

Two Regimes: Ceiling-Bound vs. Realizability-Bound Domains

The joint failure rate constrains orchestration in two distinct regimes:

• Ceiling-bound: Domains such as open-ended mathematics and execution-graded code consistently manifest positive $\beta$6, resulting in a strict upper bound on accuracy. Here, the best models increasingly fail together, and the achievable gains from orchestration are negligible.
• Realizability-bound: Domains such as graduate-level science (GPQA-Diamond, multiple-choice) exhibit $\beta$7. The maximal achievable improvement ($\beta$8) is resolvable disagreement, theoretically possible for a deployable router but empirically unreached.

Changing only the answer format (multiple-choice $\beta$9 free-response) on identical GPQA questions flips the regime and opens up co-failure blocks, marking answer format—not content—as the underlying driver.

Figure 2: Format, not content, sets the regime. Content-controlled GPQA questions display a co-failure block only in free-response.

Figure 3: Two regimes map: co-failure ($\beta$0) per domain. Ceiling-bound (open-ended math/code/free-response science) versus realizability-bound (multiple-choice science/MMLU-Pro).

Economic Framework and Diversification Limit

Orchestration is framed as a budget-constrained allocation across priced, correlated pools, leveraging LP duality. Under a fixed budget, the optimal policy involves a per-type bang-per-buck rule with a shadow price $\beta$1 on the inference dollar. Cost-aware fusion yields a tight diversification limit $\beta$2, decreasing in error correlation $\beta$3 and cost $\beta$4, extending classical portfolio theory to inference composition.

At matched quality, diverse low-$\beta$5 ensembles outperform high-$\beta$6 Self-MoA, validating information-theoretic predictions. However, naive mixture of unequal-quality models is shown to be a liability, as weaker models can outvote stronger ones. Query-level block covariance structures, rather than global $\beta$7, are shown to govern the true diversification limit.

Figure 4: Pillar A cost–quality frontier: per-query oracle versus single-best; cheap models populate the frontier.

Figure 5: Majority-vote gain versus inter-model $\beta$8 at matched quality, indicating a negative diversification slope in line with theory.

Cascade Calibration and Churn Optionality

The study analyzes cascade strategies, showing that as the verifier's AUC approaches $\beta$9, cascade advantage collapses to random mixing. Empirically, calibrated cascades only dominate when the deferred tail offers a positive accuracy edge.

Furthermore, market churn is modeled as a Poisson process, with frontier releases yielding significant option value for broad access rather than committing to a single provider or configuration—a regime where orchestration value is driven by churn and heterogeneity, not absolute model accuracy.

Figure 6: Pillar A, learned router performance versus cost-aware oracle frontier and single-best; realized router gain is near zero.

Figure 7: Optionality under churn: realized route value versus commitment.

Practical and Theoretical Implications

Practical implications:

• On open-ended, verifiable tasks, orchestration rarely beats the single best model without a strong query-level routing signal; gains stem from models failing on different queries, not numerical pool expansion.
• Pairwise $\rho$0 is insufficient for orchestration decisions; $\rho$1—the rate of joint failure—should be directly estimated and certified.
• As LLM providers converge in quality and error modes, orchestration opportunities shrink; market churn and task format become the primary levers for buyers.

Theoretical implications:

• The non-identifiability of $\rho$2 from pairwise correlation highlights a fundamental ambiguity in multivariate binary error distributions (Fréchet class).
• Gaussian copula models, commonly used for ensemble error floors, systematically underestimate co-failure in large pools due to zero lower tail dependence.
• Information-theoretic benefits of diversification only manifest at matched quality and relevant block covariance; global measures may misrepresent actual gains.

Future Directions

Empirical extension to generative, non-verifiable benchmarks is warranted. Further theoretical work is needed to characterize non-Gaussian joint failure modes and to calibrate diversification limits across varying market churn rates and price structures. Improved query-level signal extraction for routing under realizability-bound domains could unlock remaining orchestration gains. Large-scale, official-judge replication in code and panel-graded science domains will sharpen external validity.

Conclusion

Achievable gains from LLM orchestration are strictly capped by the co-failure rate $\rho$3, which pairwise error correlation $\rho$4 is blind to and systematically underprices as pool size increases. Across the 2026 frontier pool, non-Gaussian common-mode failures dominate orchestration headroom, and naive fusion strategies—absent matched quality and controlled block covariance—are often counterproductive. Practical orchestration must directly estimate and certify $\rho$5; theoretical models must extend beyond pairwise copula representations. Task format, error dispersion, and market churn—not mere model count or diversity—determine when and how combining LLMs helps.

Figure 8: Optionality under churn: realized option value of breadth through epochal frontier releases, alongside rapid dollars-per-correct reduction.