A Jury, Not an Electorate: Label-Free Reliability Estimation for Ensembles of Language Models

Abstract: An ensemble of LLMs is a jury, not an electorate. The answers are testimony from witnesses of unknown reliability, and the question is not what the majority said but who is credible. Most ensemble methods assign each expert a fixed reputation, often learned from labels. We ask whether it can instead be inferred from the pattern of agreement and disagreement among the experts. We model the experts as a jury: each question has a latent true answer, each expert reports it with unknown reliability, and those reliabilities are estimated by Expectation Maximisation (EM) from unlabelled answer configurations alone. We show that its decision rule reduces to a log-odds weighted linear score over the experts' answer distributions. What changes is not how the jury votes, but how it learns whom to trust. On a 1560-question benchmark, the jury, with nine experts of up to 3B parameters, performs comparably to the same fusion rule with label-fitted weights, showing no statistically significant advantage from supervision. Instead, it outperforms the best single model with matched self-consistency and the strongest label-free alternative on the full benchmark.

• Label-Free Reliability Estimation for Ensembles of Language Models presents a model that estimates expert reliability from observed agreement and disagreement patterns, achieving $76.9 \%$ accuracy.
• A key function of the model is an Expectation-Maximization estimation method that reduces per-expert parameters from $|E||V|^2$ to $|E|$ enabling the probability of correct reporting for each category.
• The paper demonstrates practical gains of the jury approach, outperforming an equal-weight baseline by 9.9% and the best unweighted model by 2.0%, but acknowledges dependencies on task composition and expert pool size.

Problem formulation and central claim

The paper studies post-hoc aggregation of heterogeneous language-model outputs when gold labels are unavailable for estimating model weights. Its central distinction is between an electorate, in which each expert has a fixed externally assigned reputation, and a jury, in which expert reliability is inferred from the observed pattern of agreement and disagreement. The proposed method estimates expert reliabilities through an unlabeled latent-variable model and then applies those estimates within a standard weighted fusion rule (2610.21085).

The paper separates two sources of ensemble improvement that are frequently conflated. First, an aggregator can retain each model’s answer distribution rather than reducing it to an argmax answer. Second, it can assign unequal weights to experts. On the evaluated benchmark, replacing argmax votes with answer distributions improves equal-weight majority aggregation by 8.4 percentage points, from $63.0 \pm 1.6\%$ to $71.4 \pm 1.8\%$. Reliability-based weighting contributes a further 5.5 points, yielding $76.9 \pm 1.3\%$. This decomposition is important: the paper’s methodological contribution concerns label-free reliability estimation, whereas distributional evidence is an orthogonal source of performance.

The benchmark contains 1,560 questions from six MMLU subjects, GSM8K, ARC-Challenge, and HellaSwag. The ensemble comprises nine instruction-tuned models with up to 3B parameters, including general-purpose, mathematical, coding, and multilingual specialists. Multiple-choice models provide distributions over four option letters; GSM8K uses eight sampled chain-of-thought generations per model, with empirical answer frequencies forming the distributions consumed by the aggregator.

Generative model and estimation procedure

For each question, the candidate set $V(q)$ is the set of distinct answers produced by the pool. The latent variable $z$ denotes the true answer. Expert $e$ has a single reliability parameter $\pi_e$, interpreted within the model as the probability of reporting $z$. Conditional on an error, the expert selects uniformly among the remaining candidates. Thus, for $m = |V(q)| - 1$,

$$P(a_e=v \mid z)= \begin{cases} \pi_e, & v=z,\ \dfrac{1-\pi_e}{m}, & v\neq z. \end{cases}$$

The model is a restricted one-coin Dawid–Skene model (2610.21085). It uses one reliability per expert rather than a full confusion matrix, reducing the number of parameters from a quantity proportional to $71.4 \pm 1.8\%$0 to $71.4 \pm 1.8\%$1. Its tractability depends on several assumptions: conditional independence of experts given the latent answer, question-independent reliabilities, and a uniform error channel over the observed candidate set. The paper explicitly notes that the candidate set is itself generated by the experts, so conditional independence is not implied by the data-generating process; it is an approximation imposed after conditioning on the observed support.

Expectation Maximisation estimates $71.4 \pm 1.8\%$2 using only unlabeled training configurations. Given posterior responsibilities $71.4 \pm 1.8\%$3 for candidate $71.4 \pm 1.8\%$4, the M-step sets each reliability to the average posterior-weighted probability mass assigned by that expert to the candidate regarded as correct:

$71.4 \pm 1.8\%$5

For point-mass answers, this reduces to the posterior-weighted frequency with which the expert’s reported answer is treated as correct. No gold answer enters this likelihood or update.

The paper also establishes a label-free identifiability result under a fixed support of size $71.4 \pm 1.8\%$6. If at least three experts have reliability different from uniform guessing, the reliability vector is identifiable from second- and third-order agreement moments. Notably, the guarantee does not require any expert to be better than chance. This is a population-level identifiability result under the stipulated conditional-independence model; it does not imply that finite-sample EM will find the global maximum. The likelihood is non-concave, and the paper documents degenerate local modes in some regimes.

The induced decision rule

The method does not introduce a new aggregation family. Its distinctive component is the estimation of the weights. For one answer per expert, the posterior decision rule simplifies exactly to a log-odds weighted vote:

$71.4 \pm 1.8\%$7

The selected candidate maximizes the sum of weights belonging to experts that proposed it. Experts with $71.4 \pm 1.8\%$8 receive negative weights and are actively voted against rather than merely ignored. This signed regime is empirically relevant: the weakest expert is estimated at $71.4 \pm 1.8\%$9 on four-option questions and therefore receives a negative weight.

For answer distributions, the same derivation yields a signed linear fusion:

$76.9 \pm 1.3\%$0

With non-negative weights this is a conventional linear opinion pool; with negative weights it is a signed fusion rule. Unlike argmax voting, distributional fusion can select a candidate that is not the top answer of any individual expert. Nevertheless, the paper’s reachability analysis shows that increased expressiveness is not the principal explanation for the empirical gain. A weighted vote over greedy answers can make the gold answer reachable on 94.2% of questions, while signed distributional fusion raises reachability to 99.7%. Yet the main benefit of distributions is ranking: they improve the ordering of candidates that are already reachable, rather than merely expanding the candidate set.

The following conceptual structure captures the paper’s comparison: the answer distributions are held fixed, while the electorate uses equal weights and the jury uses EM-estimated log-odds weights.

Figure 1: Accuracy against average parameters activated per question; distribution-consuming methods occupy the upper portion of the accuracy–compute frontier, with the jury and supervised linear fusion separated by approximately half a point.

Main empirical results

The jury reaches $76.9 \pm 1.3\%$1 with repeated sampling. It exceeds the strongest individual model, qwen3b, by 2.0 points, with a 95% bootstrap interval of $76.9 \pm 1.3\%$2 and $76.9 \pm 1.3\%$3. When qwen3b receives the same eight-sample self-consistency treatment on GSM8K, its accuracy is $76.9 \pm 1.3\%$4, and the jury remains 1.6 points higher, with interval $76.9 \pm 1.3\%$5 and $76.9 \pm 1.3\%$6. The comparison therefore attributes a residual advantage to cross-model aggregation beyond giving the strongest model additional samples, although the interval for the latter comparison reaches zero at its lower endpoint.

The strongest gain is concentrated in GSM8K. The jury improves over the best single model by 20.0 points on that source, where olmo2-1b is the pool’s strongest expert at $76.9 \pm 1.3\%$7 while qwen3b achieves only $76.9 \pm 1.3\%$8. On MMLU the gain is $76.9 \pm 1.3\%$9-0.1$V(q)$0+0.2$$points respectively. Consequently, the overall advantage is not uniform across tasks and is substantially driven by a domain in which the best generalist model is not the best specialist.</p> <p>The jury also performs strongly on questions for which the gold answer is not the unique plurality answer. On this minority stratum, <a href="https://www.emergentmind.com/topics/majority-voting" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">majority voting</a> achieves only <strong>6.2 <p>The label-free result is particularly notable in comparison with supervised weighting. The supervised linear fusion reaches$$V(q)$1, while the jury is 0.6 points higher conditional on the ten fitted splits. The corresponding bootstrap interval is $V(q)$2, but a nested bootstrap that also resamples the train/test split and refits each method gives $V(q)$3. Thus, the paper does not establish that label-free estimation is superior to supervised estimation, nor that the two procedures are statistically equivalent. It establishes that, in this benchmark and under this parameterisation, supervised weight estimation provides no detectable advantage over agreement-based estimation.

The comparison with other label-free methods is more favorable. On the full benchmark, the greedy jury scores $V(q)$4, compared with $V(q)$5 for ISP. With answer distributions, the jury reaches $V(q)$6, exceeding the equal-weight pool by 5.5 points and ISP by 7.9 points. On the 1,440 multiple-choice questions, SkillAggregation reaches $V(q)$7 without a labeled development set, while a test-selected SkillAggregation variant reaches $V(q)$8, still 2.7 points below the jury’s $V(q)$$9 on that subset. The paper appropriately qualifies this comparison: the SkillAggregation protocol and implementation require several choices, and its strongest reported variant is selected using test performance and therefore serves only as an upper reference.</p> <h2 class='paper-heading' id='robustness-across-pool-composition'>Robustness across pool composition</h2> <p>The paper evaluates all 246 sub-pools containing three, five, or seven of the nine experts. With the default multinomial evidence weight, the jury beats equal-weight fusion on 79 <p><img src="https://images.emergentmind.com/paper-images/2610-21085/pool_replication.png" alt="Figure 2" title="" class="markdown-image" loading="lazy"></p> <p><p class="figure-caption">Figure 2: Jury margins over equal-weight fusion, supervised fusion, and the best expert across sub-pools of different sizes; each point represents one sub-pool and horizontal bars show mean margins.</p></p> <p>The average margins over equal-weight fusion are$$z$0, $z$1, and $z$2 points for three-, five-, and seven-expert pools. The jury is less consistently superior to the best individual expert in small pools: its mean margins are $z$3, $z$4, and $z$5 points. This is consistent with the paper’s broader conclusion that aggregation is most useful when the pool contains complementary competence rather than merely weaker replicas of the best model.

The evidence-weight ablation reveals a significant optimisation issue. The default $z$6 corresponds to the eight sampled answers and approximates the multinomial likelihood. A fractional $z$7 is close in performance on the full pool, but causes severe failures in three-expert sub-pools: 58 of 84 such pools collapse to degenerate solutions with average accuracy of 15.8%, far below chance. At these solutions, all fitted reliabilities fall below $z$$8, resulting in negative weights for every expert. The model explains the observations as systematic collective misses rather than correctly identifying a latent answer.</p> <p><img src="https://images.emergentmind.com/paper-images/2610-21085/pool_replication_alpha.png" alt="Figure 3" title="" class="markdown-image" loading="lazy"></p> <p><p class="figure-caption">Figure 3: Sub-pool replication with the fractional evidence weight overlaid; the low-accuracy <a href="https://www.emergentmind.com/topics/cluster" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">cluster</a> for three-expert pools corresponds to degenerate EM fits.</p></p> <p>The default initialisation at$$z$9 avoids these failures in the tested sampled-distribution experiments, but this is not a theoretical guarantee. Random initialisations reached a degenerate mode in 3 of 200 runs. In the greedy $e$0 regime, a higher-likelihood degenerate mode exists on every split, and 19 of 20 random starts reach it on at least one split. This result is methodologically important: the paper’s practical success depends not only on the generative model but also on evidence scaling and EM basin selection.

Calibration and model diagnostics

The jury’s posterior ranks candidates usefully but is poorly calibrated as a probability distribution. With $e$1, mean top-1 posterior confidence is 0.991 against an accuracy of $e$2, yielding an expected calibration error of 0.223, Brier score $e$3, and negative log-likelihood $e$4. Reducing the evidence weight to $e$5 improves calibration but does not remove the issue. The posterior should therefore be treated as a decision statistic unless an external recalibration procedure is applied.

The conditional-independence assumption is empirically questionable. Experts agree on incorrect answers more frequently than independence predicts, with excess error agreement of $e$6 points for same-lineage pairs and $e$7 points for different-lineage pairs. This finding also contradicts a simple lineage-based explanation of shared failures: pre-training lineage does not predict the observed dependence reliably. Adding a shared-lure channel reduces accuracy by 0.4 to 0.6 points under repeated sampling, so the richer correlated-error model does not improve the evaluated decision rule.

Limitations and open questions

The evaluation is restricted to nine models and 1,560 questions, with three-quarters of the benchmark drawn from ARC-Challenge and HellaSwag. Most of the improvement over the best single model comes from GSM8K, so the overall result depends materially on the inclusion of a task where model specialization is pronounced. The support construction is also task-dependent: for GSM8K, the answer space is approximated by the set of answers sampled by the pool, and candidates never observed receive probability zero.

The jury requires querying every expert and, for generative tasks, performing repeated sampling. Its comparison with a greedy best expert therefore includes additional inference cost. Sampled variants activate approximately 22.2B parameters per question, compared with 14.4B for greedy variants. The method is consequently a post-hoc accuracy–cost tradeoff rather than a cost-neutral replacement for model selection.

The one-reliability-per-expert assumption is strong. qwen3b scores 76.5% on multiple-choice questions but 57.5% on GSM8K, whereas olmo2-1b shows the reverse pattern. Allowing task-specific reliabilities improves the jury from 76.9% to 77.3%, while the supervised counterpart improves to 76.4%. Whether label-free estimation retains its advantage with richer query-conditional or per-task reliability models remains unresolved. More broadly, the nested bootstrap does not support a claim of equivalence between label-free and supervised fitting, and generalisation beyond this pool, benchmark composition, and answer protocol remains untested.

Conclusion

The paper provides a precise account of how unlabeled agreement can be used to estimate reliability in language-model ensembles. Its main empirical conclusion is not that EM creates a new aggregation rule, but that it can estimate the weights of an existing log-odds linear fusion without gold labels. On the evaluated benchmark, answer distributions provide the largest isolated gain, while label-free reliability weighting adds a further 5.5 points over equal-weight fusion and yields $e$8 overall accuracy.

The strongest claim is that supervised reliability estimation offers no demonstrated advantage in this setting; the evidence supports comparable performance, not formal equivalence. The method’s practical validity depends on the conditional-independence approximation, the observed-support construction, appropriate evidence scaling, and stable EM optimisation. Within those constraints, the results show that a language-model ensemble can infer which experts to trust from their collective outputs rather than relying exclusively on labeled reputations (2610.21085).