Multi-LLM Jury Protocol

• Multi-LLM Jury Protocol involves using multiple language models to evaluate outputs, relying on consensus for more accurate results.
• This methodology mitigates the biases and errors of single models, improving robustness across diverse domains like medical safety and code evaluation.
• Key aggregation methods include majority voting, weighted scoring, and unanimity, balancing accuracy and safety needs.

A Multi-LLM Jury Protocol is a structured methodology in which multiple LLMs are used as parallel evaluators ("jurors" or "judges") whose outputs are aggregated by a formal rule—typically unanimity, majority, dynamic weighting, or an inference system—to produce a system-level verdict about the correctness, safety, or acceptability of a candidate output. Such protocols provide a practical and statistically grounded alternative to single-LLM judgments, aiming to combine the complementary strengths and mitigate the weaknesses, biases, and error modes of individual models across diverse tasks and domains.

1. Foundational Principles and Motivation

Multi-LLM jury protocols are grounded in the notion that individual LLMs, despite recent advances, exhibit systematic variance in judgment, idiosyncratic biases, and instability across tasks (e.g., text-to-SQL, NLG evaluation, code judgment, clinical safety) (Ullah et al., 12 Feb 2026, Reese et al., 20 Mar 2026, Li et al., 1 Dec 2025). By establishing formal rules for aggregation—such as unanimity, majority, reliability-weighted voting, or Bayesian/discriminative calibration—the protocol can yield more robust, reliable, and interpretable evaluation signals.

Key motivations include:

• Robustness to individual errors: Idiosyncratic failures, hallucinations, or bias in any one model are diluted in the aggregate, reducing false positives or negatives.
• Complementarity: Different LLMs are trained on differing corpora and objectives, exhibiting distinct error profiles (blind spots, error correlations), thereby enabling greater coverage of edge cases.
• Safety and risk management: In high-stakes settings (e.g., code acceptance, medical AI), protocols are often constructed to minimize false accepts via conservative decision rules (e.g., unanimous acceptance) (Ullah et al., 12 Feb 2026).
• Quantitative interpretability: Protocols support direct computation of trade-offs: true positive/false positive rates, Youden’s J statistic, agreement metrics (Cohen’s κ, Fleiss’ κ), and calibration measures (e.g., isotonic regression for aligning model outputs with expert panels) (Rouillard et al., 16 Apr 2026).

2. Jury Construction and Aggregation Rules

Protocol instantiation involves selection, configuration, and orchestration of a pool of LLMs, followed by formal aggregation:

• Model Pool Selection: Dominant strategies include vendor/architecture diversity (Rouillard et al., 16 Apr 2026), empirical leaderboard-driven selection, or model ranking by prior accuracy (Ullah et al., 12 Feb 2026, Zhou et al., 27 May 2025).
• Aggregation Mechanisms:
  • Unanimous Voting (AND Rule): Accept if and only if all jurors agree (e.g., accepting SQL only if every judge votes "correct") for conservative risk management (Ullah et al., 12 Feb 2026).
  • Majority Voting: Accept the label chosen by the majority of jurors; often used in clinical safety, content moderation, and adversarial detection (Reese et al., 20 Mar 2026, Hou et al., 8 Apr 2026).
  • Weighted/Calibrated Voting: Aggregate scores by per-judge reliability or inferred discriminative power (e.g., dynamic reliability modeling via Jury-on-Demand (Li et al., 1 Dec 2025) or BT-σ unsupervised judge-aware aggregation (Qian et al., 18 Feb 2026)).
  • Sequential/Chain Voting: Fixed-priority tie-breaker chains (e.g., DA→Maxim→Fallback in Amulet) (Ramnath et al., 26 May 2025).
  • Dynamic Juries: Input- or instance-specific selection of the most reliable jurors, with context-aware aggregation (Jury-on-Demand (Li et al., 1 Dec 2025), AutoLaw (Nguyen et al., 20 May 2025)).
• Aggregation Rules Table:

Aggregation Rule	Acceptance Mechanism	Application Example
Unanimity (AND)	All jurors agree (strict accept)	Code vetting (Ullah et al., 12 Feb 2026)
Majority	≥ ⌈n/2⌉ votes	Clinical safety (Reese et al., 20 Mar 2026)
Weighted average	Score by reliability weights	Summarization (Li et al., 1 Dec 2025)
Dynamic selection	Jury composition is input-adaptive	Legal compliance (Nguyen et al., 20 May 2025)
Sequential tie-breaker	Chain of judges, fixed order	Dialogue evaluation (Ramnath et al., 26 May 2025)

3. Protocol Instantiations and Domain-Specific Adaptations

Multi-LLM jury protocols have been instantiated and empirically analyzed in several domains:

Software Engineering and Code Review

The "Vibe Coding on Trial" study evaluates 15 LLMs on MySQL text-to-SQL, assembling unanimous committees of size 1–6 from the best six generators. Unanimous AND rules minimize false accepts; committee size and composition critically affect the trade-off between safety (FPR) and productivity (TPR). An optimal committee of three to four strong LLMs achieves peak Youden’s J (≈0.52), with committee composition validated per generator (Ullah et al., 12 Feb 2026). SE-Jury uses five logic-disjoint judges (direct, rethink, equivalence, condition extraction, test-gen-and-verify), dynamically selecting a subset to maximize agreement with human correctness scores (Zhou et al., 27 May 2025).

Clinical Safety Evaluation

The "LLM-as-a-Jury" paradigm evaluates LLMs for detecting safety issues in mental health application responses, using majority voting across three diverse LLMs per binary criterion. Empirical results show the ensemble achieves substantial agreement with human consensus (jury κ=0.74, single-best judge κ=0.75), with both advantages (scalability, clinical grounding) and limitations (ensemble ceiling effects, variable criterion difficulty) explicitly quantified (Reese et al., 20 Mar 2026).

Legal Compliance and Policy

AutoLaw dynamically generates edge-case adversarial scenarios per regulation and employs k-juror panels selected by past reliability (measured via an LLM verifier), aggregating binary votes by majority. Dynamic jury selection sharply reduces detection variance and increases accuracy (e.g., detection rate +22 points over random-majority baselines on Law-SG) (Nguyen et al., 20 May 2025). The DTDMR-LJGF paradigm realizes jury deliberation with multi-role agents (Judge, Prosecutor, Defense, Jurors) and optimizes for consistency-acceptability, tuning the compound objective (λ) to balance technical and social stakes (MingDa et al., 10 Jul 2025).

Structured Multi-Agent Deliberation

Protocols such as CollabEval implement three-phase evaluation: independent initial scoring, multi-round discussion, and final resolution by a "strong" final judge. Consensus thresholds are formally specified (exact, ε or half-point agreement), and empirical evaluation shows consistent improvements in label accuracy and error rates (Qian et al., 1 Mar 2026).

Security Evaluation

SkillSieve’s Layer 3 employs a three-LLM jury (different vendors) to judge AI agent skills, running up to two debate rounds if there is no unanimous verdict, then finalizing with a majority vote. Disagreement triggers structured debate, and unresolved splits escalate to human review. This reduces correlated error and enables robust detection even against adversarial evasion (Hou et al., 8 Apr 2026).

4. Theoretical Frameworks, Metrics, and Formal Guarantees

Metrics, statistical frameworks, and mathematical formalizations underpin jury composition and evaluation:

• Classic Probabilistic Modeling: The two-level jury model uses log-odds weighting, with judges estimating expert competence and aggregating via geometric/log-odds means. Optimality is established if the geometric mean of judges’ perceived odds matches the expert’s true odds (Abramowitz et al., 2022).
• BT-σ (Judge-Aware Bradley-Terry): This unsupervised ranking framework infers both item skills and judge reliabilities (σₖ) from pairwise judgments alone—automatically down-weighting unreliable judges. The learned reliability scores correlate with cycle consistency and judge-human correlation (Qian et al., 18 Feb 2026).
• Calibration: Isotonic regression calibrates jury outputs to human expert scales, correcting for systematic model leniency or strictness (Rouillard et al., 16 Apr 2026).
• Agreement Metrics: Inter-judge and jury–human agreement is characterized by Cohen's κ, Fleiss' κ, Spearman's ρ, Kendall's τ, and intraclass correlation (ICC), facilitating direct comparability to human panels (Reese et al., 20 Mar 2026, Rouillard et al., 16 Apr 2026, Amin et al., 30 Apr 2026).
• No-Knowledge Alarms: Juror consistency is monitored via integer linear programming; infeasibility signals that no possible ground truth could make all judges simultaneously meet their minimum accuracy requirements, providing a logically tight alarm for misalignment (Corrada-Emmanuel, 10 Sep 2025).

5. Operating Characteristics and Empirical Trade-offs

Protocol effectiveness is quantified by absolute and relative improvements on standard evaluation benchmarks. Core empirical findings include:

• Unanimous Juries: Strict unanimity sharply reduces false accepts (FPR) but decreases true positive acceptance (TPR), with optimal committees typically size 3 or 4. Composition (choice of LLMs) affects Youden’s J more than size once past basic thresholds (Ullah et al., 12 Feb 2026).
• Majority and Weighted Juries: Majority vote is efficient for dichotomous safety or acceptability criteria but can suffer ceiling effects if judges vary widely in calibration. Weighted reliability-based aggregation, as in Jury-on-Demand and BT-σ, empirically raises agreement with human raters, especially on difficult or ambiguous cases (Li et al., 1 Dec 2025, Qian et al., 18 Feb 2026).
• Dynamic Juries: Instance-level adaptation (e.g., AutoLaw, Jury-on-Demand) enables fine-grained exploitation of judge strengths, outperforming static panels or unweighted majority in both discrimination (AUC, τ) and variance reduction (Nguyen et al., 20 May 2025, Li et al., 1 Dec 2025).
• Bias and Safety: Rigorous assessments show that well-designed LLM juries do not exhibit self-preference bias, and systematic calibration plus outlier detection suppresses severe error probabilities below that observed in human panels (Rouillard et al., 16 Apr 2026).

6. Design Guidelines, Limitations, and Extensions

Recommendations for deploying robust multi-LLM jury protocols:

• Model Diversity: Pool LLMs from varied architectures, vendors, and alignment regimes to improve error coverage and mitigate shared blind spots.
• Aggregation Rule Selection: Align acceptance thresholds (unanimity for high safety, reliabilty-weighted average for discrimination) to application risk and calibration needs.
• Panel Validation: Always validate committee composition and thresholding on the production generator, especially for generator-specific idiosyncrasies (Ullah et al., 12 Feb 2026).
• Calibration and Bias Testing: Use isotonic regression, mixed-effects testing, and repeated-seed stability checks to ensure alignment with expert judgment, reject self-preference, and limit instability (Rouillard et al., 16 Apr 2026).
• Adaptivity: Instance-specific composition and weighting should be dynamically tuned for domains where judge reliability is variable across input space (Li et al., 1 Dec 2025, Nguyen et al., 20 May 2025).
• Monitoring and Logical Consistency: Employ no-knowledge alarms to detect and flag logically impossible accuracy claims without requiring ground truth (Corrada-Emmanuel, 10 Sep 2025).

Limitations include ceiling effects in ensemble improvement (jury may not outperform best expert judge in every scenario), cost scaling with the number of LLM calls, and the requirement of labeled pilot datasets for jury or reliability model calibration.

Potential extensions include weighted voting calibrated on per-criterion reliability (Reese et al., 20 Mar 2026), expansion to multi-round collaborative deliberation protocols (Qian et al., 1 Mar 2026), trade-off tuning between consistency and stakeholder acceptability in socially sensitive settings (MingDa et al., 10 Jul 2025), and proactive adaptation to adversarial input distributions (Nguyen et al., 20 May 2025, Hou et al., 8 Apr 2026).

7. Impact and Scope of Application

Multi-LLM jury protocols now underpin evaluation methodology across coding, natural language generation, clinical safety, legal compliance, security scanning, dialog evaluation, and high-stakes decision automation. They provide not only superior empirical correlation with human panels, but also fine-grained control over safety, reliability, calibration, and adaptation. Design choices—committee size, selection, aggregation rule, calibration—must be matched to domain risk profile and downstream tolerance for false positive and false negative rates.

Empirical consensus holds that, for safety-first deployments and high alignment requirements, protocolized multi-LLM juries offer a rigorous, auditable, and operationally scalable alternative to both single-judge LLM scoring and traditional rule-based or human-only review (Ullah et al., 12 Feb 2026, Reese et al., 20 Mar 2026, Nguyen et al., 20 May 2025, Li et al., 1 Dec 2025).