JudgeBoard: Robust Multi-Model Judgments

• JudgeBoard is a framework that aggregates model judgments using robust methods like the geometric median to ensure calibrated outputs.
• It uses diverse panels and statistical estimators to counteract bias, with applications spanning legal simulation, compliance monitoring, and automated evaluation.
• Its design emphasizes panel composition, calibration diagnostics, and domain-specific adaptations while addressing inter-judge dependencies and adversarial failures.

A JudgeBoard is a framework, architecture, or evaluation pipeline that aggregates judgments from multiple models—typically LLMs or small LLMs (SLMs)—to render robust, calibrated, and interpretable decisions across diverse domains such as automated evaluation, judicial simulation, compliance monitoring, and high-stakes adjudication. JudgeBoard implementations formalize the jury or panel paradigm, with explicit attention to aggregation functions, statistical failure modes, inter-judge dependencies, and the mapping between machine outcomes and principled decision-theoretic or legal standards.

1. Formal and Statistical Foundations

The statistical structure of a JudgeBoard is generally formulated as follows: Each evaluation instance (e.g., prompt–response pair) has an unobservable "true" score vector $y^*(x)\in [0,K]^d$, with $d$ rubric or decision dimensions. Each judge $f_i$ maps the instance to an observed output $\hat{y}_i\in\mathbb{R}^d$. Panels $\mathcal{J} = \{f_1,\ldots,f_N\}$ often share a rubric but differ in architecture, pretraining, or instruction format.

Classical aggregation by the arithmetic mean ("PoLL"—Panel of LLM Judges) is vulnerable to severe bias under the Huber $\epsilon$-contamination model: when even a single judge exhibits LLM-specific failure modes (mode collapse, sycophancy, safety refusal, heavy-tailed hallucination, cross-dimensional confusion), mean aggregation incurs unbounded conditional bias as adversarial or pathological outputs can dominate the consensus. This result holds regardless of panel size and is formalized by the bias formula

$$\mathbb{E}[\bar{y}|y^*] - y^* = \frac{1}{N}\sum_{i=1}^N \alpha_i(\mu_{Q_i} - y^*)$$

with $\alpha_i$ the per-judge contamination rate and $Q_i$ the adversarial distribution (Acharya et al., 29 Jun 2026).

Statistically principled alternatives, notably robust mean estimators (geometric median, trimmed mean, coordinate-wise median), are thus necessary to ensure resistance to minority but highly biased panel members. The geometric median is the unique tuning-free, multivariate estimator with maximal breakdown point $1/2$.

2. Robust Aggregation, Algorithms, and Guarantees

The RoPoLL (Robust Panel of LLM Judges) framework replaces the mean with the geometric median for panel aggregation. For inputs $d$0, the geometric median is

$d$1

This estimator is orthogonally and translation equivariant, tuning-free, and resistant to up to $d$2 corruptions. The Weiszfeld iteration provides an efficient, convergent method for its computation.

Finite-sample theoretical analysis yields the error bound: with contamination rate $d$3 and slack $d$4,

$d$5

where $d$6 and $d$7 for $d$8. The corresponding minimax lower bound, tight up to a factor of $d$9, confirms that this rate is optimal for polynomial-time estimators (Acharya et al., 29 Jun 2026).

3. JudgeBoard Design: Panel Composition and Calibration

Selection of the JudgeBoard panel and its calibration is critical:

• Diversity trumps scale: Empirical studies show that LLM/SLM panels exhibit high error correlation (mean pairwise $f_i$0 up to $f_i$1), resulting in Kish effective sample sizes $f_i$2 for 9-judge panels, independent of model family growth (Kohli, 28 May 2026). Information gain saturates quickly; $f_i$3 diverse judges capture nearly all available independence.
• Inter-judge correlation sharply limits achievable accuracy. The "Condorcet gap" (ideal independent-voter vs. actual accuracy) reaches 8–22 percentage points depending on task. Naive addition of models or sophisticated aggregation (weighted voting, Dawid-Skene EM) closes at most 11% of this gap (Kohli, 28 May 2026).
• Diagnostics and finite-calibration regime maps are essential for practical JudgeBoard deployment. For finite human-label budgets $f_i$4, model selection over (panel size $f_i$5, judge order $f_i$6, aggregator family $f_i$7) should be based on validation-set risk, estimation-complexity tradeoff (entropy-effective support $f_i$8), and marginal value of added judges. On current LLM outputs, low-dimensional or reliability-based aggregators dominate for most real tasks. Only in rare regimes with dense, high-order interactions and ample calibration labels do full joint tables outperform scalar models (Zhu et al., 31 May 2026).

4. Domain-Specific Architectures: Legal, Regulatory, and Debate JudgeBoards

JudgeBoard principles generalize beyond core evaluation:

• Judicial Deliberation: Multi-agent, role-diverse panels (e.g., AgentsBench, SAMVAD) structure JudgeBoard as collegial agents with explicit reasoning profiles: presiding judges, lay jurors, legal counsel, each with distinct prompting, argument exchange, and consensus dynamics. Retrieval-Augmented Generation (RAG) pipelines ground arguments in authoritative sources (e.g., statute books, case law), ensuring verifiable outputs and citation fidelity (Jiang et al., 2024, Devadiga et al., 4 Sep 2025).
• Compliance Monitoring: The govllm JudgeBoard for regulatory compliance assigns one specialized LLM per criterion (Profile-as-Jury), achieving higher veracity than unspecialized ensembles. Inter-judge disagreement (variance $f_i$9) is treated as a meaningful regulatory uncertainty signal, triggering human arbitration as required by the EU AI Act (Dussert, 23 May 2026).
• Debate Judging: Debatrix exemplifies multi-dimensional, iterative JudgeBoard architectures, with dimension-specific LLM judges operating over speeches in chronological order, synthesizing verdicts by weighted scoring and winner selection (Liang et al., 2024).

5. Performance, Best Practices, and Failure Modes

Empirical and theoretical findings converge on several best practices and cautionary points:

• Robust aggregation via RoPoLL (geometric median) eliminates catastrophic bias under adversarial or heavy-tailed corruption, dominating mean aggregation across diverse benchmarks and judge pools (Acharya et al., 29 Jun 2026).
• Jury-size ablation demonstrates a "knee" at $\hat{y}_i\in\mathbb{R}^d$0 judges due to high inter-judge correlation, and adding further judges of similar architecture or training offers negligible marginal information (Kohli, 28 May 2026).
• Calibration and diagnostics should always accompany panel construction. Marginal information gain, entropy-effective support, and unseen-pattern rate determine regime selection (scalar vs. joint or hybrid aggregators) (Zhu et al., 31 May 2026).
• Structural vulnerabilities persist: correlated error herding, prompt-architecture/position sensitivity, and pathologies such as sycophancy or mode collapse can compromise both performance and interpretability (Kohli, 28 May 2026, Dussert, 23 May 2026).
• Specialized panels ("Profile-as-Jury") empirically outperform naive averaging, especially in compliance and high-stakes settings (Dussert, 23 May 2026).

6. Application Domains, Limitations, and Future Directions

JudgeBoard architectures have been applied in reasoning evaluation (benchmarking SLMs/LLMs), legal judgment, digital justice simulation, debate adjudication, and regulatory compliance. Multi-agent orchestration, retrieval-grounded deliberation, and transparent logging are increasingly standard for high-trust settings (Bi et al., 20 Nov 2025, Jiang et al., 2024, Devadiga et al., 4 Sep 2025, Dussert, 23 May 2026).

Despite rapid progress, several limits remain: LLM-as-a-judge evaluations often overrate fluency, underdetect substantive flaws, and cannot yet replace experts for high-complexity, logic-heavy drafting (e.g., formal legal judgments) (Karp et al., 6 Nov 2025). RAG and fine-tuning partially close these gaps, but continuous human oversight, interdisciplinary rubric design, and ongoing corpus updates are mandatory for reliable, real-world deployment.

Key research directions for JudgeBoard include automated judge profiling, adaptive debate/deliberation protocols, hybrid human–LLM adjudication loops, and meta-learning "meta-judges" from multi-agent consensus (Bi et al., 20 Nov 2025, Jiang et al., 2024). The transition from static, audit-time verdicts to continuous, observable governance marks a foundational shift aligned with legal and societal requirements for AI decision accountability (Dussert, 23 May 2026).