Stopping and Routing LLM Judge Panels

Abstract: LLM evaluation pipelines often have many candidate judges: general LLM-as-a-judge prompts, reward models, safety classifiers, confidence variants, and task-specific verifiers. The deployment question is not only which judge is best, but which judges should be called, on which examples, and when panel construction should stop. We formulate judge-panel design as a role-conditioned allocation problem. From a small labeled audit set, declared slices, and judge costs, the method estimates target-relative roles: copies add no conditional information, complements improve the global panel, and specialists help only on slices. These roles induce a policy: drop copies, add complements globally, route specialists conditionally, and stop when validation gain falls below a threshold. Across reasoning, code, safety, preference, reward-model, summarization, and math audits, the method is compared with single judges, flat panels, matched diversity heuristics, full-call stacking, reliability juries, and frugal cascades. The result is a regime map for judge calls: route specialists on deployable slices, stop in saturated verifier regimes, keep broad ensembles when their risk benefit is worth the cost, and ignore conditional copies. The output is a reusable, auditable call plan for the next evaluation batch.

• The paper introduces a role-conditioned allocation framework that classifies judges as copies, complements, or specialists and uses validation-based stopping to build cost-aware evaluation policies.
• Experiments across eight benchmarks show that selective routing can approach full-panel accuracy at substantially lower cost, including 0.9900 accuracy on MBPP with 1.52 calls and near-best JBB safety risk with roughly one-third as many calls.
• The framework provides practical stopping reports and resists redundant judge copies, but its guarantees depend on audit data, predefined route signals, finite pattern calibration, and single-judge greedy search.

Motivation and problem statement

Modern LLM evaluation pipelines accumulate a heterogeneous pool of candidate judges: general LLM-as-a-judge prompts, rubric variants, reward models, safety classifiers, confidence-conditioned outputs, and deterministic verifiers. The operational question facing an evaluation owner is not which judge ranks highest in isolation, but which judges to actually invoke, on which examples, and when panel construction should terminate. Static judge leaderboards cannot answer this, because a judge's value is conditional on the current panel composition, the target distribution, and the slice of examples where it would be deployed. Zhu, Xie, and Rao formalize this as a role-conditioned allocation problem: from a small labeled audit set, declared slices, and per-judge costs, the method classifies each candidate as a copy (no conditional information), a complement (broad residual information), or a specialist (slice-concentrated value), and converts that classification directly into a calling policy with validation-based stopping (2608.19802).

Conditional gain and the role taxonomy

The framework fixes a target distribution $P$ over evaluated items with audit labels $Y \in [0,1]$ and defines the oracle risk $R^\star_{P,S}$ of a panel $S$ via the conditional expectation $\eta_{P,S}(z) = E_P[Y \mid Z_S = z]$. The conditional value of adding judge $j$ is the risk reduction

$$g_P(j \mid S) = R^\star_{P,S} - R^\star_{P,S\cup\{j\}},$$

which a projection-gain identity shows equals the expected squared difference between the two oracle predictors, requiring no independence assumption among judges. Roles are then defined target-relatively: broad gain $C_P(j\mid S)$ against the global distribution and slice gains $A_f(j\mid S)$ against declared slices $f$. The profile is deliberately multi-label — a judge can be both a broad complement and a slice specialist, and its role can change after another judge enters the panel. A specialization ratio $Y \in [0,1]$0 flags concentration of value without forcing mutually exclusive roles.

Construction is greedy over cost-adjusted validation gain. Each audit set is split into construction-fit, construction-validation, and final-test portions; pattern calibrators estimate $Y \in [0,1]$1 by cell means over canonicalized joint judge-output patterns, with unseen patterns falling back to the fit-split label mean. A candidate enters the global panel only if its cost-discounted validation gain exceeds $Y \in [0,1]$2; after global stopping, each slice runs the same search with the global panel fixed. The output is a policy $Y \in [0,1]$3 plus a stopping report enumerating every failed threshold inequality — an operational certificate that no remaining single call is justified under the audit split, thresholds, and cost model. The paper is explicit that deployable route signals must be computable before the routed call (metadata, verifier outputs, classifier outputs, observed disagreement); human-label strata are valid only for audit analysis.

Experimental protocol

The evaluation spans eight non-saturated settings chosen so no single narrative explains all results: hard GSM8K rationale audits, MBPP public-test overfit audits, JailbreakBench safety, LLMBar preference under DeepSeek/Qwen3/JudgeLM anchors, RewardBench, Arena100K, SummEval scalar judging, and MATH-500, with HumanEval and ordinary GSM8K reserved as saturated stopping checks. The judge pool comprises Qwen2.5 7B, Llama 3.1 8B, Mistral v0.3 7B, Prometheus 2, Gemma 3 IT 12B, Atla Selene Mini, and DeepSeek V4 Flash, with normalized costs of 1.0 for LLM calls and 0.1 for verifiers. Baselines cover single-best judges, flat all-call panels, matched-$Y \in [0,1]$4 top-$Y \in [0,1]$5/correlation/quality-diversity panels, full-call ridge/logistic stacking, Dawid–Skene-style reliability juries, and FrugalGPT/RouteLLM-style confidence cascades. All results average 10 random splits with 95% confidence intervals; selection uses validation data only, and reported numbers come exclusively from held-out final-test splits.

Results: a regime map rather than a universal winner

The central empirical claim is not that role allocation dominates everywhere, but that it identifies the correct deployment action per regime:

Regime	Setting	Role policy outcome
Specialist routing	LLMBar-7	Risk 0.1884 / acc. 0.7334 at 3.46 calls vs. flat-all 0.2118 / 0.6692 at 7 calls
Verifier-first + selective adds	MBPP public-overfit	Risk 0.0097 / acc. 0.9900 at 1.52 calls vs. flat-all 0.9767 at 6.10 calls
Few-judge complements	Hard GSM8K rationale	Acc. 0.6843 at ~2.9 calls vs. single-best 0.6253
One-step stop	Arena100K, SummEval	Keeps the strong single judge (cost 1.00); expansion adds nothing
Broad-ensemble boundary	RewardBench, MATH-500	Full-call stacking wins on risk (0.0201 vs. 0.0291; 0.0536 vs. 0.0678); role policy is cheaper

On JBB safety, routing on the deployable gpt4_cf classifier proxy reaches risk 0.1094 at 2.29 calls versus 0.1069 for the 7-call stack — nearly matching the best endpoint at roughly one third of the cost. Matched-budget comparisons isolate what role conditioning contributes beyond nominal diversity: top-$Y \in [0,1]$6, correlation-diverse, and quality-diverse panels spend similar budgets but select by standalone quality or pairwise correlation, and lose on MBPP, JBB, and all three LLMBar anchors. Reliability juries and frugal cascades fail precisely where information is slice-conditional, since neither models specialists.

Three mechanism audits strengthen the causal reading. Copy stress test: injecting four exact copies of an existing judge leaves role risk and cost unchanged (LLMBar 0.1884/3.46; JBB 0.1094/2.29), while reliability jury risk degrades sharply (LLMBar 0.2058 → 0.2860; JBB 0.1382 → 0.1594) because duplicated votes are overweighted. Threshold sensitivity: raising $Y \in [0,1]$7 monotonically reduces calls (e.g., JBB from 3.68 to 1.00), and on LLMBar a more conservative threshold actually improves risk to 0.1834 by avoiding sparse expansions — making $Y \in [0,1]$8 a transparent risk-cost dial rather than a tuned hyperparameter. Near-duplicate prompt variants: calling all ten JSON-schema/letter-prompt variants raises LLMBar risk to 0.2633, yet the role-routed policy reaches 0.7143 accuracy at 3.64 calls, exceeding the full-call logistic stack's 0.6862 — prompt variants should be routed conditionally, neither bought wholesale nor deleted wholesale.

Deployment robustness audits add practical caveats. Pattern-table fallback is concentrated in routed specialists (4.4% test fallback on LLMBar routes). Halving the audit set preserves risk and cost but collapses plan Jaccard similarity (e.g., 0.246 on LLMBar), so exact plan identity requires more labels even when performance is stable. Slice-mix shift experiments show asymmetric transfer: adversarial-heavy LLMBar construction transfers cleanly to natural-heavy batches, but natural-heavy construction does not support adversarial-heavy deployment. Pair-addition audits find zero pair-only misses on LLMBar and MBPP but 3 on JBB and 2 in the prompt-variant pool, meaning the greedy stopping report certifies only single-addition optimality unless the search space is widened.

Limitations and open questions

The paper concedes several boundaries plainly. The calibrator is a finite pattern table whose cell means degrade under sparsity; the authors propose smoothed or cross-fitted estimators as extensions but do not evaluate them. The stopping guarantee is operational, not statistical — it holds only under the finite audit split, declared slices, thresholds, and cost model, and the frequency audit for stable routes is explicitly "not a statistical guarantee." Route keys must be pre-available; any model call needed to compute a route must be added to the cost model, and the paper does not study learned route predictors. Complete-case filtering excludes unparseable rows (158 of 838 in the prompt-variant audit), so parse robustness is outside scope. Whether pairwise or higher-order complementarity matters systematically — beyond the small pair-only counts observed on JBB — remains open, as does automatic slice discovery validated end-to-end within this allocation objective.

Conclusion

This work reframes judge-panel design from descriptive diversity measurement into auditable call planning. Target-relative conditional gain yields copy/complement/specialist roles; validation-stopped greedy construction turns those roles into global panels, routed specialists, and explicit stopping reports; held-out evaluation across eight settings produces a regime map distinguishing when to route, stop, drop copies, or pay for full-call aggregation. The strongest quantitative results — near-full-stack safety risk at one-third cost, 0.9900 accuracy at 1.52 calls on code audits, and immunity to injected redundant copies that corrupt jury baselines — support the paper's core position: panel size is neither a reliability nor an efficiency guarantee, and future evaluations should report which additional judges were worth calling, where, and why construction stopped.