Retrieval-Based Multi-Agent Judge Framework

• The paper introduces a retrieval-based multi-agent judge framework that integrates evidence retrieval, multi-agent debate, and score aggregation to evaluate model outputs.
• It employs specialized agents that retrieve domain-specific evidence and exchange critiques, thereby mitigating biases inherent in single LLM evaluators.
• Empirical results across domains show improved accuracy and alignment with human judgments, though increased latency and cost remain key challenges.

A retrieval-based multi-agent judge framework is a synthesized class of evaluation systems in which multiple LLM-based judge agents assess model outputs or agent trajectories while grounding their assessments in retrieved evidence, tool outputs, domain documents, or execution traces. The exact term is not defined as such in the review literature, but its constituent elements are explicitly present: agent-as-a-judge as a formal paradigm, multi-agent debate and committee architectures, and retrieval- or tool-augmented evaluators. Several systems instantiate this pattern in translation, legal question answering, FAQ annotation, analytic essay scoring, fact verification, and judicial decision-making (Yu, 5 Aug 2025, Chen et al., 2024, Wang et al., 31 Aug 2025, Hegazy et al., 19 May 2025, Keramati et al., 4 Jun 2026, Jeong et al., 8 Jan 2026, He et al., 2024).

1. Conceptual foundations and formalization

The broad agent-as-a-judge paradigm distinguishes a single static evaluator from an autonomous evaluator endowed with tools, memory, and multi-step reasoning. For a task input $x$, a system under evaluation $M$, and an output $y=M(x)$, a judge can be written as $J(x,y)\in\mathbb{R}^k$, where $k$ may denote a scalar overall score or a multi-dimensional score vector. In the agentic setting, the object of evaluation can be the full interaction trajectory

$\tau=(s_0,a_0,o_0,s_1,a_1,o_1,\dots,s_T,a_T,o_T),$

and the judge itself becomes an agent whose output is a function of trajectory, tools, and guidelines: $$J(\tau)=g_\phi(\tau,\text{tools},\text{guidelines})\in\mathbb{R}^k.$$ The retrieval-conditioned extension adds explicit evidence $D$, usually obtained by a retrieval function $R$, so that scoring takes the context-aware form $s(x,y,D)=f_\theta(x,y,D)$ (Yu, 5 Aug 2025).

A retrieval-based multi-agent judge framework, in this synthesized sense, therefore combines three ideas. First, judging is no longer restricted to the final string $M$0; it may inspect trajectories, tool calls, intermediate states, or debate transcripts. Second, multiple judges $M$1 interact under an explicit protocol and aggregate their outputs. Third, each judge may retrieve its own evidence $M$2, where the query $M$3 depends on the task, the candidate output, the trajectory, and the judge’s role or persona. This suggests a generic retrieval-aware interaction step of the form

$M$4

followed by per-agent scoring and an aggregation rule $M$5 (Yu, 5 Aug 2025).

The formal distinction between single-judge and agentic judge systems matters because retrieval changes both the epistemic basis and the failure profile of evaluation. A single LLM judge can rely heavily on priors or stylistic cues; a retrieval-grounded judge is expected to cite or at least condition on evidence. Multi-agent organization adds another axis: a committee or debate can expose disagreements, missing evidence, or overconfident reasoning that a single evaluator would not surface.

2. Canonical architectures and agent roles

Across the literature, recurring architectures separate retrieval, reasoning, criticism, and aggregation into distinct agents. Some are lightweight committees; others are full stateful workflows with iterative search and verification. The most common roles are a primary grader or scorer, an adversarial critic, a defender or supporting advocate, a retriever or search agent, and a meta-judge or aggregator (Yu, 5 Aug 2025).

System	Agents	Judge function
CRAT	Unknown Terms Detector, Knowledge Graph Constructor, Causality-enhanced Judge, Translator	Filters retrieved documents as [CORRECT] or [INCORRECT]
L-MARS	Query Agent, Search Agent, Judge Agent, Summary Agent	Decides SUFFICIENT or INSUFFICIENT and issues refinement queries
MAFA	Query Planning Agent, four FAQ Mapper agents, Judge Agent Reranker	Reranks pooled FAQ candidates
MADRAG	Advocate, Skeptic, Judge, Supervisor	Aggregates debate using rubric-aligned exemplar retrieval
Tool-MAD	RAG debater, Search debater, Judge	Uses debate history plus faithfulness and answer relevance

CRAT is a concrete retrieval-based multi-agent judge architecture specialized for translation. It comprises an Unknown Terms Detector $M$6, a Knowledge Graph Constructor $M$7, a Causality-enhanced Judge $M$8, and a Translator $M$9. The pipeline detects uncertain terms, retrieves internal and external knowledge, judges each retrieved document against the contextual knowledge graph, and forwards only vetted evidence to generation (Chen et al., 2024). L-MARS adopts a state-driven workflow using LangGraph: a Query Agent structures the problem, a Search Agent queries heterogeneous sources, a Judge Agent checks sufficiency, jurisdiction, temporal validity, and contradictions, and a Summary Agent synthesizes the final answer (Wang et al., 31 Aug 2025).

MAFA shows the same logic in ranking form rather than free-form generation. A Query Planning Agent expands the query; four specialized FAQ Mapper agents provide diverse candidate lists; and a Judge Agent Reranker performs meta-reasoning over the union of candidates, original agent scores, and rationales (Hegazy et al., 19 May 2025). MADRAG decomposes analytic essay scoring into trait-wise debates between an Advocate and a Skeptic, with a Judge that reads the debate transcript and retrieved scored exemplars before assigning an integer score (Keramati et al., 4 Jun 2026). Tool-MAD distributes retrieval asymmetrically: one debater uses RAG over a static vector store, another uses web or domain-specific search, and the Judge resolves the debate using both the evidence trail and quantitative grounding scores (Jeong et al., 8 Jan 2026).

A plausible abstraction is that the “judge” in these systems is not always a single node. In some frameworks it is the gatekeeper between raw retrieval and generation; in others it is the controller of an iterative search loop; in still others it is a reranker or final adjudicator over argument sets. The common feature is that evaluation authority is distributed across specialized roles and then reassembled.

3. Retrieval, grounding, and evidence curation

Retrieval is the mechanism that turns a judge from a prior-based evaluator into an evidence-aware evaluator. The surveyed literature emphasizes several concrete retrieval patterns: claim-level evidence checking, tool-mediated validation, heterogeneous source orchestration, and adaptive query reformulation (Yu, 5 Aug 2025).

CRAT offers a minimal but explicit design. The Knowledge Graph Constructor extracts internal knowledge from the source context, retrieves external documents $y=M(x)$0, and passes each pair $y=M(x)$1 to the Causality-enhanced Judge. The judge returns a discrete confidence label in $y=M(x)$2; only items labeled [CORRECT] are added to external knowledge $y=M(x)$3 and exposed to the Translator (Chen et al., 2024). This enforces a two-stage filter: term-level retrieval is triggered only for flagged unknown terms, and retrieved evidence is then contractively filtered by the judge.

L-MARS generalizes retrieval beyond a single corpus. Its Search Agent orchestrates Serper web search, local RAG with BM25 over markdown files, and CourtListener case law. The Judge Agent evaluates not only factual support but also source quality, date or temporal validity, jurisdiction match, and contradictions between sources, and it returns concrete new search queries when evidence is insufficient (Wang et al., 31 Aug 2025). Tool-MAD extends this logic to debate: each debater has a different tool, and queries are reformulated after each round using the opponent’s latest answer. This makes retrieval adaptive to argumentative state rather than static over the initial question (Jeong et al., 8 Jan 2026).

MAIN-RAG shifts the judge into document filtering. A Predictor generates a candidate answer for each retrieved document, and a Judge evaluates the resulting document-question-answer triplet with a binary “Yes” or “No” decision. The framework converts this into a numeric relevance score

$y=M(x)$4

then computes an adaptive threshold from the score distribution and keeps only documents above that bar (Chang et al., 2024). This is still a judge framework, but the judged object is the evidence pool rather than only the final answer.

The broader survey draws the same conclusion in more general terms: judges should ground their verdicts in documents, logs, or tools rather than pure priors; claim-level evaluation is often more reliable than global “looks factual” scoring; and evidence-aware, domain-specific panels tend to align better with human judgment than generic ungrounded evaluators (Yu, 5 Aug 2025).

4. Deliberation, aggregation, and scoring objectives

Once multiple judges are present, the decisive design question becomes how they interact and how their outputs are fused. The surveyed patterns include group discussion, adversarial deliberation, stakeholder-group debate, judge reranking, and exemplar-calibrated aggregation (Yu, 5 Aug 2025).

The survey describes a general multi-agent ensemble $y=M(x)$5 together with an interaction protocol and aggregation rule. In ChatEval-style setups, agents with different personas debate for several rounds and then produce scores that are aggregated by mean or vote. CourtEval uses a Grader, Critic, and Defender, with the judge revising an initial score after hearing both prosecution-like and defense-like arguments. DEBATE uses a Scorer, Critic, and Commander. MAJ-EVAL groups agents by stakeholder and then aggregates group-level outputs, often with stakeholder weights: $y=M(x)$6 This makes evaluation explicitly multi-dimensional and stakeholder-conditioned (Yu, 5 Aug 2025).

MAJ-EVAL operationalizes this by deriving stakeholder roles and evaluation dimensions from external domain documents and then instantiating personas that debate within their stakeholder groups. It reports improved agreement with human experts in both educational and medical evaluation, particularly on domain-specific dimensions such as Educational Appropriateness, Effect Direction, and Evidence Strength (Chen et al., 28 Jul 2025). MAFA uses an LLM meta-ranker rather than a numeric voting rule: candidate FAQ lists from multiple agents are pooled, deduplicated, and reranked by a judge agent that sees the original scores, rationales, and full FAQ content (Hegazy et al., 19 May 2025).

MADRAG illustrates a more tightly calibrated scoring objective. For each essay trait, the Judge receives the trait rubric, the debate transcript $y=M(x)$7, first-token confidence proxies from the Advocate and Skeptic, and a retrieved set of rubric-aligned exemplars that span the score range. It then outputs a single integer score

$y=M(x)$8

where $y=M(x)$9 bundles trait context, retrieval, debate, and confidences (Keramati et al., 4 Jun 2026). This shows a specific way in which retrieval alters aggregation: retrieved exemplars anchor the judge’s score scale, while debate supplies pro and con evidence.

A plausible implication is that retrieval-based multi-agent judging is best understood not as “more judges” but as a coupling of three layers: evidence construction, adversarial or plural reasoning over that evidence, and an aggregation mechanism that is itself evidence-sensitive.

5. Empirical instantiations across domains

Reported systems support two recurring empirical claims. First, retrieval alone often improves over a plain baseline. Second, retrieval plus an explicit judge, or retrieval plus multi-agent deliberation, often improves further. The gain is task-dependent and usually comes with higher latency or cost (Chen et al., 2024, Wang et al., 31 Aug 2025, Hegazy et al., 19 May 2025, Keramati et al., 4 Jun 2026).

System	Domain	Reported result
CRAT	Translation	Qwen-72B ablation: BLEU 29.9→30.4→30.8; COMET 83.0→83.9→84.6; CONSIS 81.7→83.6→85.0
L-MARS	Legal QA	Accuracy 0.96 in Simple mode and 0.98 in Multi-Turn mode; U-Score 0.42 and 0.39
MAFA	FAQ annotation	Bank dataset Top-1 0.355 vs 0.270 for best single agent; without Judge Agent 0.275
Tool-MAD	Fact verification	GPT-4o-mini average EM 71.0 vs MADKE 68.0 and MAD 52.9
MADRAG	Essay scoring	Set 7 extreme-score subset: Agree@1 0.795 and MAE 1.001
MAJ-EVAL	Education / medicine evaluation	Spearman $J(x,y)\in\mathbb{R}^k$0 up to 0.47 on StorySparkQA and 0.39 on MSLR-Cochrane
AgentsCourt	Judicial decision-making	Legal grounds F1 0.203 in first instance and 0.277 in second instance

CRAT’s ablation isolates the judge cleanly. A vanilla Qwen-72B baseline scored 29.9 BLEU, 83.0 COMET, and 81.7 CONSIS; adding unrefined TransKG raised these to 30.4, 83.9, and 83.6; adding the Causality-enhanced Judge produced 30.8, 84.6, and 85.0. The largest additional gain appears on CONSIS, which is explicitly targeted at semantic fidelity and consistency of unknown terms (Chen et al., 2024). MAFA shows the same pattern in ranking: on the bank dataset, the full specialized framework reached Top-1 accuracy 0.355, whereas removing the Judge Agent reduced Top-1 to 0.275 (Hegazy et al., 19 May 2025).

L-MARS demonstrates the cost-quality trade-off clearly. On LegalSearchQA, pure GPT-4o achieved 0.89 accuracy and 0.55 U-Score; L-MARS Simple reached 0.96 and 0.42; L-MARS Multi-Turn with the Judge Agent reached 0.98 and 0.39, but average time increased from 13.62 seconds to 55.67 seconds (Wang et al., 31 Aug 2025). Tool-MAD reports analogous improvements in fact verification: with GPT-4o-mini, average exact match rose from 68.0 for MADKE to 71.0 for Tool-MAD, and the paper reports up to 5.5% accuracy improvement over prior MAD frameworks (Jeong et al., 8 Jan 2026).

The review literature places these domain systems in a broader pattern. GPT-4-class single judges often achieve Spearman correlations around $J(x,y)\in\mathbb{R}^k$1–$J(x,y)\in\mathbb{R}^k$2 with aggregate human preferences on dialogue and summarization, and multi-agent evaluators such as ChatEval, CourtEval, DEBATE, and MAJ-EVAL report roughly $J(x,y)\in\mathbb{R}^k$3–$J(x,y)\in\mathbb{R}^k$4 relative improvements in correlation with human judgments over single-judge baselines. In a structured code-agent setting, Agent-as-a-Judge differed from majority human vote only about $J(x,y)\in\mathbb{R}^k$5 of the time, versus about $J(x,y)\in\mathbb{R}^k$6 disagreement for a single LLM judge (Yu, 5 Aug 2025).

6. Reliability, failure modes, and research directions

The same literature that motivates retrieval-based multi-agent judges also identifies systematic risks. Single judges exhibit style bias, length bias, self-model bias, and cognitive biases such as anchoring and confirmation. Multi-agent deliberation can mitigate some of these but can also produce mode collapse, collusion, or persuasion bias, especially when weaker agents defer to stronger or more eloquent ones (Yu, 5 Aug 2025).

Retrieval changes the failure profile rather than eliminating it. The review explicitly notes that retrieval does not solve adversarial or cherry-picked evidence, domain shift, or corpus staleness. If a judge retrieves biased, outdated, or incomplete material, those defects propagate into the verdict. Tool-MAD addresses this partly by integrating faithfulness and answer relevance scores into a “stability score,” with thresholds $J(x,y)\in\mathbb{R}^k$7 and $J(x,y)\in\mathbb{R}^k$8, so that low-grounded or off-topic rounds are not accepted prematurely (Jeong et al., 8 Jan 2026). MAIN-RAG addresses another dimension by using adaptive filtering thresholds derived from the relevance-score distribution, attempting to balance noise suppression and recall (Chang et al., 2024).

A more severe vulnerability appears when the judge itself becomes deceptive or adversarial. The analysis in "Helpful Agent Meets Deceptive Judge" introduces a taxonomy spanning intent and knowledge access and constructs WAFER-QA, with 574 contextual and 708 non-contextual examples containing critiques grounded in retrieved web evidence. Under grounded malicious judges, most high-end models except the latest o4-mini suffered performance drops of over 50% from Acc@R$J(x,y)\in\mathbb{R}^k$9 to Acc@R$k$0 (Ming et al., 3 Jun 2025). This directly challenges the assumption that grounding alone guarantees trustworthy judging.

Meta-evaluation is therefore a central open problem. The survey lists meta-evaluation, robustness under adversarial examples, domain shift, tool and retrieval orchestration, and complexity or cost as pressing research directions (Yu, 5 Aug 2025). JudgeAgent proposes one validation strategy: if an evaluator’s feedback is meaningful, then supplying its suggestions should improve target-model performance. On MedQA, for example, GLM4-Flash improved from ACC1 80.09 to ACC2 92.82, with CR 13.46 and CtE 0.73, after receiving JudgeAgent’s suggestions (Shi et al., 2 Sep 2025). This suggests that a judge framework should be assessed not only by agreement with static labels, but also by whether its critiques are actionable and non-destructive.

The cumulative picture is therefore neither replacementist nor purely skeptical. Retrieval-based multi-agent judge frameworks can deliver finer-grained, better-grounded, and often more human-aligned evaluation than single ungrounded judges, but they remain bounded by the quality of their retrieval, the robustness of their interaction protocol, and the validity of their own meta-evaluation. The review literature’s position is correspondingly cautious: agent-based judging can complement, but not replace, human oversight (Yu, 5 Aug 2025).