Agent-as-Judge Evaluation Paradigm

• Agent-as-Judge is an evaluation paradigm that employs autonomous agents to plan, gather evidence, verify processes, and assess complex, multi-step tasks.
• It decentralizes evaluation by enabling interactive, multi-agent deliberation that overcomes the bias and limitations of static, single-pass LLM judges.
• The paradigm demonstrates enhanced alignment with human judgment and robust verification across diverse domains, from software development to social simulations.

Agent-as-Judge denotes an evaluation paradigm in which the evaluator is itself an agentic system rather than a single-pass LLM. In this formulation, the judge can plan, act, verify, remember, collaborate, and inspect intermediate states, tool traces, or environments before issuing a score, ranking, or decision about another model or agent (You et al., 8 Jan 2026, Yu, 5 Aug 2025). The paradigm is motivated by the claim that contemporary evaluands are increasingly multi-step, tool-using, multimodal, and domain-specialized, so evaluation becomes a sequential process rather than a static prompt-response mapping (Zhuge et al., 2024, You et al., 8 Jan 2026).

1. Conceptual lineage and formal framing

The literature places Agent-as-Judge within a broader historical progression from traditional automatic metrics, to single LLM-as-a-Judge, to multi-agent judges, and finally to agent-as-a-judge systems that inspect trajectories, use tools, and evaluate process as well as outcome (Yu, 5 Aug 2025). Survey work characterizes this transition as a response to three structural limitations of LLM-as-a-Judge: bias, shallow single-pass reasoning, and lack of real-world verification (You et al., 8 Jan 2026).

In formal treatments of LLM-as-a-Judge, the evaluator is abstracted as

$$\mathcal{E} \leftarrow \mathcal{P}_{\mathcal{LLM}}(x \oplus \mathcal{C}),$$

where $\mathcal{E}$ is the evaluation output, $x$ is the object being evaluated, and $\mathcal{C}$ is the evaluation context (Gu et al., 2024). Agent-as-Judge extends this by turning evaluation into a process with planning, evidence gathering, sub-agent delegation, verification, memory update, and final judgment (You et al., 8 Jan 2026).

A major survey organizes the field into three stages of agency: procedural Agent-as-a-Judge, where evaluation follows predefined workflows or fixed multi-agent protocols; reactive Agent-as-a-Judge, where the judge can route execution, invoke tools, or call sub-agents depending on intermediate evidence; and self-evolving Agent-as-a-Judge, where the judge can refine internal evaluation components such as rubrics or memory during operation (You et al., 8 Jan 2026). This taxonomy frames the paradigm not as a single architecture but as a continuum of increasingly autonomous evaluators.

2. Core capabilities and system architecture

The paradigm shift is commonly described along several dimensions: robustness moves from monolithic to decentralized evaluation, verification moves from intuition to execution, granularity moves from global scoring to fine-grained assessment, agency moves from fixed workflow to adaptive planning, statefulness moves from stateless inference to memory, and learning moves from prompt engineering to optimization (You et al., 8 Jan 2026). These dimensions recur across otherwise heterogeneous judge systems.

A proof-of-concept code-evaluation system makes these capabilities explicit through eight modules: graph, locate, read, search, retrieve, ask, memory, and planning (Zhuge et al., 2024). The same paper argues that agentic judges differ from LLM-as-a-Judge because they can actively gather evidence from project structure, files, outputs, and trajectories rather than evaluate only what fits into a prompt window (Zhuge et al., 2024).

Environment-aware variants make agency central to verification. AJ-Bench defines Agent-as-a-Judge as a verifier that can interact with environments and tools to acquire missing information, verify environment state, and verify whether the process itself was correct; its benchmark spans search, data systems, and graphical user interfaces (Shi et al., 20 Apr 2026). Online Agent-as-a-Judge goes further by embedding the evaluator inside the same simulation world as the target agent. In that setting, the judge acts through the native dialogue/action protocol, uses read-only tools for scene and interaction state, and iterates through an Inspect → Plan → Elicit/Run → Observe → Decide loop until enough non-judge evidence exists for a verdict (Ryu et al., 6 Jun 2026).

These systems share a common architectural thesis: verification quality depends not only on reasoning sophistication, but on the judge’s ability to obtain the right evidence under the right interaction constraints. This suggests that, in agentic settings, evaluation is itself an active information-acquisition problem.

3. Rubrics, decomposition, and multi-agent deliberation

A defining feature of Agent-as-Judge systems is structured decomposition. In DevAI, the judge evaluates each requirement separately as <SATISFIED> or <UNSATISFIED>, rather than reducing an entire task to a single terminal score (Zhuge et al., 2024). The multilingual localization study on the same benchmark preserves this requirement-level format, with tasks containing multiple requirements $R_0, R_1, \dots$ and judges returning SAT or UNSAT per requirement (Mahmood et al., 6 Apr 2026).

Tree-structured rubrics are another common design. Mind2Web 2 constructs a task-specific judge agent for each long-horizon web-browsing task, using a rubric tree whose leaf nodes are binary checks and whose internal nodes recursively aggregate correctness and source attribution (Gou et al., 26 Jun 2025). PentestJudge likewise collapses penetration-testing evaluation into a tree of operational objectives, operational security, and tradecraft requirements, with binary judgments at the leaves and weighted averaging over subtrees (Caldwell et al., 4 Aug 2025). These designs replace holistic impressionistic grading with hierarchical verification over smaller, operationally meaningful units.

Multi-agent deliberation introduces a second axis of structure. MAJ-EVAL first constructs evaluator personas from task-relevant documents, then instantiates stakeholder agents that debate within groups before an aggregator synthesizes qualitative feedback and a numerical final score (Chen et al., 28 Jul 2025). CollabEval implements a three-phase collaborative process—initial evaluation, multi-round discussion, and final judgment—with strategic consensus checking and a maximum of three discussion rounds (Qian et al., 1 Mar 2026). In both cases, the judge is not a lone evaluator but a coordinated panel intended to reduce single-model bias and broaden evaluative coverage.

Reasoning-trace evaluation makes decomposition even finer-grained. "The Confident Liar" (Keramati et al., 9 Jun 2026) studies a debate architecture with a Constructor, an Auditor, and a Synthesizer, and applies an LLM judge only to the Constructor and Auditor. The judge reconstructs prompt context and scores responses on three 3-point dimensions—Instruction Following, Justification Quality, and Evidence Grounding—plus a binary critical-failure flag. The aggregate score is

$$Q_1 = s_{\text{instruction}} + s_{\text{justification}} + s_{\text{evidence}},$$

with critical failures overriding the score to $Q=0$ (Keramati et al., 9 Jun 2026). This formulation exemplifies Agent-as-Judge as process diagnostics rather than answer-only assessment.

4. Evaluation objects, domains, and benchmark ecosystems

The domain range of Agent-as-Judge is unusually broad. In software and development, DevAI provides 55 realistic automated AI development tasks with 365 hierarchical requirements and 125 preferences, explicitly designed as a proof-of-concept testbed for agentic evaluation (Zhuge et al., 2024). The multilingual localization study evaluates the same 55 development tasks across three developer-agent frameworks, five languages, and six judge backbones, totaling 4,950 judge runs (Mahmood et al., 6 Apr 2026). Auto-Eval Judge extends the setting to general task-completion evaluation, judging the Magentic-One Actor Agent on GAIA and BigCodeBench through checklist generation, log retrieval, and evidence-aware verification (Bhonsle et al., 7 Aug 2025).

Environment-aware evaluation has become a major subfield. AJ-Bench comprises 155 tasks and 516 annotated trajectories across search, data systems, and graphical user interfaces, explicitly targeting information acquisition, state verification, and process verification (Shi et al., 20 Apr 2026). Mind2Web 2 introduces 130 realistic, high-quality, long-horizon web tasks that require real-time browsing and citation-backed synthesis, and evaluates both correctness and source attribution with task-specific judge agents (Gou et al., 26 Jun 2025).

Interactive and social settings require yet another formulation. Online Agent-as-a-Judge uses 32 designer-authored social criteria across eight domains in a five-character family life-simulation sandbox, with labels in $\{pass, fail, insufficient\}$ determined only after active situation generation and evidence collection (Ryu et al., 6 Jun 2026). SAGE, or Sentient Agent as a Judge, evaluates higher-order social cognition by simulating a user with persona, background, goals, hidden intention, and an evolving emotion state, then using the final emotion score $e_T$ as the main metric over 100 supportive-dialogue scenarios across eight topics (Zhang et al., 1 May 2025).

Domain-specific professional applications are similarly diverse. MAJ-EVAL targets educational and medical evaluation with stakeholder-grounded personas (Chen et al., 28 Jul 2025). FinResearchBench evaluates financial research agents through a logic-tree intermediate representation over 70 questions spanning seven task types (Sun et al., 22 Jul 2025). PentestJudge evaluates penetration-testing trajectories in the GOAD environment against operational requirements (Caldwell et al., 4 Aug 2025). Enterprise document review systems use specialized agents for template compliance, factual accuracy, terminology consistency, completeness, clarity, redundancy, and bias monitoring on highly structured business documents (Dasgupta et al., 23 Jun 2025). NarrativeFactScore evaluates long-narrative summarization by constructing a Character Knowledge Graph and checking atomic facts against narrative evidence (Jeong et al., 17 Jan 2025).

Taken together, these systems show that Agent-as-Judge is not tied to a single modality or task type. What unifies the literature is the need to judge artifacts whose validity depends on intermediate reasoning, environmental state, evidence access, or multi-dimensional domain criteria.

5. Empirical alignment, calibration, and diagnostic findings

Several papers report that agentic judges align more closely with human or task-grounded evaluation than output-only baselines. On DevAI, Agent-as-a-Judge achieves alignment rates of 92.07%, 90.44%, and 90.16% against human consensus for MetaGPT, GPT-Pilot, and OpenHands, compared with 84.15%, 65.30%, and 60.38% for LLM-as-a-Judge (Zhuge et al., 2024). The same study reports an API cost of $30.58** and runtime of **118.43 minutes**, compared with **86.5 hours** and an estimated **$1297.50 for human evaluation (Zhuge et al., 2024).

AJ-Bench shows that enabling environment interaction yields average F1 improvements of about 13 percentage points over text-only LLM-as-a-Judge on the same base model. For gpt-5-mini-low, overall F1 rises from 59.00 to 72.41; for deepseek-v3.2, it rises from 64.49 to 77.34 (Shi et al., 20 Apr 2026). The same benchmark also notes that the agentic baseline’s average F1 is still only 0.72, leaving substantial headroom (Shi et al., 20 Apr 2026).

Online Agent-as-a-Judge reports similarly strong gains when the relevant behavior must be elicited rather than merely observed. Criteria coverage rises to 0.92, compared with 0.56 for an offline LLM judge and 0.54 for an offline agent judge; human-label agreement rises to 0.70, compared with 0.33 and 0.40 (Ryu et al., 6 Jun 2026). The largest improvements occur in criteria where the triggering situation rarely arises spontaneously, such as Conflict/Norm Violation and Emotional/Social Support (Ryu et al., 6 Jun 2026).

Other benchmarks emphasize judge reliability directly. Mind2Web 2 reports human validation with only 7/720 true verifier errors after excluding human mistakes and one source-consistency edge case, corresponding to an approximate judge correctness rate of 99.03% (Gou et al., 26 Jun 2025). Auto-Eval Judge reports 4.76% higher alignment accuracy than a GPT-4o LLM-as-a-Judge baseline on GAIA and 10.52% higher alignment accuracy on BigCodeBench (Bhonsle et al., 7 Aug 2025). SAGE reports a Pearson correlation of $\mathcal{E}$0 between the final emotion score and BLRI-based thought ratings, and $\mathcal{E}$1 between the emotion score and utterance-level empathy metrics, supporting the claim of psychological fidelity (Zhang et al., 1 May 2025).

At the same time, empirical studies repeatedly show that judge quality is conditional rather than uniform. "The Confident Liar" (Keramati et al., 9 Jun 2026) reports a consistent four-phase confidence trajectory in debate, but also a strong role asymmetry: mean confidence-quality correlation is 0.335 for the Constructor and 0.177 for the Auditor, and critical-failure detection from confidence features yields AUROC 0.804 versus 0.634. This suggests that the same internal or external judging signal can have different semantics depending on the agent’s role and the structure of the task (Keramati et al., 9 Jun 2026).

6. Limits, controversies, and alternative formulations

The literature is explicit that Agent-as-Judge does not eliminate the classic reliability problems of automated evaluation. A review of the field lists bias, collusion or mode collapse, prompt sensitivity, implementation complexity, cheatability and adversarial manipulation, benchmark overfitting, and limited true domain expertise as recurring failure modes, and argues that agent-based judging should complement rather than replace human oversight (Yu, 5 Aug 2025).

Language sensitivity adds another layer of instability. The multilingual prompt-localization study shows that changing the judge language can invert backbone rankings: GPT-4o achieves the highest satisfaction in English at 44.72%, while Gemini leads in Arabic at 51.72% and in Hindi at 53.22%; inter-backbone agreement is only modest, with Fleiss’ $\mathcal{E}$2 (Mahmood et al., 6 Apr 2026). A controlled ablation reports that Hindi satisfaction drops from 42.8% to 23.2% under partial localization, indicating that the full prompt stack—not just benchmark content—can be decisive (Mahmood et al., 6 Apr 2026).

A more fundamental controversy concerns whether probabilistic judges can ever verify state-constrained validity. GroundEval argues for a deterministic, judge-free alternative in stateful agent evaluation. Its case study describes a run in which two frontier LLM judges scored a response at 0.90 and 0.85, but the trace showed that the agent had never fetched the required artifact, yielding $\mathcal{E}$3 and $\mathcal{E}$4 (Flynt, 22 Jun 2026). The claim is not merely that judges can err, but that prose plausibility and evidence-path validity are different evaluation objects (Flynt, 22 Jun 2026).

FormalJudge pushes this critique into a neuro-symbolic oversight paradigm. Instead of letting an LLM directly estimate a verdict, it decomposes intent into atomic binary facts, extracts them from trajectories, compiles them into Dafny specifications, and uses Z3-based SMT verification to return a deterministic result (Zhou et al., 11 Feb 2026). The reported outcomes include a 16.6% average improvement over LLM-as-a-Judge baselines and weak-to-strong generalization in which a 7B judge achieves over 90% accuracy detecting deception from 72B agents (Zhou et al., 11 Feb 2026). This suggests that some forms of agentic oversight may increasingly shift from deliberative judgment to executable verification.

Future directions in the survey literature include personalization, context-aware rubric generation, adaptive multi-granularity scoring, environmental probing, human-agent collaborative calibration, reinforcement learning for sequential planning and tool use, and learned multi-agent coordination (You et al., 8 Jan 2026). A plausible implication is that Agent-as-Judge will continue to bifurcate into two families: deliberative evaluators that approximate expert panels, and verification-oriented systems that minimize reliance on probabilistic final judgment.