JudgeAgent: LLM-Based Evaluator

• JudgeAgent is an LLM-based role that separates evaluative judgment from task execution by analyzing both final outputs and process trajectories.
• It employs diverse architectures—reward-model, verification-gate, and modular decomposition—in domains like legal QA, GUI mining, and scientific simulations.
• Empirical results show JudgeAgent reduces costs and improves decision accuracy, although challenges include latency, bias, and sensitivity to prompt design.

JudgeAgent denotes an LLM-based agent whose primary role is evaluation, verification, or adjudication rather than direct task execution. In the literature, this role appears in several forms: as an evaluator of other models or agents in the broader “agent-as-a-judge” paradigm, as a verifier of evidence sufficiency and jurisdictional validity in legal QA, as a process-level reward model inside search-based GUI mining, as a courtroom-style arbiter in multi-agent debate, and as a trajectory auditor for general task completion, penetration testing, and scientific simulation (Yu, 5 Aug 2025, Wang et al., 31 Aug 2025, Lv et al., 5 Feb 2026, Bhonsle et al., 7 Aug 2025, Yang, 26 Mar 2026). This suggests that JudgeAgent is best understood not as a single architecture, but as a recurring functional role: an agent that observes outputs, trajectories, retrieved evidence, or debate transcripts and produces verdicts, critiques, rewards, or certificates that reshape downstream behavior.

1. Conceptual scope and defining role

Within the surveyed systems, a JudgeAgent is consistently separated from the “actor” or “generator” role. In the review literature, an AI JudgeAgent is defined as an LLM-based agent whose primary role is to evaluate other models or agents, often with access to tools, memory, multi-step reasoning, and full trajectories rather than only final outputs (Yu, 5 Aug 2025). Auto-Eval Judge makes the same division explicit by defining an Actor as the agent responsible for executing a task and a Judge as the agent that evaluates the Actor’s performance by analyzing both intermediate reasoning states and final output artifacts (Bhonsle et al., 7 Aug 2025). PentestJudge adopts the same pattern for penetration-testing trajectories, where the judge reads prompts, tool calls, and outputs to determine whether operational requirements are satisfied (Caldwell et al., 4 Aug 2025).

This separation is operational rather than merely descriptive. In M$^2$-Miner, JudgeAgent is the central evaluation component inside Monte Carlo Tree Search (MCTS): it decides whether an expanded GUI state is terminal, assigns immediate rewards, and replaces expensive rollout-based simulation with a single evaluation call (Lv et al., 5 Feb 2026). In L-MARS, the Judge Agent is a separate LLM instance, implemented as a LangGraph node, that determines whether the accumulated evidence is legally sufficient and whether retrieval should continue (Wang et al., 31 Aug 2025). In courtroom-style systems such as AgenticSimLaw, SAMVAD, AgentsBench, and AgentsCourt, judge roles become more institutionally inflected: they may maintain a private belief state, issue legal instructions, moderate deliberation, or synthesize a final decision (Chun et al., 29 Jan 2026, Devadiga et al., 4 Sep 2025, Jiang et al., 2024, He et al., 2024).

A central distinction across these works is between output-only judging and process-aware judging. The review on agent-as-a-judge explicitly frames the latter as trajectory-aware evaluation over states, actions, and environment observations, rather than a single pass over a final answer (Yu, 5 Aug 2025). Auto-Eval Judge, PentestJudge, M$^2$-Miner, and the scientific simulation Judge Agent all belong to this process-aware family, whereas benchmark-style evaluators or static legal graders sit closer to output-level assessment (Bhonsle et al., 7 Aug 2025, Caldwell et al., 4 Aug 2025, Lv et al., 5 Feb 2026, Yang, 26 Mar 2026).

2. Architectural patterns

Across domains, JudgeAgent architectures recur in several stable patterns.

System	Domain	JudgeAgent role
M$^2$-Miner (Lv et al., 5 Feb 2026)	Mobile GUI mining	Outcome/process reward evaluator inside MCTS
L-MARS (Wang et al., 31 Aug 2025)	Legal QA	Sufficiency verifier for evidence, jurisdiction, date, and contradictions
AgenticSimLaw (Chun et al., 29 Jan 2026)	Recidivism prediction	Debate listener with private belief updates and final verdict
SAMVAD (Devadiga et al., 4 Sep 2025)	Indian judicial simulation	RAG-grounded judge issuing impartial legal instructions
Auto-Eval Judge (Bhonsle et al., 7 Aug 2025)	General agent evaluation	Criteria–proof–verdict pipeline over logs and outputs
PentestJudge (Caldwell et al., 4 Aug 2025)	Offensive security	Tool-using trajectory auditor against hierarchical rubrics

M$^2$-Miner exemplifies a reward-model judge. Its JudgeAgent is split into an Outcome Reward Model and a Process Reward Model, and it directly shapes MCTS values and pruning decisions (Lv et al., 5 Feb 2026). L-MARS exemplifies a verification-gate judge: it sits between retrieval and answer synthesis and produces a sufficiency decision plus refinement notes (Wang et al., 31 Aug 2025). Auto-Eval Judge exemplifies a modular decomposition judge: it first generates checklist criteria, then retrieves proofs from logs, then routes each criterion to factual, logical, reasoning, or coding checks before aggregating a final verdict (Bhonsle et al., 7 Aug 2025).

Courtroom and bench simulations instantiate a different archetype. AgenticSimLaw’s judge listens to prosecutor and defense arguments, maintains a private evolving belief state over the debate, and issues a final public verdict with confidence and explanation (Chun et al., 29 Jan 2026). SAMVAD’s Judge Agent does not vote directly; it generates impartial legal instructions for Adjudicators and anchors deliberation in Indian legal sources retrieved from a vectorized corpus (Devadiga et al., 4 Sep 2025). AgentsBench distributes judging across a simulated bench of professional and lay judges, with a presiding judge evaluating consensus and synthesizing the final sentencing decision (Jiang et al., 2024). AgentsCourt uses a Judge Agent together with plaintiff, defendant, and judge-assistant agents to emulate opening remarks, court debate, legal retrieval, preliminary judgment, and judgment refinement (He et al., 2024).

A further architectural extension is cohort-level judging. Judge Agent Forest (JAF) replaces isolated judgment of each query–response pair with joint inference over overlapping neighborhoods $S_i$, where the judge outputs an accept/refine label $J_i^l$ and a critique $J_i^c$ conditioned on both the focal case and peer exemplars (Garg et al., 29 Jan 2026). This yields a knowledge-graph-like evaluation structure and repeated randomized judgments that function as a context-sensitive ensemble (Garg et al., 29 Jan 2026).

3. Internal representations, prompts, and decision rules

JudgeAgent systems are typically defined by explicit state representations and structured outputs. In M$^2$-Miner, the intent-trajectory tree is formalized as

$T = (V, A, P, I),$

and each node $v$ stores a screenshot, action metadata, value estimate $^2$0, visit count $^2$1, and status $^2$2 (Lv et al., 5 Feb 2026). Its Outcome Reward Model classifies states as success, not_yet_succeeded, or impossible_to_succeed, while its Process Reward Model scores the current action as valid or invalid (Lv et al., 5 Feb 2026). For intermediate states, the reward is a softmax over the two logits: $^2$3 and node values are updated by

$^2$4

This makes JudgeAgent part of the search dynamics rather than a post hoc grader (Lv et al., 5 Feb 2026).

L-MARS uses a different state formalism. A query state is modeled as $^2$5, where $^2$6 is the query, $^2$7 optional user clarifications, and $^2$8 the set of retrieved documents; the overall goal is

$^2$9

Within that loop, the Judge Agent outputs a sufficiency decision $^2$0 and refinement notes $^2$1 (Wang et al., 31 Aug 2025). Its prompt imposes an explicit checklist over source authority, date, jurisdiction, and contradiction resolution, and it runs deterministically with GPT-o3 at temperature $^2$2 (Wang et al., 31 Aug 2025).

Auto-Eval Judge decomposes evaluation into explicit modules,

$^2$3

where $^2$4 is the task, $^2$5 the Actor log, $^2$6 the final answer, $^2$7 binary checklist questions, $^2$8 proof snippets, and $^2$9 the final task-completion verdict (Bhonsle et al., 7 Aug 2025). PentestJudge similarly grounds judgment in a rubric tree whose leaf nodes are atomic yes/no operational requirements, and leaf decisions are later aggregated by weighted averages up the rubric tree (Caldwell et al., 4 Aug 2025).

Several systems constrain the judge’s outputs with strict schemas. M$^2$0-Miner’s Outcome Reward Model returns JSON containing "thought", "is_terminal", and "status" (Lv et al., 5 Feb 2026). L-MARS requires executable refinement queries and a binary sufficiency label (Wang et al., 31 Aug 2025). AgenticSimLaw parses judge outputs into prediction, confidence, and reasoning, with regex/JSON fallbacks when formatting fails (Chun et al., 29 Jan 2026). The dynamic evaluation system titled JudgeAgent uses a dataset

$^2$1

and a context graph $^2$2, then iteratively updates difficulty through $^2$3 during interactive extension (Shi et al., 2 Sep 2025).

A more formal variant appears in scientific simulation. There, the Judge Agent $^2$4 verifies whether a problem lies in the simulability class $^2$5, whose defining conditions are finite specifiability, Hadamard stability, approximability, and certifiability, and it consumes a structured spec.md representation of $^2$6 (Yang, 26 Mar 2026). This is a rare case where JudgeAgent is explicitly tied to mathematical validation rather than only prompt-level criteria (Yang, 26 Mar 2026).

4. Interaction with search, planning, and self-refinement

JudgeAgents often sit inside iterative control loops. In M$^2$7-Miner, JudgeAgent replaces simulation rollouts in MCTS, is reused in intent recycling to validate newly generated intents against trajectories, and also provides labels for model-in-the-loop training of later JudgeAgent versions (Lv et al., 5 Feb 2026). This turns judgment into both a real-time search signal and a source of supervision.

In L-MARS, the Judge Agent governs an iterative retrieval process: $^2$8 If sufficiency is denied, the Query Agent refines search queries using the judge’s notes and the Search Agent retrieves again (Wang et al., 31 Aug 2025). The judge therefore controls when to stop, what to search next, and which deficiencies matter most—wrong jurisdiction, outdated sources, only unofficial forums, or unresolved contradictions (Wang et al., 31 Aug 2025).

Cohort-structured self-refinement appears in JAF. For each focal instance $^2$9, the judge evaluates

$S_i$0

where $S_i$1 is a sampled neighborhood of peer query–response pairs (Garg et al., 29 Jan 2026). Overlapping neighborhoods induce a knowledge graph $S_i$2, and repeated randomized evaluations yield acceptance probabilities

$S_i$3

which behave like an ensemble of context-sensitive judgments (Garg et al., 29 Jan 2026).

The interviewer-style JudgeAgent also turns judging into an adaptive dialogue. It first grades benchmark batches, then uses a context graph to retrieve entity-linked background material, generates difficulty-controlled extension questions, and finally produces batch-level suggestions $S_i$4 attached to each question batch $S_i$5 (Shi et al., 2 Sep 2025). The paper operationalizes validation through a second-round querying scheme: ask the model once normally, then query again with JudgeAgent’s suggestions and measure the change in accuracy, correction rate, and correct-to-error rate (Shi et al., 2 Sep 2025).

Auto-Eval Judge and PentestJudge emphasize tool-augmented evidence gathering. Auto-Eval Judge indexes long execution logs into summarized chunks, retrieves proof snippets with a cross-encoder, and sometimes delegates factual or coding verification to Magentic-One sub-agents (Bhonsle et al., 7 Aug 2025). PentestJudge gives the judge tools to search through arbitrary long trajectories, inspect tool definitions, and write intermediate memory notes before issuing a binary pass/fail requirement judgment (Caldwell et al., 4 Aug 2025). These designs make JudgeAgent closer to an auditor than a static scorer.

5. Empirical effects across domains

The reported empirical effects are heterogeneous because tasks differ, but the shared pattern is that JudgeAgents improve reliability, calibration, or efficiency in their local workflows.

In mobile GUI mining, adding JudgeAgent to MCTS changes both cost and data quality. The paper reports that, at task length 9, the multi-agent setup with JudgeAgent yields a 64× speedup over vanilla MCTS using only InferAgent; the overall dataset cost is \$S_i$60.02 per image</strong> versus about <strong>\$0.36 for manual baselines, an 18× reduction (Lv et al., 5 Feb 2026). The same work reports higher Mining Success Ratio (MSR) and Data Quality Accuracy (DQA), and states that GUI agents fine-tuned on the mined data reach state-of-the-art performance on multiple mobile GUI benchmarks (Lv et al., 5 Feb 2026).

In legal QA, the multi-turn L-MARS configuration—the one that includes the Judge Agent—achieves 0.98 accuracy and 0.39 U-Score on 200 LegalSearchQA questions, compared with 0.96 accuracy and 0.42 U-Score for the simple mode without the judge loop (Wang et al., 31 Aug 2025). The gain is paired with higher latency, 55.67 s versus 13.62 s, which makes the judge’s reliability/latency trade-off explicit (Wang et al., 31 Aug 2025).

In general task-completion evaluation, Auto-Eval Judge reports 4.76% higher alignment accuracy than a GPT-4o LLM-as-a-Judge baseline on GAIA and 10.52% higher alignment accuracy on BigCodeBench (Bhonsle et al., 7 Aug 2025). In offensive security, PentestJudge’s best evaluated model reaches F1 = 0.83, and the paper notes that weaker and cheaper models can judge the trajectories of pentests performed by stronger and more expensive models, suggesting that verification may be easier than generation for this task (Caldwell et al., 4 Aug 2025).

Scientific simulation provides the strongest reliability claim. The Judge Agent for scientific computation reduces the silent-failure rate from 42% to 1.5% across 134 test cases spanning 12 scientific domains (Yang, 26 Mar 2026). On 72 blinded tasks from 12 independent scientists, the pipeline reaches 89% success with automated error bounds, with a 95% CI of $S_i$7, versus 53% without the Judge (Yang, 26 Mar 2026). On the powered clinical CT study (n = 200), the system reaches 99% of expert quality (Yang, 26 Mar 2026).

Legal decision-support systems also report gains from judge-centric designs. AgentsBench improves prison-term prediction with GPT-4 from 80.98% under a standard prompt to 86.33%, and the human-rated morality metric rises to 76.2% under the multi-agent bench simulation (Jiang et al., 2024). AgentsCourt reports that its legal-grounds F1 improves by 8.6% and 9.1% in first- and second-instance settings over strong baselines (He et al., 2024). Judge-R1, a judge-style framework for judgment document generation, reports retrieval gains from 0.4535 to 0.6383 in P@5 over a strong dense baseline and improves convicting F1 and referencing F1 in generation benchmarks (Su et al., 3 May 2026).

6. Limitations, adversarial pressures, and open problems

JudgeAgent systems inherit model biases, prompt sensitivity, and data dependence, and several papers make these failure modes explicit. M$S_i$8-Miner notes early-stage judge weakness during warm-up, prompt-based sensitivity in the Outcome Reward Model, and a self-reinforcing loop in which the Process Reward Model is trained on data derived from prior JudgeAgent outputs (Lv et al., 5 Feb 2026). L-MARS reports a tendency to over-reject partially relevant sources, leading to unnecessary search iterations, and its multi-turn mode carries substantial latency and remains bounded by retrieval quality (Wang et al., 31 Aug 2025).

Courtroom-style judges introduce different concerns. AgenticSimLaw states that its explanations are plausible, not necessarily faithful, and that confidence scores do not correlate with correctness; it also includes an explicit non-deployment declaration for sentencing, parole, and other consequential criminal-justice settings (Chun et al., 29 Jan 2026). AgentsBench improves transparency and morality scores, but the paper still treats the system as a support tool rather than an autonomous court decision-maker (Jiang et al., 2024). SAMVAD’s Judge Agent is single-shot, grounded only in the Constitution of India, IPC, and CrPC, and does not yet reason over precedent or intervene during deliberation (Devadiga et al., 4 Sep 2025).

Several works highlight that JudgeAgents can themselves be attacked or destabilize other agents. “Helpful Agent Meets Deceptive Judge” shows that even strong agents are vulnerable to persuasive yet flawed critiques, often switching correct answers after a single round of misleading feedback (Ming et al., 3 Jun 2025). In grounded settings such as WAFER-QA, most high-end models except o4-mini experience more than a 50% drop in accuracy from initial answer to first-round answer under grounded judges, and non-reasoning models display oscillatory correctness patterns across multiple rounds (Ming et al., 3 Jun 2025). This makes judge reliability a system-level vulnerability rather than a mere evaluation nuisance.

Process-auditing judges also have practical constraints. Auto-Eval Judge is currently text-only, does not yet handle file- or attachment-based requirements in its Criteria Generator, and assumes a single log file (Bhonsle et al., 7 Aug 2025). PentestJudge depends strongly on rubric wording, uses a small number of trajectories in one environment, and shows that models with weak tool use miss evidence and underperform human experts (Caldwell et al., 4 Aug 2025). The scientific Judge Agent formally localizes its residual failures at bifurcation points where certifiability breaks down; the paper identifies this boundary as the simulability frontier $S_i$9 (Yang, 26 Mar 2026).

The broader survey literature is explicit that judge agents can complement but not replace human oversight (Yu, 5 Aug 2025). A plausible implication is that the field is moving toward trajectory-aware judges, verifier ensembles, uncertainty-sensitive stop rules, and human-in-the-loop escalation, especially in domains where correctness is not fully observable from final outputs alone.