---
title: 'JudgeAgent: LLM-Based Evaluator'
url: https://www.emergentmind.com/topics/judgeagent
type: topic
---

# JudgeAgent: LLM-Based Evaluator

JudgeAgent denotes an LLM-based agent whose primary role is evaluation, verification, or adjudication rather than direct task execution. In the literature, this role appears in several forms: as an evaluator of other models or agents in the broader “agent-as-a-judge” paradigm, as a verifier of evidence sufficiency and jurisdictional validity in legal QA, as a process-level reward model inside search-based GUI mining, as a courtroom-style arbiter in multi-agent debate, and as a trajectory auditor for general task completion, penetration testing, and scientific simulation [2508.02994] [2509.00761] [2602.05429] [2508.05508] [2603.25780]. This suggests that JudgeAgent is best understood not as a single architecture, but as a recurring functional role: an agent that observes outputs, trajectories, retrieved evidence, or debate transcripts and produces verdicts, critiques, rewards, or certificates that reshape downstream behavior.

## 1. Conceptual scope and defining role

Within the surveyed systems, a JudgeAgent is consistently separated from the “actor” or “generator” role. In the review literature, an AI JudgeAgent is defined as an LLM-based agent whose primary role is to evaluate other models or agents, often with access to tools, memory, multi-step reasoning, and full trajectories rather than only final outputs [2508.02994]. Auto-Eval Judge makes the same division explicit by defining an **Actor** as the agent responsible for executing a task and a **Judge** as the agent that evaluates the Actor’s performance by analyzing both intermediate reasoning states and final output artifacts [2508.05508]. PentestJudge adopts the same pattern for penetration-testing trajectories, where the judge reads prompts, tool calls, and outputs to determine whether operational requirements are satisfied [2508.02921].

This separation is operational rather than merely descriptive. In M$^2$-Miner, JudgeAgent is the central evaluation component inside Monte Carlo Tree Search (MCTS): it decides whether an expanded GUI state is terminal, assigns immediate rewards, and replaces expensive rollout-based simulation with a single evaluation call [2602.05429]. In L-MARS, the Judge Agent is a separate LLM instance, implemented as a LangGraph node, that determines whether the accumulated evidence is legally sufficient and whether retrieval should continue [2509.00761]. In courtroom-style systems such as AgenticSimLaw, SAMVAD, AgentsBench, and AgentsCourt, judge roles become more institutionally inflected: they may maintain a private belief state, issue legal instructions, moderate deliberation, or synthesize a final decision [2601.21936] [2509.03793] [2412.18697] [2403.02959].

A central distinction across these works is between **output-only judging** and **process-aware judging**. The review on agent-as-a-judge explicitly frames the latter as trajectory-aware evaluation over states, actions, and environment observations, rather than a single pass over a final answer [2508.02994]. Auto-Eval Judge, PentestJudge, M$^2$-Miner, and the scientific simulation Judge Agent all belong to this process-aware family, whereas benchmark-style evaluators or static legal graders sit closer to output-level assessment [2508.05508] [2508.02921] [2602.05429] [2603.25780].

## 2. Architectural patterns

Across domains, JudgeAgent architectures recur in several stable patterns.

| System | Domain | JudgeAgent role |
|---|---|---|
| M$^2$-Miner [2602.05429] | Mobile GUI mining | Outcome/process reward evaluator inside MCTS |
| L-MARS [2509.00761] | Legal QA | Sufficiency verifier for evidence, jurisdiction, date, and contradictions |
| AgenticSimLaw [2601.21936] | Recidivism prediction | Debate listener with private belief updates and final verdict |
| SAMVAD [2509.03793] | Indian judicial simulation | RAG-grounded judge issuing impartial legal instructions |
| Auto-Eval Judge [2508.05508] | General agent evaluation | Criteria–proof–verdict pipeline over logs and outputs |
| PentestJudge [2508.02921] | Offensive security | Tool-using trajectory auditor against hierarchical rubrics |

M$^2$-Miner exemplifies a **reward-model judge**. Its JudgeAgent is split into an Outcome Reward Model and a Process Reward Model, and it directly shapes MCTS values and pruning decisions [2602.05429]. L-MARS exemplifies a **verification-gate judge**: it sits between retrieval and answer synthesis and produces a sufficiency decision plus refinement notes [2509.00761]. Auto-Eval Judge exemplifies a **modular decomposition judge**: it first generates checklist criteria, then retrieves proofs from logs, then routes each criterion to factual, logical, reasoning, or coding checks before aggregating a final verdict [2508.05508].

Courtroom and bench simulations instantiate a different archetype. AgenticSimLaw’s judge listens to prosecutor and defense arguments, maintains a private evolving belief state over the debate, and issues a final public verdict with confidence and explanation [2601.21936]. SAMVAD’s Judge Agent does not vote directly; it generates impartial legal instructions for Adjudicators and anchors deliberation in Indian legal sources retrieved from a vectorized corpus [2509.03793]. AgentsBench distributes judging across a simulated bench of professional and lay judges, with a presiding judge evaluating consensus and synthesizing the final sentencing decision [2412.18697]. AgentsCourt uses a Judge Agent together with plaintiff, defendant, and judge-assistant agents to emulate opening remarks, court debate, legal retrieval, preliminary judgment, and judgment refinement [2403.02959].

A further architectural extension is **cohort-level judging**. Judge Agent Forest (JAF) replaces isolated judgment of each query–response pair with joint inference over overlapping neighborhoods \(S_i\), where the judge outputs an accept/refine label \(J_i^l\) and a critique \(J_i^c\) conditioned on both the focal case and peer exemplars [2601.22269]. This yields a knowledge-graph-like evaluation structure and repeated randomized judgments that function as a context-sensitive ensemble [2601.22269].

## 3. Internal representations, prompts, and decision rules

JudgeAgent systems are typically defined by explicit state representations and structured outputs. In M$^2$-Miner, the intent-trajectory tree is formalized as
\[
T = (V, A, P, I),
\]
and each node \(v\) stores a screenshot, action metadata, value estimate \(Q_v\), visit count \(N_v\), and status \(\text{stat}_v \in \{\text{success}, \text{failure}, \text{intermediate}\}\) [2602.05429]. Its Outcome Reward Model classifies states as `success`, `not_yet_succeeded`, or `impossible_to_succeed`, while its Process Reward Model scores the current action as `valid` or `invalid` [2602.05429]. For intermediate states, the reward is a softmax over the two logits:
\[
R_{\text{intermediate}} =
\frac{\exp(\text{logits}_{\text{valid}})}
{\exp(\text{logits}_{\text{valid}})+\exp(\text{logits}_{\text{invalid}})},
\]
and node values are updated by
\[
Q_i = \frac{Q_{i-1} \times N_{i-1} + R_i}{N_{i-1}+1}, \quad N_i = N_{i-1}+1.
\]
This makes JudgeAgent part of the search dynamics rather than a post hoc grader [2602.05429].

L-MARS uses a different state formalism. A query state is modeled as \((q, U, D)\), where \(q\) is the query, \(U\) optional user clarifications, and \(D\) the set of retrieved documents; the overall goal is
\[
(q, U, D) \mapsto (R, a).
\]
Within that loop, the Judge Agent outputs a sufficiency decision \(s \in \{\text{Sufficient}, \text{Insufficient}\}\) and refinement notes \(E\) [2509.00761]. Its prompt imposes an explicit checklist over source authority, date, jurisdiction, and contradiction resolution, and it runs deterministically with GPT-o3 at temperature \(T=0\) [2509.00761].

Auto-Eval Judge decomposes evaluation into explicit modules,
\[
C = \text{CriteriaGenerator}(T), \quad
P = \text{ArtifactContentParser}(L, C), \quad
R = \text{C3}(T, C, P, A), \quad
V = \text{VerdictGenerator}(T, C, P, R, A),
\]
where \(T\) is the task, \(L\) the Actor log, \(A\) the final answer, \(C\) binary checklist questions, \(P\) proof snippets, and \(V\in\{\text{Yes},\text{No}\}\) the final task-completion verdict [2508.05508]. PentestJudge similarly grounds judgment in a rubric tree whose leaf nodes are atomic yes/no operational requirements, and leaf decisions are later aggregated by weighted averages up the rubric tree [2508.02921].

Several systems constrain the judge’s outputs with strict schemas. M$^2$-Miner’s Outcome Reward Model returns JSON containing `"thought"`, `"is_terminal"`, and `"status"` [2602.05429]. L-MARS requires executable refinement queries and a binary sufficiency label [2509.00761]. AgenticSimLaw parses judge outputs into `prediction`, `confidence`, and `reasoning`, with regex/JSON fallbacks when formatting fails [2601.21936]. The dynamic evaluation system titled JudgeAgent uses a dataset
\[
\mathcal{D} = \{(q_i, a_i, d_i)\}_{i=1}^{N}
\]
and a context graph \(\mathcal{G}=(\mathcal{N},\mathcal{E})\), then iteratively updates difficulty through \(\text{Decide\_Difficulty}(sc_\mathcal{B}, RND_{total})\) during interactive extension [2509.02097].

A more formal variant appears in scientific simulation. There, the Judge Agent \(A_J\) verifies whether a problem lies in the simulability class \(S\), whose defining conditions are finite specifiability, Hadamard stability, approximability, and certifiability, and it consumes a structured `spec.md` representation of \((\Omega,\mathcal{E},\mathcal{B},\mathcal{I},\mathcal{O},\varepsilon)\) [2603.25780]. This is a rare case where JudgeAgent is explicitly tied to mathematical validation rather than only prompt-level criteria [2603.25780].

## 4. Interaction with search, planning, and self-refinement

JudgeAgents often sit inside iterative control loops. In M$^2$-Miner, JudgeAgent replaces simulation rollouts in MCTS, is reused in intent recycling to validate newly generated intents against trajectories, and also provides labels for model-in-the-loop training of later JudgeAgent versions [2602.05429]. This turns judgment into both a real-time search signal and a source of supervision.

In L-MARS, the Judge Agent governs an iterative retrieval process:
\[
(\text{suff}, \text{notes}) \gets \text{JudgeAgent}(R \cup R_i, T{=}0).
\]
If sufficiency is denied, the Query Agent refines search queries using the judge’s notes and the Search Agent retrieves again [2509.00761]. The judge therefore controls when to stop, what to search next, and which deficiencies matter most—wrong jurisdiction, outdated sources, only unofficial forums, or unresolved contradictions [2509.00761].

Cohort-structured self-refinement appears in JAF. For each focal instance \(i\), the judge evaluates
\[
(J_i^l, J_i^c) = J\bigl((Q_i, R_i), S_i\bigr),
\]
where \(S_i\) is a sampled neighborhood of peer query–response pairs [2601.22269]. Overlapping neighborhoods induce a knowledge graph \(G^{\mathrm{KG}}=(V,E)\), and repeated randomized evaluations yield acceptance probabilities
\[
\hat{p}_i = \frac{1}{R}\sum_{r=1}^{R}\mathbf{1}\{J_i^{(r)}=\text{accept}\},
\]
which behave like an ensemble of context-sensitive judgments [2601.22269].

The interviewer-style JudgeAgent also turns judging into an adaptive dialogue. It first grades benchmark batches, then uses a context graph to retrieve entity-linked background material, generates difficulty-controlled extension questions, and finally produces batch-level suggestions \(s_i\) attached to each question batch \(\mathcal{B}_i\) [2509.02097]. The paper operationalizes validation through a second-round querying scheme: ask the model once normally, then query again with JudgeAgent’s suggestions and measure the change in accuracy, correction rate, and correct-to-error rate [2509.02097].

Auto-Eval Judge and PentestJudge emphasize tool-augmented evidence gathering. Auto-Eval Judge indexes long execution logs into summarized chunks, retrieves proof snippets with a cross-encoder, and sometimes delegates factual or coding verification to Magentic-One sub-agents [2508.05508]. PentestJudge gives the judge tools to search through arbitrary long trajectories, inspect tool definitions, and write intermediate memory notes before issuing a binary pass/fail requirement judgment [2508.02921]. These designs make JudgeAgent closer to an auditor than a static scorer.

## 5. Empirical effects across domains

The reported empirical effects are heterogeneous because tasks differ, but the shared pattern is that JudgeAgents improve reliability, calibration, or efficiency in their local workflows.

In mobile GUI mining, adding JudgeAgent to MCTS changes both cost and data quality. The paper reports that, at task length 9, the multi-agent setup with JudgeAgent yields a **64×** speedup over vanilla MCTS using only InferAgent; the overall dataset cost is **\$466 for 20k images**, about **\$0.02 per image** versus about **\$0.36** for manual baselines, an **18×** reduction [2602.05429]. The same work reports higher Mining Success Ratio (MSR) and Data Quality Accuracy (DQA), and states that GUI agents fine-tuned on the mined data reach state-of-the-art performance on multiple mobile GUI benchmarks [2602.05429].

In legal QA, the multi-turn L-MARS configuration—the one that includes the Judge Agent—achieves **0.98** accuracy and **0.39** U-Score on 200 LegalSearchQA questions, compared with **0.96** accuracy and **0.42** U-Score for the simple mode without the judge loop [2509.00761]. The gain is paired with higher latency, **55.67 s** versus **13.62 s**, which makes the judge’s reliability/latency trade-off explicit [2509.00761].

In general task-completion evaluation, Auto-Eval Judge reports **4.76%** higher alignment accuracy than a GPT-4o LLM-as-a-Judge baseline on GAIA and **10.52%** higher alignment accuracy on BigCodeBench [2508.05508]. In offensive security, PentestJudge’s best evaluated model reaches **F1 = 0.83**, and the paper notes that weaker and cheaper models can judge the trajectories of pentests performed by stronger and more expensive models, suggesting that verification may be easier than generation for this task [2508.02921].

Scientific simulation provides the strongest reliability claim. The Judge Agent for scientific computation reduces the silent-failure rate from **42%** to **1.5%** across **134** test cases spanning **12** scientific domains [2603.25780]. On **72** blinded tasks from **12** independent scientists, the pipeline reaches **89%** success with automated error bounds, with a **95% CI of \([80\%,95\%]\)**, versus **53%** without the Judge [2603.25780]. On the powered clinical CT study (**n = 200**), the system reaches **99% of expert quality** [2603.25780].

Legal decision-support systems also report gains from judge-centric designs. AgentsBench improves prison-term prediction with GPT-4 from **80.98%** under a standard prompt to **86.33%**, and the human-rated morality metric rises to **76.2%** under the multi-agent bench simulation [2412.18697]. AgentsCourt reports that its legal-grounds F1 improves by **8.6%** and **9.1%** in first- and second-instance settings over strong baselines [2403.02959]. Judge-R1, a judge-style framework for judgment document generation, reports retrieval gains from **0.4535** to **0.6383** in P@5 over a strong dense baseline and improves convicting F1 and referencing F1 in generation benchmarks [2605.02011].

## 6. Limitations, adversarial pressures, and open problems

JudgeAgent systems inherit model biases, prompt sensitivity, and data dependence, and several papers make these failure modes explicit. M$^2$-Miner notes early-stage judge weakness during warm-up, prompt-based sensitivity in the Outcome Reward Model, and a self-reinforcing loop in which the Process Reward Model is trained on data derived from prior JudgeAgent outputs [2602.05429]. L-MARS reports a tendency to **over-reject partially relevant sources**, leading to unnecessary search iterations, and its multi-turn mode carries substantial latency and remains bounded by retrieval quality [2509.00761].

Courtroom-style judges introduce different concerns. AgenticSimLaw states that its explanations are **plausible**, not necessarily **faithful**, and that confidence scores do not correlate with correctness; it also includes an explicit non-deployment declaration for sentencing, parole, and other consequential criminal-justice settings [2601.21936]. AgentsBench improves transparency and morality scores, but the paper still treats the system as a support tool rather than an autonomous court decision-maker [2412.18697]. SAMVAD’s Judge Agent is single-shot, grounded only in the Constitution of India, IPC, and CrPC, and does not yet reason over precedent or intervene during deliberation [2509.03793].

Several works highlight that JudgeAgents can themselves be attacked or destabilize other agents. “Helpful Agent Meets Deceptive Judge” shows that even strong agents are vulnerable to persuasive yet flawed critiques, often switching correct answers after a single round of misleading feedback [2506.03332]. In grounded settings such as WAFER-QA, most high-end models except o4-mini experience more than a **50%** drop in accuracy from initial answer to first-round answer under grounded judges, and non-reasoning models display oscillatory correctness patterns across multiple rounds [2506.03332]. This makes judge reliability a system-level vulnerability rather than a mere evaluation nuisance.

Process-auditing judges also have practical constraints. Auto-Eval Judge is currently text-only, does not yet handle file- or attachment-based requirements in its Criteria Generator, and assumes a single log file [2508.05508]. PentestJudge depends strongly on rubric wording, uses a small number of trajectories in one environment, and shows that models with weak tool use miss evidence and underperform human experts [2508.02921]. The scientific Judge Agent formally localizes its residual failures at bifurcation points where certifiability breaks down; the paper identifies this boundary as the simulability frontier \(\partial S\) [2603.25780].

The broader survey literature is explicit that judge agents can complement but not replace human oversight [2508.02994]. A plausible implication is that the field is moving toward trajectory-aware judges, verifier ensembles, uncertainty-sensitive stop rules, and human-in-the-loop escalation, especially in domains where correctness is not fully observable from final outputs alone.

Source: https://www.emergentmind.com/topics/judgeagent