Trajectory-Aware Judge

• Trajectory-aware judging is an evaluation paradigm that assesses dynamic processes by inspecting intermediate actions, context, and subsequent effects across a trajectory.
• It applies to diverse fields like autonomous driving, robotics, and LLM-agent monitoring, where conventional endpoint metrics may overlook critical process failures.
• It employs recurrent design patterns such as planning-sensitive metrics, iterative evidence aggregation, and process modeling to capture performance nuances and improve decision-making.

A trajectory-aware judge is an evaluation paradigm that assigns quality by inspecting a trajectory as a structured process rather than reducing judgment to a final answer, a terminal state, or a task-agnostic geometric error. Across recent work, the judged object may be a forecasted motion plan, a robotic execution video, a tool-use trace, a stateful evidence path, or a multi-turn co-creation history; what unifies these settings is that correctness depends on intermediate actions, context, environment interaction, and downstream consequence, not only on endpoint plausibility (Ivanovic et al., 2021, Ji et al., 23 Mar 2026, Mittapalli et al., 5 Jun 2026, Flynt, 22 Jun 2026).

1. Conceptual scope and defining properties

The basic motivation for trajectory-aware judgment appears in several communities under different names. In trajectory forecasting for autonomous driving, conventional metrics such as displacement error and likelihood are criticized as task-agnostic because two predictions can be equally wrong geometrically yet have very different downstream planning consequences for the ego vehicle (Ivanovic et al., 2021). In long-horizon LLM-agent monitoring, single-pass or fixed-window judges are criticized because weak malicious evidence can be temporally distributed and only becomes meaningful when distant steps are linked (Mittapalli et al., 5 Jun 2026). In tool-using agents, judging only the final answer misses whether the correct tools were selected, parameterized, ordered, and grounded in valid evidence (He et al., 6 Oct 2025, KC, 21 Jun 2026).

A second defining property is environment or context dependence. Some judges are trajectory-aware because they condition judgment on who acted, what was accessible, which subsystems were visible, or what local semantics surrounded the trajectory. Context-aware mobility anomaly detection, for example, models trajectories conditional on agent identity and POI-derived geographic context rather than treating rare paths as globally anomalous in isolation (Hu et al., 2024). GroundEval goes further by treating the evidence path itself as part of correctness: a response can be factually right yet still be invalid if the agent never retrieved the artifact it depended on, used role-inaccessible evidence, or answered from the wrong temporal horizon (Flynt, 22 Jun 2026).

A third defining property is process sensitivity. In robotics, PRM-as-a-Judge argues that binary success collapses rich execution structure and obscures progress, efficiency, regression, and stagnation; the proposed judge therefore scores dense progress over the full trajectory rather than only the terminal outcome (Ji et al., 23 Mar 2026). In agentic anomaly detection, Trajectory Guard similarly treats the trajectory as an ordered sequence whose internal structure must be modeled, rather than a bag of pooled step embeddings (Advani, 2 Jan 2026). A trajectory-aware judge is therefore not merely “long-context evaluation.” It is an evaluator whose semantics change when the same endpoint is reached through a different path.

2. Recurrent architectural patterns

Recent work does not present one canonical architecture. Instead, several recurrent design patterns appear.

Pattern	Representative work	Core mechanism
Planning-sensitive metric layer	(Ivanovic et al., 2021)	Reweight base forecast errors by planning sensitivity
Adaptive evidence aggregation loop	(Mittapalli et al., 5 Jun 2026)	Triage–Inspect–Judge with persistent cross-window evidence
Pairwise trajectory reliability audit	(KC, 21 Jun 2026)	Controlled degradation of reference trajectories
Dense progress-potential judge	(Ji et al., 23 Mar 2026)	Per-state progress potential and OPD metrics
Sequence-aware anomaly discriminator	(Advani, 2 Jan 2026)	Task-plan alignment plus trajectory reconstruction
Rule-grounded environment-aware judge	(Sun et al., 15 Jun 2026)	VLM decides which deterministic driving rules to invoke
Deterministic evidence-path verifier	(Flynt, 22 Jun 2026)	Contract checking over searches, fetches, citations, access, and time

One family begins with an existing metric and adds trajectory relevance. The planning-aware metric proposed for trajectory forecasting does not replace ADE or FDE; it learns a proxy planning cost, differentiates that cost with respect to predicted trajectories, and uses the resulting sensitivities to reweight standard forecast errors (Ivanovic et al., 2021). A closely related logic appears in DriveJudge, where a VLM does not directly score a trajectory but first predicts which rule-based evaluators are relevant in the current scene, after which deterministic rule functions compute the actual grounded sub-scores (Sun et al., 15 Jun 2026).

Another family treats the judge as an active investigator. TRACE is the clearest example: instead of scoring the full trace in one pass, it performs Triage, then iterative Inspect actions with persistent “OBSERVATIONS SO FAR,” and finally Judge over an accumulated evidence log (Mittapalli et al., 5 Jun 2026). AJ-Bench generalizes this idea into environment-aware verification: the judge can open web pages, inspect filesystems or databases, interact with GUI environments, and use intermediate trajectory steps when direct state inspection is insufficient (Shi et al., 20 Apr 2026). In these systems, trajectory awareness is inseparable from evidence acquisition.

A third family is explicitly process-modeling. PRM-as-a-Judge maps each observation along a robotic trajectory to a scalar progress potential and derives outcome, process, and diagnosis metrics from the resulting trace (Ji et al., 23 Mar 2026). Trajectory Guard compresses an action sequence with a GRU encoder, reconstructs it with a GRU decoder, and jointly learns task-trajectory alignment and sequential validity, thereby combining contextual anomaly detection with structural anomaly detection (Advani, 2 Jan 2026). CriticVLA applies the same principle to autonomous driving in a judge-and-revise setting: a VLA first proposes a rough future trajectory and then reuses the backbone as a critic that analyzes and refines that trajectory (Yang et al., 30 Apr 2026).

3. Formal scoring constructions

Several papers make the trajectory-aware judge mathematically explicit. In planning-aware trajectory forecasting, the scene at time $t$ contains ego and other agents, and the learned proxy planning cost is

$$c(\mathbf{s}^{(t)}, \mathbf{u}_\text{R}^{(t)}, \mathbf{\hat{s}}^{(t:T)}) = \theta^T \phi(\mathbf{s}^{(t)}, \mathbf{u}_\text{R}^{(t)}, \mathbf{\hat{s}}^{(t:T)}).$$

Planning relevance is quantified by the gradient $\nabla_{\mathbf{\hat{s}}^{(t:T)}} c$, and a generic planning-informed metric is defined as

$$\text{PI-Metric} = \frac{1}{|\mathcal{A}|} \sum_{a \in \mathcal{A}} f(a, |\nabla_{\mathbf{\hat{s}}^{(t:T)}} c|) \cdot \text{Metric}(\mathbf{\hat{s}}^{(t:T)}_a, \mathbf{s}^{(t:T)}_a),$$

with example weighting schemes based on normalization or softmax over planning sensitivities (Ivanovic et al., 2021). The formal point is that the base metric remains recognizable, but its aggregation becomes trajectory- and task-sensitive.

TRACE formalizes the monitoring problem as trajectory-level binary classification over a full post hoc trajectory,

$$\tau = (x_1, x_2, \dots, x_T), \qquad f(\tau) \rightarrow \{benign, malicious\},$$

but its aggregation is procedural rather than equation-based. The final decision rule is explicit: TRACE assigns a final suspicion score on a $1$–$5$ scale, and trajectories scoring $\ge 4$ are classified as malicious (Mittapalli et al., 5 Jun 2026). The important formalization is therefore not a closed-form score but a stateful evidence protocol.

BabelJudge provides explicit pairwise reliability formulas for judge models. It defines order-balanced accuracy,

$$\text{acc} = \frac{|\{j \in \mathcal{J} : j.\text{choice} = \text{reference}\}|}{|\{j \in \mathcal{J} : j.\text{choice} \in \{\text{ref}, \text{pert}\}\}|},$$

position bias,

$$\Delta_{\text{pos}} = \text{WinRate}(\text{ref-first}) - \text{WinRate}(\text{pert-first}),$$

verbosity susceptibility $$c(\mathbf{s}^{(t)}, \mathbf{u}_\text{R}^{(t)}, \mathbf{\hat{s}}^{(t:T)}) = \theta^T \phi(\mathbf{s}^{(t)}, \mathbf{u}_\text{R}^{(t)}, \mathbf{\hat{s}}^{(t:T)}).$$0, order consistency $$c(\mathbf{s}^{(t)}, \mathbf{u}_\text{R}^{(t)}, \mathbf{\hat{s}}^{(t:T)}) = \theta^T \phi(\mathbf{s}^{(t)}, \mathbf{u}_\text{R}^{(t)}, \mathbf{\hat{s}}^{(t:T)}).$$1, and the composite reliability score

$$c(\mathbf{s}^{(t)}, \mathbf{u}_\text{R}^{(t)}, \mathbf{\hat{s}}^{(t:T)}) = \theta^T \phi(\mathbf{s}^{(t)}, \mathbf{u}_\text{R}^{(t)}, \mathbf{\hat{s}}^{(t:T)}).$$2

For agent trajectories, the paper defines tool accuracy, argument accuracy, hallucination detection, and trajectory-length bias in prose, but does not print the exact trajectory-level formula (KC, 21 Jun 2026). That omission is itself methodologically revealing: trajectory-aware judging remains easier to motivate than to fully axiomatize.

PRM-as-a-Judge gives the densest trajectory formalism. A trajectory is

$$c(\mathbf{s}^{(t)}, \mathbf{u}_\text{R}^{(t)}, \mathbf{\hat{s}}^{(t:T)}) = \theta^T \phi(\mathbf{s}^{(t)}, \mathbf{u}_\text{R}^{(t)}, \mathbf{\hat{s}}^{(t:T)}).$$3

and the judge induces a task-aligned progress potential $$c(\mathbf{s}^{(t)}, \mathbf{u}_\text{R}^{(t)}, \mathbf{\hat{s}}^{(t:T)}) = \theta^T \phi(\mathbf{s}^{(t)}, \mathbf{u}_\text{R}^{(t)}, \mathbf{\hat{s}}^{(t:T)}).$$4. Pairwise progress is

$$c(\mathbf{s}^{(t)}, \mathbf{u}_\text{R}^{(t)}, \mathbf{\hat{s}}^{(t:T)}) = \theta^T \phi(\mathbf{s}^{(t)}, \mathbf{u}_\text{R}^{(t)}, \mathbf{\hat{s}}^{(t:T)}).$$5

which satisfies macro-consistency by additivity. The OPD system then defines Milestone Coverage,

$$c(\mathbf{s}^{(t)}, \mathbf{u}_\text{R}^{(t)}, \mathbf{\hat{s}}^{(t:T)}) = \theta^T \phi(\mathbf{s}^{(t)}, \mathbf{u}_\text{R}^{(t)}, \mathbf{\hat{s}}^{(t:T)}).$$6

Max Progress,

$$c(\mathbf{s}^{(t)}, \mathbf{u}_\text{R}^{(t)}, \mathbf{\hat{s}}^{(t:T)}) = \theta^T \phi(\mathbf{s}^{(t)}, \mathbf{u}_\text{R}^{(t)}, \mathbf{\hat{s}}^{(t:T)}).$$7

Path-weighted Progress Length,

$$c(\mathbf{s}^{(t)}, \mathbf{u}_\text{R}^{(t)}, \mathbf{\hat{s}}^{(t:T)}) = \theta^T \phi(\mathbf{s}^{(t)}, \mathbf{u}_\text{R}^{(t)}, \mathbf{\hat{s}}^{(t:T)}).$$8

Cumulative Regret Area, and Stagnation Ratio (Ji et al., 23 Mar 2026). Here the trajectory-aware judge is literally a scalar field over execution history.

GroundEval formalizes evidence-path validity rather than semantic preference. In context mode, valid cited evidence is

$$c(\mathbf{s}^{(t)}, \mathbf{u}_\text{R}^{(t)}, \mathbf{\hat{s}}^{(t:T)}) = \theta^T \phi(\mathbf{s}^{(t)}, \mathbf{u}_\text{R}^{(t)}, \mathbf{\hat{s}}^{(t:T)}).$$9

where $\nabla_{\mathbf{\hat{s}}^{(t:T)}} c$0 is the submitted evidence set, $\nabla_{\mathbf{\hat{s}}^{(t:T)}} c$1 the injected context, $\nabla_{\mathbf{\hat{s}}^{(t:T)}} c$2 the actor-visible artifacts, and $\nabla_{\mathbf{\hat{s}}^{(t:T)}} c$3 the artifacts available by the relevant time. Its compliance-adjusted score is

$\nabla_{\mathbf{\hat{s}}^{(t:T)}} c$4

with $\nabla_{\mathbf{\hat{s}}^{(t:T)}} c$5 the violation rate over access, subsystem, and horizon breaches (Flynt, 22 Jun 2026). This yields a trajectory-aware judge that is deterministic, contract-based, and explicitly governance-sensitive.

The most general comparative formalism in the corpus is the risk-aware rulebook. Given a rule $\nabla_{\mathbf{\hat{s}}^{(t:T)}} c$6, a trajectory-conditioned environment map $\nabla_{\mathbf{\hat{s}}^{(t:T)}} c$7, and a risk measure $\nabla_{\mathbf{\hat{s}}^{(t:T)}} c$8, the induced risk-aware rule is

$\nabla_{\mathbf{\hat{s}}^{(t:T)}} c$9

where $$\text{PI-Metric} = \frac{1}{|\mathcal{A}|} \sum_{a \in \mathcal{A}} f(a, |\nabla_{\mathbf{\hat{s}}^{(t:T)}} c|) \cdot \text{Metric}(\mathbf{\hat{s}}^{(t:T)}_a, \mathbf{s}^{(t:T)}_a),$$0. A rulebook preorder over rules then induces a preorder over trajectories, ensuring consistency and preventing cyclic preferences (Wongpiromsarn, 4 Mar 2026). This formalism is especially important because it treats the environment response as trajectory-dependent rather than exogenous.

4. Domain instantiations

In autonomous driving and motion planning, trajectory-aware judging appears in at least four distinct forms. Planning-aware forecast evaluation asks whether prediction error matters for ego action, not merely whether it is geometrically close to the realized future (Ivanovic et al., 2021). DriveJudge asks whether a proposed driving behavior is reasonable in context by using a VLM to decide which deterministic metrics—LK, DDC, DAC, TTC, NC, EP, and COMFORT—are useful in the current scenario, then aggregating grounded rule outputs (Sun et al., 15 Jun 2026). CriticVLA judges a rough candidate future trajectory $$\text{PI-Metric} = \frac{1}{|\mathcal{A}|} \sum_{a \in \mathcal{A}} f(a, |\nabla_{\mathbf{\hat{s}}^{(t:T)}} c|) \cdot \text{Metric}(\mathbf{\hat{s}}^{(t:T)}_a, \mathbf{s}^{(t:T)}_a),$$1 and refines it into $$\text{PI-Metric} = \frac{1}{|\mathcal{A}|} \sum_{a \in \mathcal{A}} f(a, |\nabla_{\mathbf{\hat{s}}^{(t:T)}} c|) \cdot \text{Metric}(\mathbf{\hat{s}}^{(t:T)}_a, \mathbf{s}^{(t:T)}_a),$$2 through a critic that outputs six structured risk booleans and action recommendations before residual trajectory correction (Yang et al., 30 Apr 2026). RideJudge, in a different mobility setting, treats planned route and executed ride-hailing trajectory as adjudicable evidence and links route motifs such as drift, deviation, reverse driving, and arrival-then-leave to platform liability rules (Wu et al., 18 Mar 2026).

In agentic LLM systems, trajectory-aware judging is often motivated by hidden failure modes of plausible final answers. TRACE targets sabotage that is only visible through long-range evidence linking across benign-looking steps (Mittapalli et al., 5 Jun 2026). Trajectory Guard detects whether a plan is anomalous for a task or internally malformed as a sequence (Advani, 2 Jan 2026). GroundEval evaluates whether the agent searched the right places, fetched the right artifacts, respected access permissions, and reasoned within the correct temporal horizon (Flynt, 22 Jun 2026). AJ-Bench broadens the notion further by giving the judge access to live environments across search, data systems, and GUI tasks, thereby benchmarking information acquisition, state verification, and process verification as judge capabilities rather than agent capabilities (Shi et al., 20 Apr 2026). TRAJECT-Bench and BabelJudge contribute benchmark logic rather than a single deployed judge: the former decomposes tool-use quality into tool selection, argument correctness, dependency/order satisfaction, and trajectory satisfiability (He et al., 6 Oct 2025), while the latter audits whether pairwise judges are robust to argument corruption, tool swaps, hallucinated calls, missing steps, and trajectory-length padding (KC, 21 Jun 2026).

In robotics, the judged trajectory is often a visual execution trace rather than a symbolic tool log. PRM-as-a-Judge evaluates policy execution from trajectory videos or observation sequences via a progress potential, allowing near-misses, regressions, and stalls to be distinguished from early collapses and smooth successes (Ji et al., 23 Mar 2026). A plausible implication is that trajectory-aware judging in robotics is best understood as dense execution auditing rather than terminal success classification.

In urban mobility anomaly detection, the judge becomes a context-conditional density model. The context-aware anomaly detector models a subtrajectory $$\text{PI-Metric} = \frac{1}{|\mathcal{A}|} \sum_{a \in \mathcal{A}} f(a, |\nabla_{\mathbf{\hat{s}}^{(t:T)}} c|) \cdot \text{Metric}(\mathbf{\hat{s}}^{(t:T)}_a, \mathbf{s}^{(t:T)}_a),$$3 conditional on a context vector $$\text{PI-Metric} = \frac{1}{|\mathcal{A}|} \sum_{a \in \mathcal{A}} f(a, |\nabla_{\mathbf{\hat{s}}^{(t:T)}} c|) \cdot \text{Metric}(\mathbf{\hat{s}}^{(t:T)}_a, \mathbf{s}^{(t:T)}_a),$$4 that includes agent embedding and POI contextual embedding, and defines anomaly score as one minus expected reconstruction likelihood (Hu et al., 2024). This suggests a trajectory-aware judge need not always be normative or preference-based; it can also be a context-sensitive detector of behavioral implausibility.

5. Empirical findings

The empirical record consistently shows that endpoint-only or context-free judges miss failure modes that trajectory-aware judges can separate. In the planning-aware forecasting paper, two alternative predictions with identical ADE $$\text{PI-Metric} = \frac{1}{|\mathcal{A}|} \sum_{a \in \mathcal{A}} f(a, |\nabla_{\mathbf{\hat{s}}^{(t:T)}} c|) \cdot \text{Metric}(\mathbf{\hat{s}}^{(t:T)}_a, \mathbf{s}^{(t:T)}_a),$$5 and FDE $$\text{PI-Metric} = \frac{1}{|\mathcal{A}|} \sum_{a \in \mathcal{A}} f(a, |\nabla_{\mathbf{\hat{s}}^{(t:T)}} c|) \cdot \text{Metric}(\mathbf{\hat{s}}^{(t:T)}_a, \mathbf{s}^{(t:T)}_a),$$6 receive very different planning sensitivities—$$\text{PI-Metric} = \frac{1}{|\mathcal{A}|} \sum_{a \in \mathcal{A}} f(a, |\nabla_{\mathbf{\hat{s}}^{(t:T)}} c|) \cdot \text{Metric}(\mathbf{\hat{s}}^{(t:T)}_a, \mathbf{s}^{(t:T)}_a),$$7 for the path-conflicting prediction and $$\text{PI-Metric} = \frac{1}{|\mathcal{A}|} \sum_{a \in \mathcal{A}} f(a, |\nabla_{\mathbf{\hat{s}}^{(t:T)}} c|) \cdot \text{Metric}(\mathbf{\hat{s}}^{(t:T)}_a, \mathbf{s}^{(t:T)}_a),$$8 for the path-clearing one—and the planning-informed weighting assigns the path-conflicting prediction a $$\text{PI-Metric} = \frac{1}{|\mathcal{A}|} \sum_{a \in \mathcal{A}} f(a, |\nabla_{\mathbf{\hat{s}}^{(t:T)}} c|) \cdot \text{Metric}(\mathbf{\hat{s}}^{(t:T)}_a, \mathbf{s}^{(t:T)}_a),$$9 higher piADE and piFDE (Ivanovic et al., 2021). That is a direct demonstration that geometry alone can understate operational harm.

TRACE reports the strongest long-horizon evidence-linking results among the LLM-agent papers. On SHADE-Arena, it achieves aggregate precision $$\tau = (x_1, x_2, \dots, x_T), \qquad f(\tau) \rightarrow \{benign, malicious\},$$0, recall $$\tau = (x_1, x_2, \dots, x_T), \qquad f(\tau) \rightarrow \{benign, malicious\},$$1, and F1 $$\tau = (x_1, x_2, \dots, x_T), \qquad f(\tau) \rightarrow \{benign, malicious\},$$2, compared with F1 $$\tau = (x_1, x_2, \dots, x_T), \qquad f(\tau) \rightarrow \{benign, malicious\},$$3 for the full-trajectory monitor and $$\tau = (x_1, x_2, \dots, x_T), \qquad f(\tau) \rightarrow \{benign, malicious\},$$4 for the sequential monitor, while using $$\tau = (x_1, x_2, \dots, x_T), \qquad f(\tau) \rightarrow \{benign, malicious\},$$5 fewer LLM calls on average than the sequential baseline (Mittapalli et al., 5 Jun 2026). The largest gains appear on tasks that require long-range linkage, such as Performance Reviews, American Airlines, Spam Filter, Jira Tickets, and API Key Calendar. The paper also reports a stronger configuration with Claude Opus 4.7 reaching aggregate F1 $$\tau = (x_1, x_2, \dots, x_T), \qquad f(\tau) \rightarrow \{benign, malicious\},$$6, precision $$\tau = (x_1, x_2, \dots, x_T), \qquad f(\tau) \rightarrow \{benign, malicious\},$$7, and recall $$\tau = (x_1, x_2, \dots, x_T), \qquad f(\tau) \rightarrow \{benign, malicious\},$$8 at $$\tau = (x_1, x_2, \dots, x_T), \qquad f(\tau) \rightarrow \{benign, malicious\},$$9 average calls (Mittapalli et al., 5 Jun 2026).

Trajectory Guard shows that lightweight sequence-aware judges can compete with much larger language-model judges when the task is anomaly filtering over plans. It reports balanced-set F1 scores of $1$0–$1$1, recall of $1$2 on RAS-Eval and $1$3 on Who/When, and $1$4 ms latency on a T4 GPU, with speedups of $1$5 to $1$6 over LLM judge baselines (Advani, 2 Jan 2026). The ablation is especially informative: contrastive only reaches F1 $1$7, reconstruction only F1 $1$8, and the hybrid model F1 $1$9, indicating that task mismatch and sequence invalidity are complementary failure modes.

DriveJudge provides one of the clearest demonstrations that trajectory-aware rule selection improves human alignment in driving evaluation. On Driving Quality Classification, DriveJudge-8B reaches AUC $5$0, exceeding EPDMS at $5$1 by $5$2 AUC; on Trajectory Preference Selection, it reaches $5$3, exceeding DriveCritic at $5$4 by $5$5 points (Sun et al., 15 Jun 2026). The gains are largest on failure-sensitive metrics such as $5$6, where DriveJudge-8B reaches $5$7 compared with $5$8 for EPDMS (Sun et al., 15 Jun 2026).

CriticVLA supplies closed-loop evidence that judge-then-refine can outperform one-shot trajectory generation in autonomous driving. On Bench2Drive it reports Driving Score $5$9 and Success Rate $\ge 4$0, surpassing SimLingo’s DS $\ge 4$1 and SR $\ge 4$2, with especially large gains on merging and overtaking (Yang et al., 30 Apr 2026). The one-step refinement ablation is consistent with the paper’s theory: the first critic step raises SR from $\ge 4$3 in the zero-step setting to $\ge 4$4, while later steps show diminishing returns (Yang et al., 30 Apr 2026).

PRM-as-a-Judge provides the strongest evidence for micro-resolution in robotics. On RoboPulse, Robo-Dopamine reaches $\ge 4$5 average accuracy overall and $\ge 4$6 on the hardest Small scale, outperforming VLAC and GVL at $\ge 4$7 overall and surpassing foundation-model judges such as Gemini 3 Pro Preview at $\ge 4$8 and GPT-5.2 at $\ge 4$9 (Ji et al., 23 Mar 2026). The largest margin appears exactly where fine-grained physical evolution matters.

RideJudge reports $$\text{acc} = \frac{|\{j \in \mathcal{J} : j.\text{choice} = \text{reference}\}|}{|\{j \in \mathcal{J} : j.\text{choice} \in \{\text{ref}, \text{pert}\}\}|},$$0 overall accuracy on its ride-hailing adjudication benchmarks, exceeding DeepSeek-V3.1 at $$\text{acc} = \frac{|\{j \in \mathcal{J} : j.\text{choice} = \text{reference}\}|}{|\{j \in \mathcal{J} : j.\text{choice} \in \{\text{ref}, \text{pert}\}\}|},$$1 and Qwen3-VL-32B-Instruct at $$\text{acc} = \frac{|\{j \in \mathcal{J} : j.\text{choice} = \text{reference}\}|}{|\{j \in \mathcal{J} : j.\text{choice} \in \{\text{ref}, \text{pert}\}\}|},$$2 (Wu et al., 18 Mar 2026). Its ablations are unusually sharp: adding Chain-of-Adjudication raises performance from $$\text{acc} = \frac{|\{j \in \mathcal{J} : j.\text{choice} = \text{reference}\}|}{|\{j \in \mathcal{J} : j.\text{choice} \in \{\text{ref}, \text{pert}\}\}|},$$3 to $$\text{acc} = \frac{|\{j \in \mathcal{J} : j.\text{choice} = \text{reference}\}|}{|\{j \in \mathcal{J} : j.\text{choice} \in \{\text{ref}, \text{pert}\}\}|},$$4; adding knowledge-aware context refinement raises it from $$\text{acc} = \frac{|\{j \in \mathcal{J} : j.\text{choice} = \text{reference}\}|}{|\{j \in \mathcal{J} : j.\text{choice} \in \{\text{ref}, \text{pert}\}\}|},$$5 to $$\text{acc} = \frac{|\{j \in \mathcal{J} : j.\text{choice} = \text{reference}\}|}{|\{j \in \mathcal{J} : j.\text{choice} \in \{\text{ref}, \text{pert}\}\}|},$$6; and adding ordinal-sensitive reward raises it from $$\text{acc} = \frac{|\{j \in \mathcal{J} : j.\text{choice} = \text{reference}\}|}{|\{j \in \mathcal{J} : j.\text{choice} \in \{\text{ref}, \text{pert}\}\}|},$$7 to $$\text{acc} = \frac{|\{j \in \mathcal{J} : j.\text{choice} = \text{reference}\}|}{|\{j \in \mathcal{J} : j.\text{choice} \in \{\text{ref}, \text{pert}\}\}|},$$8 on the Appeal benchmark (Wu et al., 18 Mar 2026).

GroundEval demonstrates a different kind of empirical gap: not higher correlation with human preference, but exposure of invalid evidence paths that judge models miss. In one case study, Kimi-K2.6 scored an agent response $$\text{acc} = \frac{|\{j \in \mathcal{J} : j.\text{choice} = \text{reference}\}|}{|\{j \in \mathcal{J} : j.\text{choice} \in \{\text{ref}, \text{pert}\}\}|},$$9 and ChatGPT-5.5 scored it $$\Delta_{\text{pos}} = \text{WinRate}(\text{ref-first}) - \text{WinRate}(\text{pert-first}),$$0, while GroundEval assigned answer score $$\Delta_{\text{pos}} = \text{WinRate}(\text{ref-first}) - \text{WinRate}(\text{pert-first}),$$1 because the trace showed the agent never fetched the artifact the answer depended on (Flynt, 22 Jun 2026). AJ-Bench makes a similar point through aggregate F1: agentic judges improve over LLM-as-a-Judge by $$\Delta_{\text{pos}} = \text{WinRate}(\text{ref-first}) - \text{WinRate}(\text{pert-first}),$$2 F1 for GPT-5-mini-low and $$\Delta_{\text{pos}} = \text{WinRate}(\text{ref-first}) - \text{WinRate}(\text{pert-first}),$$3 for DeepSeek-v3.2 on average across domains (Shi et al., 20 Apr 2026).

Not all contributions are empirically complete. BabelJudge introduces trajectory-level perturbations and metrics such as tool accuracy, hallucination detection, and trajectory-length bias, but the paper does not report trajectory-side benchmark numbers beyond the framework definition (KC, 21 Jun 2026). TRAJECT-Bench is presented as a benchmark blueprint with metrics such as EM, Inclusion, Usage, Traj-Satisfy, Acc, and retrieval rate, but its most judge-relevant contribution is diagnostic decomposition rather than a single consolidated judge score (He et al., 6 Oct 2025).

6. Limitations and unresolved questions

A recurring limitation is dependence on a proxy model of what “matters.” In planning-aware forecasting, the usefulness of planning sensitivity depends on the quality of the learned proxy planning cost; if inverse optimal control captures the wrong driving objectives, the resulting judge can mis-weight forecast errors (Ivanovic et al., 2021). In PRM-as-a-Judge, the OPD metrics inherit any misalignment in the progress potential $$\Delta_{\text{pos}} = \text{WinRate}(\text{ref-first}) - \text{WinRate}(\text{pert-first}),$$4, and micro-resolution is an empirical property rather than a guaranteed theorem (Ji et al., 23 Mar 2026). In DriveJudge, the VLM still decides which rules to invoke; if it misreads the scene, it can suppress the wrong criterion even though the downstream rule functions are deterministic (Sun et al., 15 Jun 2026).

A second limitation is incomplete formalization. TRACE’s core mechanism is procedural rather than equation-based: there is no explicit scoring function for window ranking, evidence aggregation, or confidence computation (Mittapalli et al., 5 Jun 2026). BabelJudge’s agentic extension defines trajectory metrics in prose but omits the exact symbolic trajectory-level reliability formula (KC, 21 Jun 2026). TRAJECT-Bench supplies formal structures such as $$\Delta_{\text{pos}} = \text{WinRate}(\text{ref-first}) - \text{WinRate}(\text{pert-first}),$$5 and sequential templates but defines EM, Inclusion, Usage, and Traj-Satisfy descriptively rather than with closed-form equations (He et al., 6 Oct 2025). This suggests that trajectory-aware judging is currently ahead in benchmark design and system architecture relative to unified mathematical treatment.

A third limitation concerns coverage. TRACE is designed for sabotage detection and shows reduced precision in some domains; it also has a documented weakness on a particular Manipulation setting where plausible local justifications can mislead the iterative loop (Mittapalli et al., 5 Jun 2026). GroundEval explicitly evaluates only contractible evidence-path validity and does not score style, creativity, or open-ended quality (Flynt, 22 Jun 2026). Context-aware trajectory anomaly detection uses only agent ID and POI-derived geography, relies on simulated data from two cities, and does not incorporate broader context such as time of day or weather (Hu et al., 2024). RideJudge depends heavily on SynTraj and rasterized route evidence, and its paper gives limited fairness analysis for a quasi-judicial deployment setting (Wu et al., 18 Mar 2026).

A fourth limitation is deployment regime. DriveJudge is validated in open-loop evaluation rather than closed-loop policy optimization (Sun et al., 15 Jun 2026). CriticVLA is closed-loop, but its critic remains a one-step refiner rather than a full ranking or listwise evaluation system (Yang et al., 30 Apr 2026). AJ-Bench replays environments to final state for post hoc inspection rather than supporting full counterfactual re-execution of target agents (Shi et al., 20 Apr 2026). A plausible implication is that different trajectory-aware judges currently specialize in different slices of the problem: post hoc auditing, online filtering, candidate ranking, or closed-loop refinement.

Across these works, the trajectory-aware judge emerges not as a single algorithmic object but as a family of evaluators that elevate process, context, and environment response to first-class status. Some instantiate this through task-aware reweighting, some through interactive evidence acquisition, some through dense progress potentials, some through deterministic contract checking, and some through context-dependent rule invocation. What remains stable is the underlying claim: evaluation is often wrong when it ignores the path by which a result was produced.