---
title: Trajectory-Aware Judge
url: https://www.emergentmind.com/topics/trajectory-aware-judge
type: topic
---

# Trajectory-Aware Judge

A trajectory-aware judge is an evaluation paradigm that assigns quality by inspecting a trajectory as a structured process rather than reducing judgment to a final answer, a terminal state, or a task-agnostic geometric error. Across recent work, the judged object may be a forecasted motion plan, a robotic execution video, a tool-use trace, a stateful evidence path, or a multi-turn co-creation history; what unifies these settings is that correctness depends on intermediate actions, context, environment interaction, and downstream consequence, not only on endpoint plausibility [2107.10297], [2603.21669], [2606.07054], [2606.22737].

## 1. Conceptual scope and defining properties

The basic motivation for trajectory-aware judgment appears in several communities under different names. In trajectory forecasting for autonomous driving, conventional metrics such as displacement error and likelihood are criticized as task-agnostic because two predictions can be equally wrong geometrically yet have very different downstream planning consequences for the ego vehicle [2107.10297]. In long-horizon LLM-agent monitoring, single-pass or fixed-window judges are criticized because weak malicious evidence can be temporally distributed and only becomes meaningful when distant steps are linked [2606.07054]. In tool-using agents, judging only the final answer misses whether the correct tools were selected, parameterized, ordered, and grounded in valid evidence [2510.04550], [2606.22329].

A second defining property is environment or context dependence. Some judges are trajectory-aware because they condition judgment on who acted, what was accessible, which subsystems were visible, or what local semantics surrounded the trajectory. Context-aware mobility anomaly detection, for example, models trajectories conditional on agent identity and POI-derived geographic context rather than treating rare paths as globally anomalous in isolation [2410.19136]. GroundEval goes further by treating the evidence path itself as part of correctness: a response can be factually right yet still be invalid if the agent never retrieved the artifact it depended on, used role-inaccessible evidence, or answered from the wrong temporal horizon [2606.22737].

A third defining property is process sensitivity. In robotics, PRM-as-a-Judge argues that binary success collapses rich execution structure and obscures progress, efficiency, regression, and stagnation; the proposed judge therefore scores dense progress over the full trajectory rather than only the terminal outcome [2603.21669]. In agentic anomaly detection, Trajectory Guard similarly treats the trajectory as an ordered sequence whose internal structure must be modeled, rather than a bag of pooled step embeddings [2601.00516]. A trajectory-aware judge is therefore not merely “long-context evaluation.” It is an evaluator whose semantics change when the same endpoint is reached through a different path.

## 2. Recurrent architectural patterns

Recent work does not present one canonical architecture. Instead, several recurrent design patterns appear.

| Pattern | Representative work | Core mechanism |
|---|---|---|
| Planning-sensitive metric layer | [2107.10297] | Reweight base forecast errors by planning sensitivity |
| Adaptive evidence aggregation loop | [2606.07054] | Triage–Inspect–Judge with persistent cross-window evidence |
| Pairwise trajectory reliability audit | [2606.22329] | Controlled degradation of reference trajectories |
| Dense progress-potential judge | [2603.21669] | Per-state progress potential and OPD metrics |
| Sequence-aware anomaly discriminator | [2601.00516] | Task-plan alignment plus trajectory reconstruction |
| Rule-grounded environment-aware judge | [2606.17362] | VLM decides which deterministic driving rules to invoke |
| Deterministic evidence-path verifier | [2606.22737] | Contract checking over searches, fetches, citations, access, and time |

One family begins with an existing metric and adds trajectory relevance. The planning-aware metric proposed for trajectory forecasting does not replace ADE or FDE; it learns a proxy planning cost, differentiates that cost with respect to predicted trajectories, and uses the resulting sensitivities to reweight standard forecast errors [2107.10297]. A closely related logic appears in DriveJudge, where a VLM does not directly score a trajectory but first predicts which rule-based evaluators are relevant in the current scene, after which deterministic rule functions compute the actual grounded sub-scores [2606.17362].

Another family treats the judge as an active investigator. TRACE is the clearest example: instead of scoring the full trace in one pass, it performs Triage, then iterative Inspect actions with persistent “OBSERVATIONS SO FAR,” and finally Judge over an accumulated evidence log [2606.07054]. AJ-Bench generalizes this idea into environment-aware verification: the judge can open web pages, inspect filesystems or databases, interact with GUI environments, and use intermediate trajectory steps when direct state inspection is insufficient [2604.18240]. In these systems, trajectory awareness is inseparable from evidence acquisition.

A third family is explicitly process-modeling. PRM-as-a-Judge maps each observation along a robotic trajectory to a scalar progress potential and derives outcome, process, and diagnosis metrics from the resulting trace [2603.21669]. Trajectory Guard compresses an action sequence with a GRU encoder, reconstructs it with a GRU decoder, and jointly learns task-trajectory alignment and sequential validity, thereby combining contextual anomaly detection with structural anomaly detection [2601.00516]. CriticVLA applies the same principle to autonomous driving in a judge-and-revise setting: a VLA first proposes a rough future trajectory and then reuses the backbone as a critic that analyzes and refines that trajectory [2604.27366].

## 3. Formal scoring constructions

Several papers make the trajectory-aware judge mathematically explicit. In planning-aware trajectory forecasting, the scene at time \(t\) contains ego and other agents, and the learned proxy planning cost is
\[
c(\mathbf{s}^{(t)}, \mathbf{u}_\text{R}^{(t)}, \mathbf{\hat{s}}^{(t:T)}) = \theta^T \phi(\mathbf{s}^{(t)}, \mathbf{u}_\text{R}^{(t)}, \mathbf{\hat{s}}^{(t:T)}).
\]
Planning relevance is quantified by the gradient \(\nabla_{\mathbf{\hat{s}}^{(t:T)}} c\), and a generic planning-informed metric is defined as
\[
\text{PI-Metric} = \frac{1}{|\mathcal{A}|} \sum_{a \in \mathcal{A}} f(a, |\nabla_{\mathbf{\hat{s}}^{(t:T)}} c|) \cdot \text{Metric}(\mathbf{\hat{s}}^{(t:T)}_a, \mathbf{s}^{(t:T)}_a),
\]
with example weighting schemes based on normalization or softmax over planning sensitivities [2107.10297]. The formal point is that the base metric remains recognizable, but its aggregation becomes trajectory- and task-sensitive.

TRACE formalizes the monitoring problem as trajectory-level binary classification over a full post hoc trajectory,
\[
\tau = (x_1, x_2, \dots, x_T), \qquad f(\tau) \rightarrow \{benign, malicious\},
\]
but its aggregation is procedural rather than equation-based. The final decision rule is explicit: TRACE assigns a final suspicion score on a \(1\)–\(5\) scale, and trajectories scoring \(\ge 4\) are classified as malicious [2606.07054]. The important formalization is therefore not a closed-form score but a stateful evidence protocol.

BabelJudge provides explicit pairwise reliability formulas for judge models. It defines order-balanced accuracy,
\[
\text{acc} = \frac{|\{j \in \mathcal{J} : j.\text{choice} = \text{reference}\}|}{|\{j \in \mathcal{J} : j.\text{choice} \in \{\text{ref}, \text{pert}\}\}|},
\]
position bias,
\[
\Delta_{\text{pos}} = \text{WinRate}(\text{ref-first}) - \text{WinRate}(\text{pert-first}),
\]
verbosity susceptibility \(v\), order consistency \(\kappa_{\text{ord}}\), and the composite reliability score
\[
R = \underbrace{(0.6 \cdot \text{acc} + 0.4 \cdot \kappa_{\text{ord}})}_{\text{competence base}}
\cdot
\underbrace{(1 - 0.8 |\Delta_{\text{pos}}|)}_{\text{position penalty}}
\cdot
\underbrace{(1 - 0.7 \max(0, 2v - 1))}_{\text{verbosity penalty}}.
\]
For agent trajectories, the paper defines tool accuracy, argument accuracy, hallucination detection, and trajectory-length bias in prose, but does not print the exact trajectory-level formula [2606.22329]. That omission is itself methodologically revealing: trajectory-aware judging remains easier to motivate than to fully axiomatize.

PRM-as-a-Judge gives the densest trajectory formalism. A trajectory is
\[
\tau = (x_0, x_1, \dots, x_T),
\]
and the judge induces a task-aligned progress potential \(\Phi(x_t)\in[0,1]\). Pairwise progress is
\[
S(x_i, x_j) = \Phi(x_j) - \Phi(x_i),
\]
which satisfies macro-consistency by additivity. The OPD system then defines Milestone Coverage,
\[
\text{MC}(\tau) = \max \{q \in \{0, 0.25, 0.5, 0.75, 1\} \mid \exists t, \Phi(x_t) \ge q\},
\]
Max Progress,
\[
\text{MP}(\tau) = \max_{t \in [0, T]} \Phi(x_t),
\]
Path-weighted Progress Length,
\[
\text{PPL}(\tau) = \Phi(x_T)\cdot \frac{[\Phi(x_T)-\Phi(x_0)]_{+}}{\sum_{t=1}^{T}\left|\Phi(x_t)-\Phi(x_{t-1})\right|+\delta},
\]
Cumulative Regret Area, and Stagnation Ratio [2603.21669]. Here the trajectory-aware judge is literally a scalar field over execution history.

GroundEval formalizes evidence-path validity rather than semantic preference. In context mode, valid cited evidence is
\[
E_{\mathrm{valid}} = E \cap C \cap V_a \cap T_t,
\]
where \(E\) is the submitted evidence set, \(C\) the injected context, \(V_a\) the actor-visible artifacts, and \(T_t\) the artifacts available by the relevant time. Its compliance-adjusted score is
\[
S_{\text{adj}} = \left[ w_a \cdot S_{\text{answer}} + w_t \cdot S_{\text{traj}} \right] \cdot (1 - v)^2,
\]
with \(v\) the violation rate over access, subsystem, and horizon breaches [2606.22737]. This yields a trajectory-aware judge that is deterministic, contract-based, and explicitly governance-sensitive.

The most general comparative formalism in the corpus is the risk-aware rulebook. Given a rule \(r : T \times E \to \mathbb{R}\), a trajectory-conditioned environment map \(\mathcal{E}\), and a risk measure \(\rho\), the induced risk-aware rule is
\[
\hat r(\tau) = \max\{\rho(r_\tau)-\delta, 0\},
\]
where \(r_\tau(\omega)=r(\tau,\mathcal{E}_\tau(\omega))\). A rulebook preorder over rules then induces a preorder over trajectories, ensuring consistency and preventing cyclic preferences [2603.04603]. This formalism is especially important because it treats the environment response as trajectory-dependent rather than exogenous.

## 4. Domain instantiations

In autonomous driving and motion planning, trajectory-aware judging appears in at least four distinct forms. Planning-aware forecast evaluation asks whether prediction error matters for ego action, not merely whether it is geometrically close to the realized future [2107.10297]. DriveJudge asks whether a proposed driving behavior is reasonable in context by using a VLM to decide which deterministic metrics—LK, DDC, DAC, TTC, NC, EP, and COMFORT—are useful in the current scenario, then aggregating grounded rule outputs [2606.17362]. CriticVLA judges a rough candidate future trajectory \(A_0\) and refines it into \(A_1\) through a critic that outputs six structured risk booleans and action recommendations before residual trajectory correction [2604.27366]. RideJudge, in a different mobility setting, treats planned route and executed ride-hailing trajectory as adjudicable evidence and links route motifs such as drift, deviation, reverse driving, and arrival-then-leave to platform liability rules [2603.17328].

In agentic LLM systems, trajectory-aware judging is often motivated by hidden failure modes of plausible final answers. TRACE targets sabotage that is only visible through long-range evidence linking across benign-looking steps [2606.07054]. Trajectory Guard detects whether a plan is anomalous for a task or internally malformed as a sequence [2601.00516]. GroundEval evaluates whether the agent searched the right places, fetched the right artifacts, respected access permissions, and reasoned within the correct temporal horizon [2606.22737]. AJ-Bench broadens the notion further by giving the judge access to live environments across search, data systems, and GUI tasks, thereby benchmarking information acquisition, state verification, and process verification as judge capabilities rather than agent capabilities [2604.18240]. TRAJECT-Bench and BabelJudge contribute benchmark logic rather than a single deployed judge: the former decomposes tool-use quality into tool selection, argument correctness, dependency/order satisfaction, and trajectory satisfiability [2510.04550], while the latter audits whether pairwise judges are robust to argument corruption, tool swaps, hallucinated calls, missing steps, and trajectory-length padding [2606.22329].

In robotics, the judged trajectory is often a visual execution trace rather than a symbolic tool log. PRM-as-a-Judge evaluates policy execution from trajectory videos or observation sequences via a progress potential, allowing near-misses, regressions, and stalls to be distinguished from early collapses and smooth successes [2603.21669]. A plausible implication is that trajectory-aware judging in robotics is best understood as dense execution auditing rather than terminal success classification.

In urban mobility anomaly detection, the judge becomes a context-conditional density model. The context-aware anomaly detector models a subtrajectory \(\mathbf{x}_i\) conditional on a context vector \(\mathbf{c}_i\) that includes agent embedding and POI contextual embedding, and defines anomaly score as one minus expected reconstruction likelihood [2410.19136]. This suggests a trajectory-aware judge need not always be normative or preference-based; it can also be a context-sensitive detector of behavioral implausibility.

## 5. Empirical findings

The empirical record consistently shows that endpoint-only or context-free judges miss failure modes that trajectory-aware judges can separate. In the planning-aware forecasting paper, two alternative predictions with identical ADE \(=0.075\) and FDE \(=0.15\) receive very different planning sensitivities—\(0.90\) for the path-conflicting prediction and \(0.21\) for the path-clearing one—and the planning-informed weighting assigns the path-conflicting prediction a \(25\%\) higher piADE and piFDE [2107.10297]. That is a direct demonstration that geometry alone can understate operational harm.

TRACE reports the strongest long-horizon evidence-linking results among the LLM-agent papers. On SHADE-Arena, it achieves aggregate precision \(0.641\), recall \(0.844\), and F1 \(0.713\), compared with F1 \(0.577\) for the full-trajectory monitor and \(0.700\) for the sequential monitor, while using \(35\%\) fewer LLM calls on average than the sequential baseline [2606.07054]. The largest gains appear on tasks that require long-range linkage, such as Performance Reviews, American Airlines, Spam Filter, Jira Tickets, and API Key Calendar. The paper also reports a stronger configuration with Claude Opus 4.7 reaching aggregate F1 \(0.790\), precision \(0.786\), and recall \(0.825\) at \(11.0\) average calls [2606.07054].

Trajectory Guard shows that lightweight sequence-aware judges can compete with much larger language-model judges when the task is anomaly filtering over plans. It reports balanced-set F1 scores of \(0.88\)–\(0.94\), recall of \(0.86\) on RAS-Eval and \(0.92\) on Who/When, and \(32.48\) ms latency on a T4 GPU, with speedups of \(17\times\) to \(27\times\) over LLM judge baselines [2601.00516]. The ablation is especially informative: contrastive only reaches F1 \(0.82\), reconstruction only F1 \(0.75\), and the hybrid model F1 \(0.92\), indicating that task mismatch and sequence invalidity are complementary failure modes.

DriveJudge provides one of the clearest demonstrations that trajectory-aware rule selection improves human alignment in driving evaluation. On Driving Quality Classification, DriveJudge-8B reaches AUC \(78.90\), exceeding EPDMS at \(57.67\) by \(21.23\) AUC; on Trajectory Preference Selection, it reaches \(82.83\%\), exceeding DriveCritic at \(76.33\%\) by \(6.5\) points [2606.17362]. The gains are largest on failure-sensitive metrics such as \(\mathrm{AP}^{-}\), where DriveJudge-8B reaches \(59.04\) compared with \(26.34\) for EPDMS [2606.17362].

CriticVLA supplies closed-loop evidence that judge-then-refine can outperform one-shot trajectory generation in autonomous driving. On Bench2Drive it reports Driving Score \(88.02 \pm 0.17\) and Success Rate \(73.33 \pm 0.27\), surpassing SimLingo’s DS \(85.07\) and SR \(67.27\), with especially large gains on merging and overtaking [2604.27366]. The one-step refinement ablation is consistent with the paper’s theory: the first critic step raises SR from \(61.36\) in the zero-step setting to \(72.73\), while later steps show diminishing returns [2604.27366].

PRM-as-a-Judge provides the strongest evidence for micro-resolution in robotics. On RoboPulse, Robo-Dopamine reaches \(0.83\) average accuracy overall and \(0.80\) on the hardest Small scale, outperforming VLAC and GVL at \(0.71\) overall and surpassing foundation-model judges such as Gemini 3 Pro Preview at \(0.66\) and GPT-5.2 at \(0.53\) [2603.21669]. The largest margin appears exactly where fine-grained physical evolution matters.

RideJudge reports \(88.41\%\) overall accuracy on its ride-hailing adjudication benchmarks, exceeding DeepSeek-V3.1 at \(75.25\%\) and Qwen3-VL-32B-Instruct at \(65.55\%\) [2603.17328]. Its ablations are unusually sharp: adding Chain-of-Adjudication raises performance from \(63.75\) to \(68.81\); adding knowledge-aware context refinement raises it from \(70.56\) to \(91.86\); and adding ordinal-sensitive reward raises it from \(70.51\) to \(91.86\) on the Appeal benchmark [2603.17328].

GroundEval demonstrates a different kind of empirical gap: not higher correlation with human preference, but exposure of invalid evidence paths that judge models miss. In one case study, Kimi-K2.6 scored an agent response \(0.9\) and ChatGPT-5.5 scored it \(0.85\), while GroundEval assigned answer score \(0.000\) because the trace showed the agent never fetched the artifact the answer depended on [2606.22737]. AJ-Bench makes a similar point through aggregate F1: agentic judges improve over LLM-as-a-Judge by \(+13.41\) F1 for GPT-5-mini-low and \(+12.85\) for DeepSeek-v3.2 on average across domains [2604.18240].

Not all contributions are empirically complete. BabelJudge introduces trajectory-level perturbations and metrics such as tool accuracy, hallucination detection, and trajectory-length bias, but the paper does not report trajectory-side benchmark numbers beyond the framework definition [2606.22329]. TRAJECT-Bench is presented as a benchmark blueprint with metrics such as EM, Inclusion, Usage, Traj-Satisfy, Acc, and retrieval rate, but its most judge-relevant contribution is diagnostic decomposition rather than a single consolidated judge score [2510.04550].

## 6. Limitations and unresolved questions

A recurring limitation is dependence on a proxy model of what “matters.” In planning-aware forecasting, the usefulness of planning sensitivity depends on the quality of the learned proxy planning cost; if inverse optimal control captures the wrong driving objectives, the resulting judge can mis-weight forecast errors [2107.10297]. In PRM-as-a-Judge, the OPD metrics inherit any misalignment in the progress potential \(\Phi\), and micro-resolution is an empirical property rather than a guaranteed theorem [2603.21669]. In DriveJudge, the VLM still decides which rules to invoke; if it misreads the scene, it can suppress the wrong criterion even though the downstream rule functions are deterministic [2606.17362].

A second limitation is incomplete formalization. TRACE’s core mechanism is procedural rather than equation-based: there is no explicit scoring function for window ranking, evidence aggregation, or confidence computation [2606.07054]. BabelJudge’s agentic extension defines trajectory metrics in prose but omits the exact symbolic trajectory-level reliability formula [2606.22329]. TRAJECT-Bench supplies formal structures such as \(G_T=(V,E)\) and sequential templates but defines EM, Inclusion, Usage, and Traj-Satisfy descriptively rather than with closed-form equations [2510.04550]. This suggests that trajectory-aware judging is currently ahead in benchmark design and system architecture relative to unified mathematical treatment.

A third limitation concerns coverage. TRACE is designed for sabotage detection and shows reduced precision in some domains; it also has a documented weakness on a particular Manipulation setting where plausible local justifications can mislead the iterative loop [2606.07054]. GroundEval explicitly evaluates only contractible evidence-path validity and does not score style, creativity, or open-ended quality [2606.22737]. Context-aware trajectory anomaly detection uses only agent ID and POI-derived geography, relies on simulated data from two cities, and does not incorporate broader context such as time of day or weather [2410.19136]. RideJudge depends heavily on SynTraj and rasterized route evidence, and its paper gives limited fairness analysis for a quasi-judicial deployment setting [2603.17328].

A fourth limitation is deployment regime. DriveJudge is validated in open-loop evaluation rather than closed-loop policy optimization [2606.17362]. CriticVLA is closed-loop, but its critic remains a one-step refiner rather than a full ranking or listwise evaluation system [2604.27366]. AJ-Bench replays environments to final state for post hoc inspection rather than supporting full counterfactual re-execution of target agents [2604.18240]. A plausible implication is that different trajectory-aware judges currently specialize in different slices of the problem: post hoc auditing, online filtering, candidate ranking, or closed-loop refinement.

Across these works, the trajectory-aware judge emerges not as a single algorithmic object but as a family of evaluators that elevate process, context, and environment response to first-class status. Some instantiate this through task-aware reweighting, some through interactive evidence acquisition, some through dense progress potentials, some through deterministic contract checking, and some through context-dependent rule invocation. What remains stable is the underlying claim: evaluation is often wrong when it ignores the path by which a result was produced.

Source: https://www.emergentmind.com/topics/trajectory-aware-judge