Preference-Based Trajectory Evaluation

• The paper introduces a framework that transforms comparative assessments of full trajectories into scalar reward functions to enhance evaluation across diverse domains.
• Preference-Based Trajectory Evaluation is a unified approach that leverages pairwise trajectory comparisons, integrating methods from robotics, control, and autonomous systems.
• It employs active learning and advanced query design to balance informative feedback with user interpretability, reducing cognitive load and improving decision criteria.

Preference-based trajectory evaluation denotes a family of formulations in which full trajectories, trajectory segments, or trajectory-induced outcomes are compared through preferences and then converted into scalar scores, rankings, or control objectives. In the cited literature, this role is played by learned reward functions for assistive robots, surrogate MPC objectives learned from trajectory pairs, offline trajectory-aware metrics that compare return-over-time profiles, persona-conditioned rankings over successful GUI execution traces, trajectory-level preference datasets for tool-using language agents, visual preference scoring for cinematic camera motion, and scene-level rankings of multi-agent futures in autonomous driving (Dennler et al., 2024, Krupa et al., 27 Nov 2025, Diaz, 16 Jun 2026, Lin et al., 13 Apr 2026, Cao et al., 11 Apr 2026, Li et al., 2 Apr 2026, Azevedo et al., 3 Jul 2025). This suggests that the topic is best understood not as a single algorithm, but as a unifying viewpoint: evaluation should act on the trajectory itself, or on its realized consequences, rather than collapsing behavior to terminal success, isolated action correctness, or a hand-crafted objective.

1. Scope and problem formulations

In assistive robotics, the problem is framed as a human-robot teaching loop in which the robot presents candidate behaviors and the user ranks them, allowing the system to learn a reward function representing what the user likes. The same paper argues that existing preference-learning methods often optimize only for eventual reward-learning performance while neglecting the user’s experience during repeated interaction, and introduces Covariance Matrix Adaptation Evolution Strategies with Information Gain (CMA-ES-IG) to balance informative queries with trajectories that appear to improve over time (Dennler et al., 2024).

In control, the objective is to learn a scalar surrogate over feasible finite-horizon trajectories from pairwise human preferences and then use that surrogate directly as the MPC objective. One formulation learns a single-trajectory score $\sigma(T;\theta)$ so that pairwise predictions are induced by comparing two scalar evaluations, while a later paper adds active learning strategies to reduce the number of human preference queries needed to obtain a well-performing classifier (Krupa et al., 27 Nov 2025, Hasnaouy et al., 15 May 2026).

In offline agent evaluation, the problem is stated differently: logged trajectories are available, but standard offline evaluation collapses them to terminal success. “Offline Preference-Based Trajectory Evaluation” instead compares trajectories directly through temporal preferences over progress and time-to-return profiles, arguing that success-only metrics induce widespread ties and weaken the ability to distinguish systems (Diaz, 16 Jun 2026). In mobile GUI agents, the preferred object is not the final task outcome but the successful execution trajectory that better matches a privacy persona; in finance, the evaluated object is the complete multi-turn tool-calling trajectory, including reasoning, tool invocations, tool responses, and final answer (Lin et al., 13 Apr 2026, Cao et al., 11 Apr 2026).

Other domains extend the same logic. In trajectory-based air traffic flow management, historical tactical flight plans are converted into per-flight, per-trajectory preference scores $G_p^f$, which are then optimized jointly with delay and capacity constraints (Giovanni et al., 2022). In cinematic camera generation, the trajectory itself is not judged directly in geometric parameter space; instead, it is rendered in Unity and scored through the visual consequences of the trajectory (Li et al., 2 Apr 2026). In multi-agent vehicle forecasting, preferences are assigned to joint future scene modes using an automatic ranking cost, and fine-tuning shifts probability mass toward more coherent futures without extra inference-time computation (Azevedo et al., 3 Jul 2025).

2. Trajectory representations and preference models

Several papers formalize trajectory evaluation through latent scalar scores. In assistive robotics, a trajectory is defined as

$\xi = (s_0,a_0,s_1,a_1,\ldots,s_T,a_T),$

with feature aggregation

$\Phi(\xi)=\sum_{i=0}^T \phi(s_i),$

and linear reward

$R(\xi)=\omega^T\cdot \Phi(\xi),$

where $\omega$ denotes latent user preference weights (Dennler et al., 2024). In MPC, the trajectory is written as $T=(X,U,Y)$, and the central object is a scalar single-trajectory surrogate $\sigma(T;\theta)$ such that

$$P_\pi(T_i,T_j;\theta)=\frac{1}{1+\exp(\sigma(T_i;\theta)-\sigma(T_j;\theta))}.$$

This makes lower $\sigma(T;\theta)$ correspond to a more preferred trajectory and allows the learned surrogate to be inserted directly into the MPC optimization (Krupa et al., 27 Nov 2025).

Preference-based RL papers adopt related pairwise comparison models over trajectory features. In “Dueling RL,” the preference probability between two trajectories is

$G_p^f$0

which allows non-Markovian rewards because $G_p^f$1 can be any feature of the entire trajectory (Pacchiano et al., 2021). “Provable Reward-Agnostic Preference-Based Reinforcement Learning” uses the Bradley-Terry-Luce model

$G_p^f$2

with linearized trajectory reward $G_p^f$3 (Zhan et al., 2023). “Efficient Preference-Based Reinforcement Learning” works in discounted MDPs with

$G_p^f$4

thereby learning a global reward model from pairwise trajectory judgments (Schlaginhaufen et al., 11 Jun 2025).

Some formulations evaluate trajectories without introducing a latent reward model over state-action features. In offline agent evaluation, a trajectory is represented as a return-over-time function $G_p^f$5 and equivalently by a time-to-return profile $G_p^f$6, where $G_p^f$7 is the first time the agent reaches return at least $G_p^f$8 (Diaz, 16 Jun 2026). In mobile GUI agents, the input is

$G_p^f$9

the output is a trajectory

$\xi = (s_0,a_0,s_1,a_1,\ldots,s_T,a_T),$0

and supervision is given as persona-conditioned preference triplets $\xi = (s_0,a_0,s_1,a_1,\ldots,s_T,a_T),$1 satisfying

$\xi = (s_0,a_0,s_1,a_1,\ldots,s_T,a_T),$2

with the key point that both trajectories are feasible and preference reflects privacy alignment rather than correctness (Lin et al., 13 Apr 2026).

3. Query construction, active learning, and hybrid optimization

A major strand of the literature concerns how to choose informative comparisons. In assistive robotics, the Bayesian active baseline selects a query maximizing expected information gain about $\xi = (s_0,a_0,s_1,a_1,\ldots,s_T,a_T),$3, but pure information gain may generate highly informative yet low-quality or seemingly arbitrary queries. CMA-ES improves perceived quality over repeated interactions but can collapse to nearly indistinguishable options. CMA-ES-IG combines both ideas by sampling candidate trajectory features from the CMA-ES distribution and then selecting the displayed subset using an information-gain-style objective under the current belief over $\xi = (s_0,a_0,s_1,a_1,\ldots,s_T,a_T),$4 (Dennler et al., 2024).

Preference-based RL papers make the same issue explicit at the policy level. “Provable Reward-Agnostic Preference-Based Reinforcement Learning” uses reward-agnostic experimental design to collect trajectory pairs whose feature differences make the hidden reward identifiable before human feedback is requested (Zhan et al., 2023). “Active RLHF via Best Policy Learning from Trajectory Preference Feedback” proposes Posterior Sampling for Preference Learning, in which two posterior samples over reward and dynamics produce two policies, two trajectories are rolled out, and preference feedback is obtained on that duel (Agnihotri et al., 31 Jan 2025). “Efficient Preference-Based Reinforcement Learning” replaces optimistic exploration with randomized exploration and then improves query complexity by collecting batches of trajectory pairs and applying D-optimal experimental design to select informative comparison queries; the batch structure also enables parallelization of preference queries (Schlaginhaufen et al., 11 Jun 2025).

Active learning appears explicitly in MPC objective learning. The pool-based strategy selects trajectory pairs that are both uncertain under the current surrogate and diverse relative to previously labeled comparisons, with acquisition

$\xi = (s_0,a_0,s_1,a_1,\ldots,s_T,a_T),$5

while the query-synthesis strategy incorporates new trajectories using the current surrogate-driven MPC (Hasnaouy et al., 15 May 2026). Human-interface work approaches the same problem from a different angle: FARPLS dynamically adjusts labeling order according to users’ familiarities, difficulties of the trajectory pair, and level of disagreements, treating query presentation itself as part of the preference-learning pipeline (Lyu et al., 2024).

4. Evaluation criteria and empirical measures

The literature also expands what counts as a useful evaluation signal. In offline agent evaluation, the proposed temporal preferences include Lexicographic Return (LR), Return-Paired Preference (RPP), and Interval-Paired Preference (IPP). Across benchmarks, standard success rate produces tied comparisons on 74.9% of instances, partial return on 49.71%, SPL on 63.42%, whereas LR, RPP, and IPP reduce ties to 33.9%, 34.82%, and 35.09%, respectively. Mean split-half ranking reliability reaches 0.85 for both LR and RPP, and average discriminative power under FDR reaches 77.81 for LR and 78.35 for RPP (Diaz, 16 Jun 2026).

In assistive robotics, simulation uses alignment, quality, regret, and area under the curve. Alignment is

$\xi = (s_0,a_0,s_1,a_1,\ldots,s_T,a_T),$6

quality is the average reward of the displayed query,

$\xi = (s_0,a_0,s_1,a_1,\ldots,s_T,a_T),$7

and regret compares the best trajectory under the true reward to the best under the estimated reward. Reported quality AUCs for CMA-ES-IG are $\xi = (s_0,a_0,s_1,a_1,\ldots,s_T,a_T),$8 for $\xi = (s_0,a_0,s_1,a_1,\ldots,s_T,a_T),$9, respectively, whereas IG remains near zero (Dennler et al., 2024).

Other domains define trajectory-sensitive metrics tailored to their process structure. TIPO evaluates Step Success Rate, PAS-S, PAS-U, Compliance, Non-compliance, and Persona Distinction; TIPO achieves 65.60% SR, 46.22 Compliance, and 66.67% PD, improving on DPO, SimPO, and IPO while preserving executability (Lin et al., 13 Apr 2026). FinTrace evaluates long-horizon financial tool-calling with nine metrics under four axes—action correctness, execution efficiency, process quality, and output quality—and constructs a trajectory-level preference dataset of 8,196 examples (Cao et al., 11 Apr 2026). In vehicle forecasting, scene consistency is measured by Scene Collision Rate and probability-weighted Scene Collision Rate, while accuracy is measured by MinJointFDE$\Phi(\xi)=\sum_{i=0}^T \phi(s_i),$0; on AV2, QCNet fine-tuned with preference optimization reduces SCR from $\Phi(\xi)=\sum_{i=0}^T \phi(s_i),$1 to $\Phi(\xi)=\sum_{i=0}^T \phi(s_i),$2 and pSCR from $\Phi(\xi)=\sum_{i=0}^T \phi(s_i),$3 to $\Phi(\xi)=\sum_{i=0}^T \phi(s_i),$4 with only a slight increase in MinJointFDE$\Phi(\xi)=\sum_{i=0}^T \phi(s_i),$5 from $\Phi(\xi)=\sum_{i=0}^T \phi(s_i),$6 to $\Phi(\xi)=\sum_{i=0}^T \phi(s_i),$7 (Azevedo et al., 3 Jul 2025).

5. Domain-specific realizations

The same evaluative pattern appears in very different trajectory spaces. In assistive robotics, users rank physical handover trajectories for a JACO2 arm and social gesture trajectories for a Blossom robot; trajectories are represented using learned nonlinear features from autoencoders, and each human-study query contains three executable robot behaviors (Dennler et al., 2024). In offline preference-based RL, trajectory-level reward inference is followed by relabeling of an offline dataset and return-to-go computation, but “In-Dataset Trajectory Return Regularization” argues that inaccurate step-wise rewards lead to optimistic trajectory stitching and proposes conditioning on in-dataset trajectory returns to regularize policy learning (Tu et al., 2024).

Mobile GUI agents treat the step as the minimal interaction unit, with each step containing executable structured action, current screenshot, XML/UI state, optional reasoning text, and semantic action description. The distinctive challenge is that privacy preferences alter trajectory structure, length, and branching pattern, so TIPO aligns variable-length trajectories using semantic no_action placeholders and then applies preference-intensity weighting and padding gating (Lin et al., 13 Apr 2026). In financial tool-calling, a trajectory is the complete multi-turn interaction

$\Phi(\xi)=\sum_{i=0}^T \phi(s_i),$8

including reasoning, tool invocations, tool responses, and final answer; DPO is then applied at the trajectory level with tool-response tokens masked out (Cao et al., 11 Apr 2026).

Camera control and air traffic management illustrate two further variants. VERTIGO evaluates camera trajectories indirectly through rendered previews in Unity, scores them with cyclic semantic similarity

$\Phi(\xi)=\sum_{i=0}^T \phi(s_i),$9

and uses the induced pairwise preferences for DPO post-training, reducing Missing Rate from 0.387 to 0.008 relative to GenDoP while keeping FCD nearly unchanged at 4.17 versus 4.22 (Li et al., 2 Apr 2026). In trajectory-based ATFM, historical tactical flight plans are clustered into consistent 4D trajectory options, a random forest assigns each flight a preference score $R(\xi)=\omega^T\cdot \Phi(\xi),$0 for each candidate trajectory, and optimization then maximizes

$R(\xi)=\omega^T\cdot \Phi(\xi),$1

subject to assignment, capacity, and delay-budget constraints (Giovanni et al., 2022).

6. Human factors, limitations, and recurring design tensions

A persistent theme is that preference quality depends on the evaluation interface and on the relationship between formal informativeness and human usability. FARPLS reports that users may overlook non-salient task features, establish biased preference criteria during preference elicitation because of partial observations, and experience mental fatigue when given many pairs to compare. In a between-subjects study with 42 participants and 105 pairs per person, FARPLS improves labeling consistency from $R(\xi)=\omega^T\cdot \Phi(\xi),$2 to $R(\xi)=\omega^T\cdot \Phi(\xi),$3 without significantly increasing cognitive load (Lyu et al., 2024). In assistive robotics, CMA-ES-IG receives the highest mean ease-of-use rating ($R(\xi)=\omega^T\cdot \Phi(\xi),$4) and the highest perceived behavioral adaptation score ($R(\xi)=\omega^T\cdot \Phi(\xi),$5), indicating that query distinguishability and visible improvement both matter in repeated human-robot teaching (Dennler et al., 2024).

Several limitations recur across domains. Many methods assume a linear reward model in learned or hand-specified features, Bradley-Terry or logistic comparison noise, and conditional independence in ranking decompositions (Dennler et al., 2024, Krupa et al., 27 Nov 2025, Zhan et al., 2023). Offline evaluation based on temporal preferences assumes normalized returns are nondecreasing and that temporal preference is an appropriate evaluative principle; the paper notes that stronger oracle evidence comes from synthetic environments with known ground-truth ordering (Diaz, 16 Jun 2026). TIPO relies on a rule-based scoring scheme with LLM assistance and on semantic alignment by inserting no_action, which may become brittle when trajectories differ radically in structure (Lin et al., 13 Apr 2026). VERTIGO depends on the availability of a rendering engine and on domain-adapted visual-language evaluators, while its pairwise supervision is score-induced rather than directly human-grounded (Li et al., 2 Apr 2026).

Taken together, these works suggest a stable conceptual shift. Effective trajectory evaluation is not exhausted by terminal success, isolated reward labels, or geometric plausibility. It requires a representation of what a full trajectory means to a rater, a mechanism for turning comparative judgments into scalar evaluators or pairwise preferences, and a query or interface design that preserves both information content and human interpretability. Across robotics, control, offline evaluation, agentic language systems, GUI agents, cinematic camera control, traffic management, and motion forecasting, preference-based trajectory evaluation serves precisely that role.