---
title: Preference-Based Trajectory Evaluation
url: https://www.emergentmind.com/topics/preference-based-trajectory-evaluation
type: topic
---

# Preference-Based Trajectory Evaluation

Preference-based trajectory evaluation denotes a family of formulations in which full trajectories, trajectory segments, or trajectory-induced outcomes are compared through preferences and then converted into scalar scores, rankings, or control objectives. In the cited literature, this role is played by learned reward functions for assistive robots, surrogate MPC objectives learned from trajectory pairs, offline trajectory-aware metrics that compare return-over-time profiles, persona-conditioned rankings over successful GUI execution traces, trajectory-level preference datasets for tool-using language agents, visual preference scoring for cinematic camera motion, and scene-level rankings of multi-agent futures in autonomous driving [2411.11182] [2511.22502] [2606.17541] [2604.11259] [2604.10015] [2604.02467] [2507.02406]. This suggests that the topic is best understood not as a single algorithm, but as a unifying viewpoint: evaluation should act on the trajectory itself, or on its realized consequences, rather than collapsing behavior to terminal success, isolated action correctness, or a hand-crafted objective.

## 1. Scope and problem formulations

In assistive robotics, the problem is framed as a human-robot teaching loop in which the robot presents candidate behaviors and the user ranks them, allowing the system to learn a reward function representing what the user likes. The same paper argues that existing preference-learning methods often optimize only for eventual reward-learning performance while neglecting the user’s experience during repeated interaction, and introduces Covariance Matrix Adaptation Evolution Strategies with Information Gain (CMA-ES-IG) to balance informative queries with trajectories that appear to improve over time [2411.11182].

In control, the objective is to learn a scalar surrogate over feasible finite-horizon trajectories from pairwise human preferences and then use that surrogate directly as the MPC objective. One formulation learns a single-trajectory score $\sigma(T;\theta)$ so that pairwise predictions are induced by comparing two scalar evaluations, while a later paper adds active learning strategies to reduce the number of human preference queries needed to obtain a well-performing classifier [2511.22502] [2605.16071].

In offline agent evaluation, the problem is stated differently: logged trajectories are available, but standard offline evaluation collapses them to terminal success. “Offline Preference-Based Trajectory Evaluation” instead compares trajectories directly through temporal preferences over progress and time-to-return profiles, arguing that success-only metrics induce widespread ties and weaken the ability to distinguish systems [2606.17541]. In mobile GUI agents, the preferred object is not the final task outcome but the successful execution trajectory that better matches a privacy persona; in finance, the evaluated object is the complete multi-turn tool-calling trajectory, including reasoning, tool invocations, tool responses, and final answer [2604.11259] [2604.10015].

Other domains extend the same logic. In trajectory-based air traffic flow management, historical tactical flight plans are converted into per-flight, per-trajectory preference scores $G_p^f$, which are then optimized jointly with delay and capacity constraints [2211.06526]. In cinematic camera generation, the trajectory itself is not judged directly in geometric parameter space; instead, it is rendered in Unity and scored through the visual consequences of the trajectory [2604.02467]. In multi-agent vehicle forecasting, preferences are assigned to joint future scene modes using an automatic ranking cost, and fine-tuning shifts probability mass toward more coherent futures without extra inference-time computation [2507.02406].

## 2. Trajectory representations and preference models

Several papers formalize trajectory evaluation through latent scalar scores. In assistive robotics, a trajectory is defined as
\[
\xi = (s_0,a_0,s_1,a_1,\ldots,s_T,a_T),
\]
with feature aggregation
\[
\Phi(\xi)=\sum_{i=0}^T \phi(s_i),
\]
and linear reward
\[
R(\xi)=\omega^T\cdot \Phi(\xi),
\]
where $\omega$ denotes latent user preference weights [2411.11182]. In MPC, the trajectory is written as $T=(X,U,Y)$, and the central object is a scalar single-trajectory surrogate $\sigma(T;\theta)$ such that
\[
P_\pi(T_i,T_j;\theta)=\frac{1}{1+\exp(\sigma(T_i;\theta)-\sigma(T_j;\theta))}.
\]
This makes lower $\sigma(T;\theta)$ correspond to a more preferred trajectory and allows the learned surrogate to be inserted directly into the MPC optimization [2511.22502].

Preference-based RL papers adopt related pairwise comparison models over trajectory features. In “Dueling RL,” the preference probability between two trajectories is
\[
\mathbb P(\tau_1 \succ \tau_2)=\sigma\!\big(\langle \phi(\tau_1)-\phi(\tau_2),\theta^*\rangle\big),
\]
which allows non-Markovian rewards because $\phi(\tau)$ can be any feature of the entire trajectory [2111.04850]. “Provable Reward-Agnostic Preference-Based Reinforcement Learning” uses the Bradley-Terry-Luce model
\[
P(o=1)=P(\tau^1 \succ \tau^0)=\sigma(r^*(\tau^1)-r^*(\tau^0)),
\]
with linearized trajectory reward $r^*(\tau)=\langle \phi(\tau),\theta^*\rangle$ [2305.18505]. “Efficient Preference-Based Reinforcement Learning” works in discounted MDPs with
\[
\phi(\tau):=\sum_{h=0}^{\infty}\gamma^h \phi(s_h,a_h),
\qquad
P(\tau\succ \tau')=\sigma\!\left(\left\langle \theta^*,\phi(\tau)-\phi(\tau')\right\rangle\right),
\]
thereby learning a global reward model from pairwise trajectory judgments [2506.09508].

Some formulations evaluate trajectories without introducing a latent reward model over state-action features. In offline agent evaluation, a trajectory is represented as a return-over-time function $f:\mathbb Z_{>0}\to [0,1]$ and equivalently by a time-to-return profile $g:[0,1]\to \mathbb Z_{>0}\cup\{\infty\}$, where $g(R)$ is the first time the agent reaches return at least $R$ [2606.17541]. In mobile GUI agents, the input is
\[
x=(g,o,h,p),
\]
the output is a trajectory
\[
y=(y_1,\dots,y_{|y|}),
\]
and supervision is given as persona-conditioned preference triplets $(x,y^+,y^-)$ satisfying
\[
\pi_\theta(y^+ \mid x) > \pi_\theta(y^- \mid x),
\]
with the key point that both trajectories are feasible and preference reflects privacy alignment rather than correctness [2604.11259].

## 3. Query construction, active learning, and hybrid optimization

A major strand of the literature concerns how to choose informative comparisons. In assistive robotics, the Bayesian active baseline selects a query maximizing expected information gain about $\omega$, but pure information gain may generate highly informative yet low-quality or seemingly arbitrary queries. CMA-ES improves perceived quality over repeated interactions but can collapse to nearly indistinguishable options. CMA-ES-IG combines both ideas by sampling candidate trajectory features from the CMA-ES distribution and then selecting the displayed subset using an information-gain-style objective under the current belief over $\omega$ [2411.11182].

Preference-based RL papers make the same issue explicit at the policy level. “Provable Reward-Agnostic Preference-Based Reinforcement Learning” uses reward-agnostic experimental design to collect trajectory pairs whose feature differences make the hidden reward identifiable before human feedback is requested [2305.18505]. “Active RLHF via Best Policy Learning from Trajectory Preference Feedback” proposes Posterior Sampling for Preference Learning, in which two posterior samples over reward and dynamics produce two policies, two trajectories are rolled out, and preference feedback is obtained on that duel [2501.18873]. “Efficient Preference-Based Reinforcement Learning” replaces optimistic exploration with randomized exploration and then improves query complexity by collecting batches of trajectory pairs and applying D-optimal experimental design to select informative comparison queries; the batch structure also enables parallelization of preference queries [2506.09508].

Active learning appears explicitly in MPC objective learning. The pool-based strategy selects trajectory pairs that are both uncertain under the current surrogate and diverse relative to previously labeled comparisons, with acquisition
\[
a(\mathcal P)=U(\mathcal P)\,D^k(\mathcal P),
\]
while the query-synthesis strategy incorporates new trajectories using the current surrogate-driven MPC [2605.16071]. Human-interface work approaches the same problem from a different angle: FARPLS dynamically adjusts labeling order according to users’ familiarities, difficulties of the trajectory pair, and level of disagreements, treating query presentation itself as part of the preference-learning pipeline [2403.06267].

## 4. Evaluation criteria and empirical measures

The literature also expands what counts as a useful evaluation signal. In offline agent evaluation, the proposed temporal preferences include Lexicographic Return (LR), Return-Paired Preference (RPP), and Interval-Paired Preference (IPP). Across benchmarks, standard success rate produces tied comparisons on **74.9%** of instances, partial return on **49.71%**, SPL on **63.42%**, whereas LR, RPP, and IPP reduce ties to **33.9%**, **34.82%**, and **35.09%**, respectively. Mean split-half ranking reliability reaches **0.85** for both LR and RPP, and average discriminative power under FDR reaches **77.81** for LR and **78.35** for RPP [2606.17541].

In assistive robotics, simulation uses alignment, quality, regret, and area under the curve. Alignment is
\[
\text{alignment}=\cos(\omega^{est},\omega^*),
\]
quality is the average reward of the displayed query,
\[
\frac{1}{|Q|}\sum_{i=0}^{|Q|} \omega^*\cdot \Phi(\xi_i),
\]
and regret compares the best trajectory under the true reward to the best under the estimated reward. Reported quality AUCs for CMA-ES-IG are \(0.746, 0.673, 0.527\) for \(d=8,16,32\), respectively, whereas IG remains near zero [2411.11182].

Other domains define trajectory-sensitive metrics tailored to their process structure. TIPO evaluates Step Success Rate, PAS-S, PAS-U, Compliance, Non-compliance, and Persona Distinction; TIPO achieves **65.60% SR**, **46.22 Compliance**, and **66.67% PD**, improving on DPO, SimPO, and IPO while preserving executability [2604.11259]. FinTrace evaluates long-horizon financial tool-calling with nine metrics under four axes—action correctness, execution efficiency, process quality, and output quality—and constructs a trajectory-level preference dataset of **8,196** examples [2604.10015]. In vehicle forecasting, scene consistency is measured by Scene Collision Rate and probability-weighted Scene Collision Rate, while accuracy is measured by MinJointFDE\(_K\); on AV2, QCNet fine-tuned with preference optimization reduces SCR from \(8.64\) to \(6.29\) and pSCR from \(2.86\) to \(1.23\) with only a slight increase in MinJointFDE\(_6\) from \(1.366\) to \(1.375\) [2507.02406].

## 5. Domain-specific realizations

The same evaluative pattern appears in very different trajectory spaces. In assistive robotics, users rank physical handover trajectories for a JACO2 arm and social gesture trajectories for a Blossom robot; trajectories are represented using learned nonlinear features from autoencoders, and each human-study query contains three executable robot behaviors [2411.11182]. In offline preference-based RL, trajectory-level reward inference is followed by relabeling of an offline dataset and return-to-go computation, but “In-Dataset Trajectory Return Regularization” argues that inaccurate step-wise rewards lead to optimistic trajectory stitching and proposes conditioning on in-dataset trajectory returns to regularize policy learning [2412.09104].

Mobile GUI agents treat the step as the minimal interaction unit, with each step containing executable structured action, current screenshot, XML/UI state, optional reasoning text, and semantic action description. The distinctive challenge is that privacy preferences alter trajectory structure, length, and branching pattern, so TIPO aligns variable-length trajectories using semantic `no_action` placeholders and then applies preference-intensity weighting and padding gating [2604.11259]. In financial tool-calling, a trajectory is the complete multi-turn interaction
\[
\tau=(m_1,\ldots,m_N),
\]
including reasoning, tool invocations, tool responses, and final answer; DPO is then applied at the trajectory level with tool-response tokens masked out [2604.10015].

Camera control and air traffic management illustrate two further variants. VERTIGO evaluates camera trajectories indirectly through rendered previews in Unity, scores them with cyclic semantic similarity
\[
s_i^{\text{sem}} = \frac{\phi(p)\cdot \phi(\hat p_i)}{\|\phi(p)\|\,\|\phi(\hat p_i)\|},
\]
and uses the induced pairwise preferences for DPO post-training, reducing Missing Rate from **0.387** to **0.008** relative to GenDoP while keeping FCD nearly unchanged at **4.17** versus **4.22** [2604.02467]. In trajectory-based ATFM, historical tactical flight plans are clustered into consistent 4D trajectory options, a random forest assigns each flight a preference score \(G_p^f\) for each candidate trajectory, and optimization then maximizes
\[
\sum_{f\in F}\sum_{p\in P_f}\sum_{d\in B_f} G_p^f\, y_{pd}^f
\]
subject to assignment, capacity, and delay-budget constraints [2211.06526].

## 6. Human factors, limitations, and recurring design tensions

A persistent theme is that preference quality depends on the evaluation interface and on the relationship between formal informativeness and human usability. FARPLS reports that users may overlook non-salient task features, establish biased preference criteria during preference elicitation because of partial observations, and experience mental fatigue when given many pairs to compare. In a between-subjects study with **42** participants and **105** pairs per person, FARPLS improves labeling consistency from \(M=0.79\) to \(M=0.89\) without significantly increasing cognitive load [2403.06267]. In assistive robotics, CMA-ES-IG receives the highest mean ease-of-use rating (\(M=5.50\)) and the highest perceived behavioral adaptation score (\(M=5.18\)), indicating that query distinguishability and visible improvement both matter in repeated human-robot teaching [2411.11182].

Several limitations recur across domains. Many methods assume a linear reward model in learned or hand-specified features, Bradley-Terry or logistic comparison noise, and conditional independence in ranking decompositions [2411.11182] [2511.22502] [2305.18505]. Offline evaluation based on temporal preferences assumes normalized returns are nondecreasing and that temporal preference is an appropriate evaluative principle; the paper notes that stronger oracle evidence comes from synthetic environments with known ground-truth ordering [2606.17541]. TIPO relies on a rule-based scoring scheme with LLM assistance and on semantic alignment by inserting `no_action`, which may become brittle when trajectories differ radically in structure [2604.11259]. VERTIGO depends on the availability of a rendering engine and on domain-adapted visual-language evaluators, while its pairwise supervision is score-induced rather than directly human-grounded [2604.02467].

Taken together, these works suggest a stable conceptual shift. Effective trajectory evaluation is not exhausted by terminal success, isolated reward labels, or geometric plausibility. It requires a representation of what a full trajectory means to a rater, a mechanism for turning comparative judgments into scalar evaluators or pairwise preferences, and a query or interface design that preserves both information content and human interpretability. Across robotics, control, offline evaluation, agentic language systems, GUI agents, cinematic camera control, traffic management, and motion forecasting, preference-based trajectory evaluation serves precisely that role.

Source: https://www.emergentmind.com/topics/preference-based-trajectory-evaluation