Offline Preference-Based Trajectory Evaluation

Abstract: Offline evaluation of agentic systems often collapses trajectories to terminal success, discarding information about partial progress and inducing widespread ties, creating substantial statistical inefficiency by reducing effective sample size and weakening the ability to distinguish systems. We propose preference-based trajectory evaluation, which compares trajectories directly through temporal preferences over progress and time-to-return profiles. We find that, across diverse agentic and interactive benchmarks, standard success-based metrics produce tied comparisons on roughly 75% of instances, whereas trajectory-aware preferences reduce ties to roughly 35%, improving discriminative power, ranking stability, and data efficiency. Our results suggest that benchmark saturation, often attributed to poor data collection or problem difficulty, may also be explained by the choice of evaluation measure.

• The paper demonstrates that scalar success metrics cause significant information loss by collapsing nuanced trajectory progress into binary outcomes.
• It introduces preference-based methods (LR, RPP, IPP) that leverage temporal structure to enhance discriminative power, reliability, and ranking accuracy.
• Empirical results confirm improved data efficiency and sensitivity, reducing tie rates and enabling robust comparisons across diverse agentic benchmarks.

Offline Preference-Based Trajectory Evaluation: An Expert Analysis

Introduction

This paper addresses the limitations of dominant evaluation schemas for agentic systems—specifically, the widespread practice of collapsing agent trajectories to scalar binary success. The authors critically analyze the consequences of this approach, notably statistical inefficiency, high tie rates, and reduced sensitivity in distinguishing system performance. Through systematic empirical analyses and the introduction of preference-based evaluation grounded in trajectory structure, the paper provides rigorous evidence that incorporating temporal preferences and trajectory-aware measures fundamentally increases the discriminative power, stability, and data efficiency of standard benchmarks. The study spans reinforcement learning (RL) environments and contemporary agentic AI benchmarks.

Core Problem: Information Loss in Scalar Success Metrics

The traditional method of computing agent evaluation via binary success introduces two forms of information loss:

1. Partial Progress Collapse: Trajectories are reduced to a single terminal bit, discarding all intermediate progress information.
2. Temporal Collapse: Different temporal performance profiles that reach the same endpoint are deemed indistinguishable, resulting in widespread ties, especially as models saturate on a benchmark.

These phenomena are illustrated in the paper via trajectory visualizations showing that even when agents ostensibly score equivalently under success-rate, their progression towards the task goal can be meaningfully different (Figure 1).

Figure 1: Two unsuccessful trajectories differentiated by partial returns, and two successful trajectories distinguished by temporal progression; both are tied under scalar success-rate measurement.

Empirically, across surveyed benchmarks, this binary collapse yields tie rates up to 75% for instance-level system comparisons. As model performance on benchmarks increases, more trajectories become indistinguishably successful, further weakening sensitivity and leading to benchmark saturation—a spurious ceiling on measure informativeness induced by metric definition rather than innate problem difficulty.

Methodology: Preference-Based Evaluation without Scalar Reduction

The authors propose preference-based evaluation frameworks centered on direct trajectory comparison via preferences, eschewing the reduction to intermediate scalar metrics. The core family of preferences is as follows:

• Lexicographic Return (LR): Prefers trajectories that achieve maximal return earliest; ties are broken by lower return levels achieved sooner.
• Return-Paired Preference (RPP): Integrates time-to-return over all return levels; measures cumulative temporal advantage, providing a fine-grained comparison at every possible subgoal.
• Interval-Paired Preference (IPP): Compares time increments required to transition between return segments, operationalizing local temporal efficiency rather than global cumulative speed.

The structure of these preferences is visualized in the paper (Figure 2).

Figure 2: Example of Lexicographic Return splitting ties by earliest achievement of return levels, compared to Return-Paired and Interval-Paired Preferences which aggregate local or global temporal advantages over all segments.

Unlike temporal discounting (linear, exponential, or SPL), these measures do not require hyperparameters relating time to utility, and do not force commensurability between temporal and task return dimensions.

Empirical Results

The paper's evaluation is comprehensive, covering a range of agentic domains (AgentBoard, OpenHands-Index, TheAgentCompany, TALES, and RL subgoal environments). Across all domains, the authors compare preference-based methods (LR, RPP, IPP) to metric-based baselines (SR, PR, SPL) with respect to validity, reliability, sensitivity, and data efficiency.

Validity

Preference-based methods maintain alignment with existing scalar measures while substantially improving system ordering on tasks where models are close in final success but divergent in trajectory structure. Correlation analyses show two clusters: scalar-metric and trajectory-preference families, which induce distinct (but aligned) model rankings. On synthetic oracle tasks with controlled system degrading, preference metrics achieve over 94% correctness in recovering ground-truth orderings, while success-rate and terminal progress detect less than 13%.

Reliability

Split-half and leave-one-out analyses demonstrate that preference-based methods yield more stable rankings. For instance, RPP/LR methods have higher mean split-half correlations ($\sim$0.83–0.85) and lower sign-flip rates under subsampling compared to SPL, which shows notable instability and sensitivity to instance perturbation.

Sensitivity

Preference-based measures materially reduce tie rates (down to $\sim$35\% vs. $\sim$75\% for SR). The distribution of pairwise preferences is correspondingly far less discrete, extracting more information per trajectory comparison. Critically, these methods maintain high discriminative power (over 78% of model pairs detected as significantly different, vs. 58% for SR and 60% for SPL after FDR correction), even as model populations achieve high success rates on a benchmark (see Figure 3).

Figure 3: Measure distributions (AgentBoard) showing preference-based metrics assign values widely in $(0, 1)$, reducing ties.

Moreover, analyses of discriminative power as a function of mean task success-rate substantiate that preference-based metrics are not only more sensitive on "middling" tasks but continue to resolve distinctions as benchmarks saturate, contrasting sharply with the collapse of scalar metrics in both low- and high-success regimes.

Data Efficiency

Preference-based evaluation methods achieve correct, significant system ranking with fewer evaluation samples. In controlled RL tasks, RPP achieves near-maximal oracle agreement at 50–60\% of the evaluation budget required by SPL. This reduction in sample complexity has direct implications for the cost and feasibility of large-scale agent evaluation (see Figure 4).

Figure 4: Subsampling analysis shows trajectory-based preferences (RPP) produce stable rankings with fewer samples compared to scalar metrics (SPL, PR, SR).

Theoretical and Practical Implications

The results challenge the prevailing attribution of benchmark saturation solely to problem difficulty or poor data curation; instead, they demonstrate that insensitive metric design is itself a primary source of saturation and inefficiency. By enabling more discriminative and statistically robust comparisons, preference-based evaluation frameworks presented here enhance the lifecycle of existing benchmarks and extend their discriminative phase, even as model performance increases.

The offline, trajectory-based preference approach also avoids the limitations of online preference collection (arena-style systems) by permitting counterfactual, multi-system, and rapid comparative evaluation strictly from logged data—without human-in-the-loop experimentation.

From a practical standpoint, the findings call for an urgent reevaluation of measurement standards in agentic AI evaluation, especially where current leaderboards and conference tracks disproportionately rely on binary or scalar metrics. On the theoretical side, preference-based evaluations align with current insights in RLHF and policy learning, where direct comparison of behavioral traces is often more expressive and statistically robust than scalar reward aggregation.

Limitations

The utility of trajectory-aware preferences relies on the suitability and alignment of intermediate and partial rewards with true progress; in domains with sparse, noisy, or misaligned reward annotations, such measures may confound real capability with reward structure artifacts. Furthermore, while temporal preference is a justified construct for many agentic and economic contexts, its blanket application must be guided by domain desiderata.

Conclusion

By exposing and remedying the inherent limitations of scalar success-rate metrics, this work rigorously demonstrates the advantages of preference-based trajectory evaluation for agentic systems. The proposed measures yield significant, robust improvements in sensitivity, validity, reliability, and data efficiency, without requiring additional data or hyperparameter tuning. The approach resets standards for offline evaluation, diminishing the impact of metric-induced saturation and enabling benchmarks to serve as more reliable discriminators among advanced agentic systems.