Generalizing Preference-based Reinforcement Learning: a Rationality Model for Incomparability

Published 13 Jul 2026 in cs.LG | (2607.11432v1)

Abstract: In this work, we study the reinforcement learning (RL) problem from pairwise trajectory comparisons provided by a human expert. We generalize preference-based RL by formalizing a novel setting in which the expert can also label trajectory pairs as incomparable, i.e., when neither trajectory dominates the other. We introduce the learning problem and the desiderata that its solution should satisfy. Then, we propose a novel Bradley-Terry-inspired rationality model that effectively captures incomparabilities and infers a multi-dimensional reward function, and we study its properties. We provide a sample complexity analysis for learning the model parameters when a dataset is available. Finally, we evaluate our model's ability to reconstruct a reward function that aligns with the expert's comparisons in simulated environments and to recover the Pareto frontier of policies, along with a robustness analysis across varying levels of expert rationality.

Abstract PDF Upgrade to Chat

Authors (4)

Summary

The paper introduces a novel framework that integrates human feedback with multi-objective rationality to handle incomparable trajectory comparisons.
It employs a multi-objective Bradley-Terry model to infer vector-valued rewards and accurately recover Pareto-optimal policy frontiers.
The framework overcomes limitations in scalar reward models and provides theoretical guarantees and empirical evidence across multiple RL environments.

Generalizing Preference-Based Reinforcement Learning with a Rationality Model for Incomparability

This paper introduces a formal framework to address comparison-based reinforcement learning (CbRL) problems, where human experts may label trajectory pairs as incomparable rather than always expressing clear preferences or indifference. The work generalizes standard preference-based RL (PbRL) by fundamentally integrating multi-objective rationality and incomparability into the learning protocol. Through a multi-objective Bradley-Terry (MOBT) model, the paper provides the first robust mathematical architecture to infer vector-valued rewards and recover the Pareto frontier of policies from trajectory-level comparisons that include incomparability.

Motivation and Problem Setting

Traditional RL assumes a scalar reward function, demanding substantial effort in reward engineering and often leading to misalignment and unwanted behaviors. Preference-based RL (PbRL) sidesteps classical reward design by inferring reward functions from pairwise preferences, but it assumes all comparisons map to ordered outcomes (preference or indifference). In practice, especially in domains with conflicting objectives (e.g., speed and safety), experts frequently encounter pairs they consider incomparable—neither trajectory dominates across all objectives.

The paper formalizes this observation by distinguishing incomparability from indifference. If expert feedback is honest, incomparability signals a multi-dimensional latent reward structure and the potential for a partial order over behaviors, cf. multi-objective RL (MORL). The CbRL formulation thus unifies MORL (multi-objective, requires dense reward) and PbRL (preference feedback but scalar utility), providing a missing mathematical link: efficiently learning Pareto frontiers through comparison-only human data.

The MDPC Framework and Model Desiderata

The authors introduce the Markov Decision Process with Comparisons (MDPC): a tuple encompassing a standard reward-free MDP and a comparison probability distribution $\rho(\cdot \mid \tau, \tau')$ , where $\tau$ and $\tau'$ are trajectories and possible labels include strict preference, reverse preference, indifference, and incomparability.

Human rationality in CbRL is modeled through a utility function $\bm{u}: \mathcal{T} \to \mathbb{R}^d$ , which quantifies multiple latent objectives. The rationality model $f: \mathbb{R}^d \to \Delta(\Comps)$ maps utility differences into outcome probabilities. A set of formal desiderata is imposed on $f$ :

In the limit where one trajectory dominates another across all objectives (large positive or negative differences), the model should predict clear preference with high likelihood.
If utilities are element-wise equal, indifference should dominate.
Along non-standard diagonals (mixed-sign differences, i.e., some objectives are improved and others degraded), incomparability must emerge as the most probable feedback.

These desiderata are illustrated geometrically for 2D utility spaces.

Figure 1: Reach the target.

The impossibility theorem is established: no comparison model meeting all desiderata yields a convex negative log-likelihood (NLL), making the learning problem inherently non-convex and challenging optimization and convergence analysis.

Multi-Objective Bradley-Terry Rationality Model (MOBT)

A novel MOBT model is defined to extend the classical Bradley-Terry model to multi-objective settings:

The scores for strict preference and reverse preference are aligned with the sum of utility differences.
The score for indifference is set as a learned offset ( $\alpha$ ).
The incomparability score is proportional to the standard deviation of the utility difference vector plus a learnable offset ( $\beta$ ), so that dispersion in utility improvements across objectives elevates the probability of incomparability.

Figure 2: Bradley-Terry model.

The MOBT is shown to satisfy all desiderata for the CbRL setting, and to reduce to classical BT in the single-objective limit. The incomparability mechanism mathematically underpins the link between multiple objectives and partial comparability.

Theoretical Analysis

Sample complexity analysis under feature-based utilities (linear in trajectory features) and boundedness yields a KL-divergence guarantee between the true and learned comparison distributions:

$\mathrm{KL}(P_{\theta^*}\Vert P_{\widehat{\theta}}) \leq \widetilde{O}\left(R \Lambda d k \sqrt{\frac{\log(1/\delta)}{N}}\right)$

where $N$ is sample count, $\tau$ 0 dimensionality, $\tau$ 1 feature dimension, $\tau$ 2 maximum parameter norm, and $\tau$ 3 maximum feature difference. For local optima (since global cannot generally be found), an additional error term proportional to the probability of observing incomparabilities is introduced, showing that the fraction of incomparable samples is the principal source of nonconvexity-induced error.

Empirical Evaluation

Experiments are conducted in multi-objective GridWorlds, LQRs, and MO-Gymnasium's Hopper, with comparison labels synthesized from ground-truth objectives. The key results and findings are as follows:

Reward Model Recovery: MOBT accurately reconstructs the multi-dimensional reward matrices from pairwise trajectory comparisons, robust to translation invariance due to the BT foundation.
Pareto Frontier Recovery: In LQR and GridWorld settings, the reward inferred via MOBT enables explicit recovery of Pareto-optimal policies; the reconstructed frontier closely matches the frontier under the true reward.
Sample Efficiency: Test KL-divergence between estimated and true comparison probabilities decreases with increased sample counts; both diversity (pairs) and redundancy (multiple labels per pair) contribute.
Inadequacy of Scalar Models: Standard PbRL models (Bradley-Terry, Thurstone-Mosteller, Rao-Kupper, Davidson) either discard incomparable pairs, misclassify incomparability as indifference, or collapse the reward to a single dimension—none match MOBT's performance or ability to generalize (see Table~\ref{tab:tv_baselines} in the appendices).
Robustness: Performance degrades gracefully under simulated irrationality (random mistakes), with the KL gap widening as the error rate approaches 100%.

Figure 3: Bradley-Terry model.

Implications and Open Challenges

Practical Implications:

The CbRL framework enables practitioners to leverage realistic human-in-the-loop comparisons, embracing the reality that not all trajectory pairs can be meaningfully ranked—especially in tasks where conflicting stakeholder objectives must be navigated. By inferring the latent Pareto frontier from preference data—including abstentions—the framework brings multi-objective decision support into RLHF pipelines, potentially lowering cognitive load on human labelers.

Theoretical Advancement:

The work rigorously demonstrates that the expressiveness required to model incomparability necessarily induces nonconvexity in the learning objective. This imposes a barrier to efficient convex optimization and directs future work toward algorithmic strategies for robust local search or exploiting problem structure (e.g., restriction to convex subsets, tailored regularization).

Future Developments:

Online CbRL: Extending analysis and algorithms to settings where pairs are selected adaptively could yield great gains in sample efficiency via active preference elicitation.
Human Subject Studies: Real-world datasets with indifference and incomparability are absent; collecting and sharing such data will be crucial for advancing CbRL as a standard in RLHF.
High-Dimensional Scaling: As the number of objectives grows, pairwise feedback becomes less informative (credit assignment ambiguity increases). Methods balancing human effort and representational power will be required.
Beyond Bradley-Terry: Non-BT models (energy-based, embedding-based) could further improve performance or explainability in more complex preference landscapes.

Conclusion

The paper provides a mathematically principled and empirically validated solution for preference-based RL with incomparability, using a multi-objective Bradley-Terry rationality model. Through both sharp theoretical analysis and comprehensive simulation, it establishes that CbRL unlocks new vistas for learning over partial orders and Pareto sets from noisy, realistic human feedback—a necessary progression for deploying RL in pluralistic, real-world environments where scalar reward alignment fails.

Reference:

"Generalizing Preference-based Reinforcement Learning: a Rationality Model for Incomparability" (2607.11432)

Markdown Report Issue