---
title: Preference & Rating Based RL
url: https://www.emergentmind.com/topics/preference-based-and-rating-based-rl
type: topic
---

# Preference & Rating Based RL

Preference-based and rating-based reinforcement learning are two human-in-the-loop RL paradigms designed to circumvent the challenges of reward engineering by leveraging direct human feedback. The central distinction lies in whether feedback is provided via relative preferences between trajectory pairs, or via scalar/ordinal ratings of individual trajectories. Both approaches have motivated extensive algorithmic, theoretical, and empirical research in recent years, with rigorous comparisons, practical algorithm design, and statistical guarantees now available.

## 1. Formal Definitions and Core Protocols

In preference-based reinforcement learning (PbRL), the agent receives pairwise comparisons: given two trajectory segments $\sigma^0$, $\sigma^1$, the supervisor provides a label $y \in \{0, 1, 0.5\}$ indicating preference or tie. The standard stochastic model for preferences is the Bradley–Terry or logistic model, parameterizing the probability of a human preferring one segment over another as
\[
P[\sigma^1 \succ \sigma^0] = \frac{\exp( R(\sigma^1) )}{ \exp( R(\sigma^0) ) + \exp( R(\sigma^1) ) }
\]
where $R(\sigma)$ is the predicted return of the segment, possibly inferred from a parametric or non-parametric reward model [2303.00957, 2306.14111].

In rating-based reinforcement learning (RbRL), feedback is a scalar or multiclass label $y \in \mathbb{R}$ or $y \in \{0, \ldots, n-1\}$ attached to a single trajectory. The rating model seeks to assign predicted returns $\hat R(\sigma)$ to bins or scales, and is typically trained with a multi-class loss on human-labeled examples [2307.16348].

Both methods use the inferred reward information to update the policy via standard RL optimization, most often with a modular structure: (1) reward inference from human feedback, (2) RL or control via learned reward.

## 2. Preference-Based RL: Algorithms and Theoretical Guarantees

Modern PbRL algorithms follow a generic loop: (1) collect policy-generated trajectories, (2) sample candidate trajectory pairs, (3) query human or simulated oracle for preference feedback, (4) update a reward/reward-like model using cross-entropy (logistic) loss, and (5) improve the policy using the latest inferred rewards [2111.03026, 1908.01289, 2310.14554]. Variants such as Dueling RL [2111.04850] use upper-confidence exploration and feature-based generalized linear preference models.

Key theoretical results:
- In tabular, linear, and general function-approximation MDPs, optimal or nearly-optimal regret and sample/query complexity bounds are attainable, matching those for scalar-reward RL up to polynomial factors in model complexity [2310.14554, 2306.14111, 2205.11140].
- For any utility-based preference model (e.g., logistic/Bradley-Terry), preference-based RL is not fundamentally more difficult than standard RL: sample and query complexity depend on the eluder dimension or other structural complexity measures of the reward/preference class [2306.14111].
- Key regret bounds are of the form $\widetilde O(d\sqrt{T})$ for $d$-dimensional models, similar to reward-based bandits [2111.04850, 1908.01289].

Algorithmic advances include:
- Active preference querying via uncertainty, disagreement, or coverage to maximize feedback informativeness [2111.03026].
- Parameter-efficient approaches such as Inverse Preference Learning, which avoid explicit reward modeling and fit the Q-function directly to preference data via inverse Bellman operators [2305.15363].
- Direct policy optimization methods that align policies to human preferences without reward modeling, using contrastive or other pairwise objectives [2301.12842].

Recent work also integrates large pretrained models and selective human supervision to reduce annotation costs without sacrificing performance [2502.01616].

## 3. Rating-Based RL: Methods and Empirical Findings

Rating-based RL algorithms use scalar (e.g., 1–5 stars) or categorical ratings to fit a reward predictor directly by regression/classification. The multi-class loss is a generalization of the cross-entropy to $n$ ordinal bins, often incorporating adaptive binning based on observed label distributions [2307.16348]. The RL agent then uses the learned scalar reward as the return for standard control methods.

Empirical and user studies show that RbRL offers important advantages over PbRL:
- RbRL achieves similar or superior policy performance with 30–60% fewer human feedback queries in both synthetic and real-user studies, attributed to the higher information content per rating [2307.16348].
- Ratings are more consistent for users, faster to provide (∼60% faster than preferences in timed studies), and less cognitively demanding, leading to higher confidence and lower frustration.
- However, ratings are subject to scale miscalibration, inter-rater variability, and bias; binning and adaptive normalization can mitigate but not eliminate such effects. Ratings may also lose fine granularity within bins or at scale boundaries.

A key limitation is that scalar ratings may obscure critical events within a trajectory, as the global score may fail to reveal temporally localized failures or successes that are easily surfaced by pairwise preferences [2303.00957].

## 4. Hybrid Algorithms and Extensions

Hybrid and generalized feedback frameworks have emerged:
- Multi-way (K-wise) preference queries generalize pairwise models, offering improved statistical efficiency; a single $K$-way query can be decomposed into up to $\lfloor K/2 \rfloor$ independent pairwise comparisons, reducing total query cost [2306.14111, 2205.11140].
- Preferences can be converted to ratings (e.g., via the Borda method), and ratings can generate pairwise preferences—a conversion used to unify theoretical analyses [2307.16348, 2301.12842].
- Some methods (e.g., Preference Transformer [2303.00957]) model human feedback as a non-Markovian, temporally aware sequence-to-label problem, with transformer architectures attending to “critical events.”
- Vision-language models and self-supervised semantic adaptation have been used to both generate pseudo-feedback and enable efficient transfer across tasks, with uncertainty filtering maintaining label quality [2502.01616].

Reward inference pipelines may also incorporate auxiliary self-supervised representation learning to capture transition dynamics (as in REED [2211.06527]), which substantially reduces label requirements in PbRL.

## 5. Robustness, Sample Complexity, and Practical Considerations

Robustness to human “irrationality,” noise, and annotation error is a central concern:
- Simulated teacher models with parameters for rationality, myopia, random mistakes, skipping, and indifference have become standard in benchmarks (see B-Pref [2111.03026]).
- Both PbRL and RbRL are affected by query selection: uncertainty-based sampling most consistently yields faster learning, while coverage-based and random sampling are less effective [2111.03026].
- Off-policy reward relabeling and pretraining for enhanced exploration further increase feedback efficiency.
- RbRL is more vulnerable to label imbalance (most ratings 'bad'), while PbRL is sensitive to ambiguity in comparisons near the decision boundary.
- Algorithms now achieve $\tilde O(\poly(d,H)\sqrt{K})$ regret with efficient query schedules under general function approximation [2205.11140]. Randomized Thompson-style approaches yield optimal trade-offs between regret and query complexity [2310.14554].

## 6. Comparative Insights and Open Directions

|          | Preference-Based RL                                  | Rating-Based RL                     |
|----------|-----------------------------------------------------|-------------------------------------|
| Query    | Pairwise or K-way comparison                        | Scalar or ordinal rating per traj.  |
| Loss     | Cross-entropy (Bradley-Terry, logistic)             | Multi-class or regression           |
| Info per query | 1 bit (binary), $< \log_2(\#items!)$ (K-way)  | $\sim \log_2(n)$ bits (n classes)   |
| Sampling | Active (uncertainty, disagreement, coverage)        | Uncertainty; ensemble variance      |
| User study | Slower, more cognitively demanding                | Faster, less frustrating            |
| Pitfalls | Can miss margin, global quality; less calibration   | Sensitive to scale, label drift     |
| Granularity | Captures relative, subtle distinctions           | Lacks fine-grained intra-bin order  |
| Theory   | Matched regret/query with reward-based RL           | Theory analogous under regression   |

Both paradigms are theoretically competitive with reward-based RL under mild assumptions, and mixture/hybrid approaches (rating followed by local preferences, etc.) are promising open directions. Further challenges include robust scale calibration, adaptive querying, handling non-stationarity in human judgment, integrating richer forms of feedback (natural language, demonstrations), and validating results in large-scale real-world systems [2307.16348, 2303.00957, 2111.03026].

## 7. Notable Models, Benchmarks, and Architectures

- **Preference Transformer**: Non-Markovian sequence model with attention weights, extracting critical events and temporal dependencies in preferences [2303.00957].
- **PEBBLE and PrefPPO**: Preference-driven pipelines with off-policy relabeling and unsupervised pretraining [2111.03026, 2211.06527].
- **Inverse Preference Learning (IPL)**: Q-function based PbRL dispensing with explicit reward models [2305.15363].
- **Direct Preference Policy Optimization (DPPO)**: Contrastive policy optimization, aligning policies directly with preferences [2301.12842].
- **B-Pref**: Benchmark for systematic study, including a suite of synthetic “irrational” teacher models [2111.03026].
- **PrefVLM**: Annotation-efficient preference RL via pre-trained vision-language models and selective human feedback [2502.01616].
- **REED**: Key innovation in integrating self-supervised dynamics modeling into reward inference, drastically improving data efficiency [2211.06527].
- **Dueling Posterior Sampling**: Bayesian posterior sampling over reward and dynamics in the preference setting, with information ratio based analysis [1908.01289].
- **Dueling RL**: Optimism-based exploration with trajectory preference models, tight finite-time regret bounds [2111.04850].

These architectures—along with associated benchmarks—represent the current state of the art for leveraging preference and rating-based human feedback in reinforcement learning, offering practical and theoretically grounded frameworks for aligning RL agents with human intent.

Source: https://www.emergentmind.com/topics/preference-based-and-rating-based-rl