Preference-Based Reward Repair (PBRR)

• Preference-Based Reward Repair (PBRR) is a framework that repairs and refines a prior reward function with incremental human preference queries to align RL agents with true human objectives.
• It improves sample efficiency and training stability by using parameterized corrections and robust loss functions to mitigate reward misspecification and hacking.
• PBRR integrates various algorithms—from residual reward models to targeted exploration—to combine hand-crafted proxies with human feedback for effective policy alignment.

Preference-Based Reward Repair (PBRR) is a framework for aligning reinforcement learning (RL) agents with human desiderata by repairing, rather than discarding or ignoring, available reward information. In PBRR, the agent leverages a prior reward function—often a human-designed proxy, a reward model learned from demonstrations, or a heuristic—and incrementally corrects it using a small number of human preference queries. The repair is performed through parameterized corrections that are directly optimized to align induced trajectory rankings with observed human (or synthetic oracle) preferences. Compared to vanilla preference-based RL, PBRR drastically improves sample efficiency and training stability, and it is robust to reward misspecification and reward hacking. A range of PBRR instantiations are supported by recent literature, notably residual reward models (Cao et al., 1 Jul 2025), targeted proxy correction with trajectory-level exploration and loss shaping (Hatgis-Kessell et al., 14 Oct 2025), credit-assignment schemes (Verma et al., 2024), robust and invariant reward correction for shortcut mitigation in LLM alignment (Ye et al., 21 Oct 2025), and more.

1. Foundations and Problem Setting

PBRR is motivated by the problem of learning capable and aligned policies in settings where a hand-crafted scalar reward function is difficult to specify or brittle under optimization. Standard RL paradigms rely on scalar rewards $r(s,a)$, but crafting such functions often yields undetected loopholes, leading to reward hacking—a phenomenon where optimized policies exploit misspecifications in $r$ to achieve high reward without meeting true objectives.

Preference-based RL (PbRL or RLHF) circumvents the need for explicit rewards by learning a reward model $\hat r$ from human preferences over trajectory pairs. However, tabula rasa learning in PbRL is data-intensive and slow to converge. To address this, PBRR incorporates any available prior reward signal $r^0$—such as a human "best guess," an IRL-recovered reward, or even an intentionally misspecified or adversarial proxy. PBRR’s objective is to learn an additive correction $g(s,a)$ so that the sum $r'(s,a) = r^0(s,a) + g(s,a)$ induces policies whose behavior better aligns with human intent as expressed through preferences (Cao et al., 1 Jul 2025, Hatgis-Kessell et al., 14 Oct 2025).

2. Mathematical Formulation and Core Losses

The canonical PBRR model assumes the true reward can be decomposed as

$r^*(s,a) = r^0(s,a) + g(s,a)$

where $r^0$ is fixed and $g$ is learned from data. Human feedback is gathered as a dataset $\mathcal D$ of trajectory (or segment) pairs, with binary labels indicating preferences.

The reward model $r$0 is trained to minimize preference loss, typically instantiated as the Bradley–Terry or Thurstone model:

$r$1

with $r$2.

Correction $r$3 is further regularized for stability and to prevent over-correction on transitions where $r$4 is already consistent with preferences (Hatgis-Kessell et al., 14 Oct 2025):

$r$5

where $r$6/$r$7 partition $r$8 by agreement with $r$9.

A diversity of loss extensions exists: for credit assignment (Verma et al., 2024), auxiliary objectives redistribute trajectory-level margins to salient states using world-model attention, and for shortcut mitigation (Ye et al., 21 Oct 2025), explicit regularization enforces invariance to specified spurious feature directions in linguistic tasks.

3. Representative Algorithms and Implementation Strategies

PBRR encompasses a broad algorithmic family, unified by incremental correction of a prior via sampled preferences. Major instantiations include:

• Residual Reward Models (RRM): The correction is parameterized by a neural network $\hat r$0 (typically an ensemble of MLPs), with the full model $\hat r$1. The residual is initialized to zero and trained via gradient descent with cross-entropy preference loss. Periodically, the entire transition buffer is re-labeled with updated rewards (Cao et al., 1 Jul 2025).
• Targeted Correction with Trajectory-Level Exploration: Rather than passively collecting preferences, PBRR policies $\hat r$2 induced by repaired rewards are compared directly against trajectories generated by a reference policy $\hat r$3, forming a curriculum targeted at surfacing proxy failings. Human or synthetic preferences are queried only for those trajectory pairs most likely to expose misalignment (Hatgis-Kessell et al., 14 Oct 2025).
• Adversarial Virtual Preferences Correction: When available data is offline and the agent’s own state distribution differs, "virtual preferences" force the reward model to favor offline data over agent rollouts, thus realigning the reward as the agent explores new behavior (Zhang et al., 2024).
• Interpretable PBRR via Trees: Reward corrections are parameterized as tree-structured, piecewise constant functions by greedy splitting and pruning, enabling direct human interpretability of the resulting reward components (Bewley et al., 2021).
• PRISM for Shortcut Mitigation: In LLM alignment, PBRR leverages explicit group-invariant kernels constructed from known spurious features (e.g., verbosity, tone, sycophancy), enforcing decorrelation of the reward from these features during training (Ye et al., 21 Oct 2025).

4. Empirical Results and Sample Efficiency

Empirical evidence consistently supports the efficiency and robustness of PBRR approaches. Exemplary findings include:

Method	Domain/Task Suite	Final Perf. (Success/Return)	Preference Count	Notes
RRM (full proxy) (Cao et al., 1 Jul 2025)	Meta-World manipulation	77.8% ± 6.7 (IQM success)	~300–400	>10% gain over PEBBLE baseline
PBRR (Hatgis-Kessell et al., 14 Oct 2025)	Reward hacking benchmarks	Near-optimal recovery	Tens–few hundred	Outperforms RLHF-from-scratch on all domains
Hindsight PRIOR (Verma et al., 2024)	MetaWorld/DMC	+20% (MetaWorld), +15% (DMC)	—	4× query reduction vs. baselines
Tree PBRR (Bewley et al., 2021)	Classic/sim environments	95–100% pilot performance	≈600 labels (1 hr)	Yields interpretable reward components
PRISM (Ye et al., 21 Oct 2025)	LLM reward learning	+1–2% o.o.d. accuracy	—	Shortcut margin correlations nearly eliminated

PBRR methods demonstrate robustness to moderate feedback noise, superior sample efficiency compared to end-to-end RLHF, and strong empirical alignment even under adversarial proxies (e.g., RRM with negated proxy still recovers high reward).

5. Theoretical Properties and Guarantees

Theoretical analysis establishes that, under standard MDP and Bradley–Terry assumptions, PBRR with tabular or linear reward corrections achieves sublinear cumulative regret matching prior PbRL and RLHF-from-scratch minimums (up to problem size constants) (Hatgis-Kessell et al., 14 Oct 2025). Specifically:

• In tabular and known dynamics settings,

$\hat r$4

where $\hat r$5 is feature dimension and $\hat r$6 the number of iterations.

• In noiseless preference settings, terminal policies induced by repaired rewards guarantee non-inferiority to the reference policy.
• Invariance-based PBRR (PRISM) offers theoretical risk bounds for the group-invariant kernel approximation (Theorem 2 in (Ye et al., 21 Oct 2025)), ensuring o.o.d. generalization and decorrelation from specified shortcut groups.
• No explicit sample complexity bounds exist for tree-structured PBRR, but empirical evidence demonstrates rapid convergence with modest feedback budgets (Bewley et al., 2021).

6. Limitations, Open Questions, and Practical Guidance

The benefits of PBRR are conditioned on the availability and informativeness of the prior reward. If the initial proxy is severely pessimistic or uninformative, more feedback is necessary to recover optimality. All current approaches require a reliable mechanism for preference elicitation and, in some variants, a safe reference policy for targeted exploration (Hatgis-Kessell et al., 14 Oct 2025).

Principal limitations and open challenges include:

• Scalability of explicit exploration or policy set construction in large/continuous domains.
• Difficulty of credit assignment for trajectory-level preferences (segment-wise or statewise queries may provide finer corrections).
• Automatic prior construction, ideally via meta-learning or task abstraction transfer, remains an open issue.
• For PRISM and similar approaches, availability and definition of shortcut detectors is a prerequisite, constraining domains where invariance-based repair can be deployed (Ye et al., 21 Oct 2025).

Among the practical considerations, decay schedules for correction regularization, tuning of preference sampling rates, and selection of architectural hyperparameters remain topics of interest for practitioners.

7. Extensions and Connections to Adjacent Domains

PBRR is a strictly modular paradigm, readily extensible to a variety of settings, including:

• Sim2Real transfer via RRM and its visual variants (Cao et al., 1 Jul 2025).
• Data-efficient LLM preference alignment under shortcut- and robustness-aware reward learning (Ye et al., 21 Oct 2025).
• Repair and adaptation of offline preference-based reward models to novel behaviors via adversarial (virtual) comparisons (Zhang et al., 2024).
• Integration with distributionally robust optimization, multi-modal or multilingual reward learning, and meta-alignment techniques as outlined in future work of PRISM (Ye et al., 21 Oct 2025).

Through its explicit leverage of prior information and targeted, preference-driven updates, PBRR provides a principled foundation for efficient, robust, and scalable human-aligned RL in real-world systems.