Preference-Aware Autoregressive Reward Modeling

• The paper introduces a framework that addresses ambiguous supervision in RLHF by integrating context-conditioned and multi-objective reward modeling.
• It details innovative architectures like CARM, PARM, and PaTaRM that enable dynamic, interpretable, and personalized reward adaptation in language models.
• Empirical evaluations demonstrate significant improvements in context-specific accuracy and RLHF performance over traditional binary preference models.

Preference-aware autoregressive reward modeling encompasses a set of techniques for training and deploying reward models in LLMs that flexibly accommodate user or annotator preferences—ranging from fine-grained context conditioning to continuous, multi-objective, and self-supervised alignment signals. These approaches directly address the challenges of ambiguous, multi-dimensional, and noisy preference data, moving beyond the limitations of classical binary pairwise supervision to support pluralism, interpretability, and real-time personalization within autoregressive language modeling paradigms.

1. Motivation and Conceptual Frameworks

Preference-aware autoregressive reward modeling arises from the limitations of under-specified or inconsistent supervision in standard reward modeling, particularly in RLHF (Reinforcement Learning from Human Feedback) LLM training. Existing binary preference-based schemes struggle with intent ambiguity, context-misalignment, and multidimensional criteria, often leading to low inter-annotator agreement and compromised alignment fidelity. Preference-aware formulations partition the reward modeling problem into context-inference and context-specific scoring (as in CARM (Pitis et al., 20 Jul 2024)), multi-objective parametrization (as in PARM (Lin et al., 6 May 2025)), or enable individualized reward adaptation (e.g., ARF-RLHF (Zhang, 3 Jul 2025)) and self-contained rationalizing reward generations (PaTaRM (Jian et al., 28 Oct 2025)).

Central themes include:

• Contextualization: Resolving ambiguity by conditioning preferences and reward signals on explicit (or inferred) contexts or user profiles.
• Multi-objective alignment: Allowing trade-offs among multiple reward axes via preference vectors or adapters.
• Preference-to-pointwise bridging: Transforming pairwise preference feedback into meaningful, continuous, and pointwise reward signals.
• Personalization and adaptation: Enabling user and task-specific alignment during or after model deployment.

2. Formal Models and Theoretical Foundations

Several formulations unify preference-aware modeling, with a common foundation in intent–utility and (generalized) Bradley–Terry models. The CARM approach (Pitis et al., 20 Jul 2024) leverages a two-step procedure by first selecting a context $z$ that coarsely partitions intent $I$ and then evaluating $y$ under $z$:

• Marginal utility: $u(x, y) = \sum_{z \in Z} p(z|x) u((x,z), y)$;
• Context-conditioned scoring: learn $$\widehat{u}(x, y) = \sum_{z \in Z} \widehat{p}(z|x) \widehat{u}((x,z), y)$$;
• Error decomposition: Decomposes margin errors into context-weighted prediction and inference residuals.

Reward-aware preference optimization (RPO) (Sun et al., 31 Jan 2025) provides a mathematical framework that aligns implicit (policy-derived) reward margins against explicit (learned or ground truth) reward models, generalizing objectives such as DPO, IPO, SimPO, and RLOO. The general RPO loss aligns the margin of an implicit reward model $r_\pi$ with that of a target $R$, using a margin-based distance $𝔻$:

$$\mathcal{L}_{\rm RPO}^{\mathcal{D}}(\theta, \phi; x, y^1, y^2) = 𝔻[ \Delta r_\pi(x) \Vert \eta \Delta R(x)]$$

where $\Delta r_\pi(x) = r_\pi(x, y^1) - r_\pi(x, y^2)$, $\Delta R(x)$ is the explicit RM margin.

Preference optimization via contrastive divergence (MC-PO) (Chen et al., 6 Feb 2025) reframes reward learning as maximum-likelihood estimation on an unnormalized model, using hard negatives sampled via contrastive divergence to approximate the partition function:

$$\mathbf{P}_\theta(y|x) \propto \exp(r_\theta(x, y))$$

3. Model Architectures and Conditioning Mechanisms

Autoregressive reward models are enhanced with several preference-aware mechanisms:

• Context-Aware Reward Models (CARM): Implemented as context-prepended autoregressive models (e.g., Mistral-7B-RM), fine-tuned via LoRA adapters with context-specific logistic loss (Pitis et al., 20 Jul 2024). Explicit context concatenation enables the RM to resolve preference reversals conditioned on context.
• Dynamic and Bilinear Adapters (PARM/PBLoRA): Instead of independently trained ARMs per criterion as in GenARM, PARM (Lin et al., 6 May 2025) introduces a single ARM with low-rank adapters bilinearly modulated by a user-specified preference vector $p \in \Delta^{k-1}$ for $k$ objectives. PBLoRA achieves expressivity proportional to the square of adapter rank and provides for continuous, fine-grained objective trade-offs at inference.
• Autoregressive Generative Reward Rollouts (PaTaRM): Rather than scalar discrimination, PaTaRM (Jian et al., 28 Oct 2025) generates natural-language critiques under dynamically adapted rubrics, aggregates subscores, and implements both pairwise and pointwise supervision by parsing generated evaluations.
• Interaction Distillation for Robustness: To counteract attention hacking, interaction distillation (Zang et al., 4 Aug 2025) trains decoder-only RMs to mimic intra/inter-sequence token interactions of a strong NLU (encoder-only) teacher, improving the stability and generalization of reward signals.

4. Training Paradigms, Loss Functions, and Sample Efficiency

Training schemes in preference-aware autoregressive RMs incorporate both classical and novel objectives:

• Pairwise logistic (Bradley–Terry) loss: Dominant in classical RMs; remains central in context-conditioned preference modeling and multi-objective ARM training (Pitis et al., 20 Jul 2024, Lin et al., 6 May 2025).
• Contrastive divergence (CD) loss: MC-PO (Chen et al., 6 Feb 2025) uses k-step MCMC to sample hard negatives approximating the normalization constant in the NLL loss—proven to yield empirically stronger gradient signals.
• Pointwise and margin-based loss bridging: PaTaRM (Jian et al., 28 Oct 2025) operationalizes pointwise scoring by aggregating multiple generative judgment rollouts, producing reinforcement signals based entirely on pairwise preference data and rubric-based scoring.
• Actor-Critic and Trace-Biased (TB) RL: ARF-RLHF (Zhang, 3 Jul 2025) replaces coarse binary feedback with real-valued sentiment analysis-derived scores, feeding them into an actor-critic-like RL loop. The trace-biased loss is directly compatible with (unclipped) PPO and DPO.

In multi-objective scenarios (e.g. PARM (Lin et al., 6 May 2025)), objective sampling over the preference simplex during training encourages the RM to approximate the entire Pareto frontier.

5. Dataset Construction and Evaluation Benchmarks

Synthetic and annotated datasets, especially those sensitized to context and user profile, are essential for calibrating and benchmarking preference-aware RMs:

• “Reasonable Preference Reversal” (RPR) datasets (Pitis et al., 20 Jul 2024): Curated with pairs of criteria or detailed scenario descriptions. With proper context, preferences flip deterministically, providing a rigorous test of context-sensitivity.
• Instance-specific rubrics (Jian et al., 28 Oct 2025): Dynamic generation of evaluation criteria enables testing fine-grained model judgment.
• Multi-objective datasets (Lin et al., 6 May 2025): Datasets spanning two or more criteria for benchmarking Pareto-optimal alignment.
• Personalization protocols (Zhang, 3 Jul 2025): Continuous tracking of real user sentiment and adaptation in real time.

Models are evaluated via contextually specific agreement, Pareto metrics (hypervolume, mean inner product), and RLHF downstream win rates compared to strong baselines (LLMs such as GPT-4, Llama3-70B, and Mistral-Large-Instruct).

6. Empirical Findings and Comparative Analyses

Empirical results in recent literature consistently demonstrate that preference-aware autoregressive RMs outperform traditional baselines in contexts requiring nuanced, context-, or profile-sensitive alignment:

Model/Method	Context-Specific Acc.	Pareto HV/MIP	RLHF ∆ vs. Baselines
Mistral CARM (Pitis et al., 20 Jul 2024)	∼0.98 RPR	–	Matches/outperforms Llama3-70B, GPT-4
PaTaRM (Jian et al., 28 Oct 2025)	+4.7% RewardBench	–	+13.6% downstream RLHF
PARM (PBLoRA) (Lin et al., 6 May 2025)	–	+14.1% HV	–
MC-PO (Chen et al., 6 Feb 2025)	+4–9 pp win rate	–	–
ARF-RLHF (TB) (Zhang, 3 Jul 2025)	–	–	+3.3% PPO, +7.6% DPO

Explicit context injection, profile-sensitivity, and multi-objective alignment lead to characterized, substantial improvements—especially under contextually mismatched, adversarial, or subjectively ambiguous conditions.

7. Interpretability, Personalization, and Future Directions

Preference-aware autoregressive reward modeling frameworks increasingly prioritize interpretability, auditability, and personalization:

• Interpretability: Generative rollout-based RMs (PaTaRM) and context-prepended models (CARM) output self-contained rationales or context-traceable justifications.
• User Adaptation: Dynamic preference vectors (PARM) and adapter-based trackers (ARF) enable real-time, low-latency adaptation to user tastes.
• Scalability and efficiency: Bilinear adapters (PBLoRA) and distilled interaction losses decouple model capacity from the number of objectives or context dimensions, promoting low-parameter, computation-efficient alignment.
• Research challenges: Key open questions include robust context inference, intrinsic reward hacking, extrapolation to unseen preferences, scaling to high-dimensional control, human-inthe-loop joint context-preference annotation, and extension to multimodal or non-text settings.

Preference-aware autoregressive RMs are rapidly evolving to close the gap between rigid, context-agnostic alignment and the demands of pluralistic, personalized, and interpretable LLM deployment (Pitis et al., 20 Jul 2024, Jian et al., 28 Oct 2025, Lin et al., 6 May 2025, Chen et al., 6 Feb 2025, Sun et al., 31 Jan 2025, Zang et al., 4 Aug 2025, Zhang, 3 Jul 2025).