Reward-Pair Classification in RLHF

• Reward-pair classification is a method that trains scalar reward models from paired human feedback, making it essential for aligning LLMs to human preferences.
• It leverages formulations like the Bradley-Terry model to compare candidate responses, using pairwise accuracy as a key metric for reliable predictions.
• Recent advances incorporate hybrid generative models, active learning, and multi-head classification to enhance alignment efficiency and reduce bias.

Reward-pair classification is a central problem in aligning LLMs through reinforcement learning from human feedback (RLHF). This task formalizes the training of a reward model to accurately predict human preferences between pairs of candidate responses, directly impacting the reliability and safety of RLHF-trained systems. Reward-pair classification is foundational for both model evaluation and the optimization of aligned policies, and has catalyzed a rich landscape of methodologies ranging from classic probabilistic models to dynamic rubric-based generative methods.

1. Formalization and Evaluation Protocols

Reward-pair classification is defined by a dataset of $N$ prompt-response trios, $S = \{(x_i, y_i^+, y_i^-)\}_{i=1}^N$, where $x_i$ is a user prompt, $y_i^+$ the human-preferred (chosen) response, and $y_i^-$ the rejected response. The goal is to train a scalar reward model $r_\theta: (x, y) \mapsto \mathbb{R}$ such that higher $r_\theta(x, y)$ indicates better alignment with human preferences. At evaluation, for each comparison, the model predicts $y_1 \succ y_2$ if $r_\theta(x, y_1) > r_\theta(x, y_2)$. The primary metric is pairwise accuracy: the fraction of test pairs for which $r_\theta(x, y^+) > r_\theta(x, y^-)$ (Lambert et al., 2024).

Benchmarking frameworks such as RewardBench curate diverse datasets across capabilities, safety, refusal, and code categories, with careful manual annotation to ensure clear preferred/rejected labels. The dataset design omits ambiguous or jointly incorrect pairs to focus on verifiable preference signals (Lambert et al., 2024). Evaluation extends beyond aggregate pairwise accuracy to include category-specific breakouts, refusal consistency, out-of-distribution (OOD) generalization, and calibration analyses (e.g., score distributions, prompt-length bias).

2. Core Modeling Paradigms

The dominant paradigm for reward-pair classification is the Bradley–Terry (BT) or Luce–Shephard model:

$S = \{(x_i, y_i^+, y_i^-)\}_{i=1}^N$0

with a cross-entropy objective for binary (and more generally, soft) preference labels (Sun et al., 2024, Lambert et al., 2024). The BT model provides a theoretically sound basis, with order-consistency: monotonic transformations of $S = \{(x_i, y_i^+, y_i^-)\}_{i=1}^N$1 preserve ranking. Recent theoretical results establish rates for BT regression with neural embeddings and clarify that high marginal classification accuracy (through e.g. binary cross-entropy on individual outputs) suffices for order consistency and downstream ranking (Sun et al., 2024).

Alternative formulations unify generative reward modeling (mask-filling, e.g., "yes"/"no") with pairwise discriminative objectives. For instance, a masked LM can be trained to fill a slot indicating if "$S = \{(x_i, y_i^+, y_i^-)\}_{i=1}^N$2 is better than $S = \{(x_i, y_i^+, y_i^-)\}_{i=1}^N$3," and the logit difference defines a pairwise score analogous to BT (Xu et al., 7 Apr 2025).

Extensions to ordinal feedback generalize BT beyond hard binary labels. Under a marginal unbiasedness condition (mean label matches oracle probability), soft or ordinal $S = \{(x_i, y_i^+, y_i^-)\}_{i=1}^N$4 allow for a cross-entropy or hinge loss over finer-grained label sets, with provable enhancement of statistical generalization (lower Rademacher complexity compared to binary) (Liu et al., 2024). Explicit modeling of ties is handled by the generalized Bradley–Terry–Ties (BTT) model, which introduces a tie-bias parameter $S = \{(x_i, y_i^+, y_i^-)\}_{i=1}^N$5 and a third outcome $S = \{(x_i, y_i^+, y_i^-)\}_{i=1}^N$6, eliminating attenuation bias in preference strength estimation when ties are present (Liu et al., 2024).

3. Contemporary Objective Functions and Training Schemes

Current reward-pair classification models employ several key loss functions and objectives:

• MLE (Binary Cross-Entropy):

$S = \{(x_i, y_i^+, y_i^-)\}_{i=1}^N$7

where $S = \{(x_i, y_i^+, y_i^-)\}_{i=1}^N$8 (Lambert et al., 2024).

• Direct Preference Optimization (DPO):

$S = \{(x_i, y_i^+, y_i^-)\}_{i=1}^N$9

with $x_i$0 as a temperature hyperparameter. DPO fine-tunes the base LM using only pairwise log-prob differences, omitting RL loops (Lambert et al., 2024).

• Classification-based Surrogates: Marginal BCE on individual outputs is order-consistent and, under mild noise, achieves comparable or better ranking than BT, with advantages under annotation noise and data scarcity (Sun et al., 2024).
• Ordinal / Soft-label Extensions: Cross-entropy or hinge losses on ordinal targets $x_i$1, with theoretical guarantees of improved generalization (Liu et al., 2024).
• Generative Reward Models: Preference is formulated as mask-filling ("yes"/"no") classification with positional-symmetry regularization to address left/right bias; scores integrate into RL policy optimization (Xu et al., 7 Apr 2025).
• Multi-head Classification for Multi-objective Alignment: Each preference dimension is captured by an independent head, with outputs mapped via z-score transforms to a scalar reward for PPO or DPO policy optimization (Zhang et al., 24 May 2025).

4. Practical Methodologies and Active Learning

Annotation efficiency and data construction are central concerns in reward-pair classification. The classical D-optimal experimental design is adapted for active selection of the most informative pairs, using Fisher information to balance exploration of feature space and informativeness (diversity and moderate difficulty comparisons). This provides robust learning dynamics and annotation efficiency, particularly when leveraging cross-prompt comparisons to expand the candidate pool. Empirical results demonstrate D-opt outperforms entropy, max-diff, and coreset-based methods in both annotation economy and downstream ranking metrics (Shen et al., 4 Feb 2025).

When ties are present or soft labels are employed, explicit model extensions (BTT) and fine-grained feedback collection further improve bias and generalization, as seen in both synthetic and human-annotated experiments (Liu et al., 2024, Liu et al., 2024).

5. Hybrid and Generative Approaches

Recent work has devised hybrid reward modeling schemes that blend process-based reasoning with outcome prediction. Generative Reward Models (GRMs) generate critiques or reasoning chains for each candidate response; the predicted preference label is then supervised either via comparison to human critiques (process reward) or outcome-only (binary) signals. Weaknesses of outcome-only learning include susceptibility to correct guessing without sound reasoning, causing noise in the RL optimization signal (Wang et al., 12 Jan 2026). The RM-NLHF framework computes process rewards via F1 similarity between model and human critiques, and introduces a Meta Reward Model to bootstrap supervision in data-sparse regimes, leading to measurable gains over outcome-only GRMs (Wang et al., 12 Jan 2026).

Other work leverages explicit two-stage reward modeling, as in Critique-out-Loud (CLoud) models: the LLM produces a free-form critique, which is then evaluated by a shallow reward head. These models report increases of 4.65–5.84 pp in RewardBench pairwise accuracy over standard scalar reward models of the same base scale, with further improvements from self-consistency decoding on short reasoning tasks (Ankner et al., 2024). On-policy critique generation (matching training and inferential distributions) is crucial for optimal performance.

6. Extensions: RL Policy Optimization and Downstream Effects

Reward-pair classification directly interfaces with RLHF policy optimization, typically via PPO or DPO variants. Pairwise rewards can be used within win-probability maximization frameworks, with custom KL penalties and clipped surrogates to ensure stability and focus on "hard" competitive pairs (Xu et al., 7 Apr 2025). Multi-objective and rubric-driven reward models (e.g., PaTaRM) allow transformation of pairwise signals to pointwise rewards for single-instance evaluation and reinforcement learning, facilitating flexible, context-aware, and interpretable objectives (Jian et al., 28 Oct 2025). Multidimensional or multi-head reward models (MOSLIM) enable unified PPO training for diverse human objectives (helpfulness, harmlessness, honesty) with significant reductions in computational overhead versus multi-policy or multi-reward pipelines (Zhang et al., 24 May 2025).

In mathematical or code verification tasks where symbolic or rule-based verifiers provide binary rewards, recasting verifiable rewards as direct categorical labels (e.g., via Rewards as Labels, REAL) has yielded monotonic and bounded gradient allocation, improved stability, and significant gains over standard GRPO/DAPO on Pass@1 benchmarks (Zhai et al., 5 Feb 2026).

7. Open Challenges and Research Directions

Key empirical findings and recommendations from recent large-scale studies (Lambert et al., 2024, Wang et al., 12 Jan 2026, Liu et al., 2024, Zhang et al., 24 May 2025, Jian et al., 28 Oct 2025) include:

• Calibration and score-distribution drift between DPO and classification-trained reward models may impair downstream optimization; temperature and normalization tuning are sometimes required.
• Length and verbosity biases are persistent: reward correlates with token count. Modular benchmarks (RewardBench) and dynamic rubrics (PaTaRM) help detect and mitigate such artifacts.
• Incorporation of soft, ordinal, or tied labels provides statistical and generalization benefits versus coarse binary annotation. Explicit handling of ties (BTT) resolves known preference-strength bias.
• Hybrid process/outcome supervision and interpretability via natural-language chains (RM-NLHF, CLoud) address reward-hacking and error-modes unique to preference learning.
• Efficient active data selection and cross-prompt diversity are critical for annotation and generalization efficiency.
• Multi-task, multi-dimensional reward modeling yields both empirical gains and resource advantages.

Open directions include calibration-aware scoring, automated rubric-generation quality control, more efficient inference/rollout strategies, and integration into multimodal or interactive tasks. Preference-aware reward modeling and dynamic rubric adaptation continue to drive advances, targeting fine-grained, context-sensitive, and robust alignment for LLMs across domains (Lambert et al., 2024, Zhang et al., 24 May 2025, Jian et al., 28 Oct 2025).