PaTaRM: Preference-Aware Task-Adaptive Reward Model

• The paper introduces PaTaRM, which converts pairwise preference data into rollout-level rewards using a generative reward model for rubric-based evaluation.
• It employs a Preference-Aware Reward mechanism combined with task-adaptive rubrics to bridge the gap between pairwise supervision and pointwise inference.
• Empirical results demonstrate up to 4.7% improvement on reward benchmarks and a 13.6% gain in downstream RLHF performance.

Preference-Aware Task-Adaptive Reward Model (PaTaRM) is a generative reward-modeling framework for reinforcement learning from human feedback (RLHF) that is designed to bridge pairwise preference supervision and pointwise reward inference. In PaTaRM, a generative reward model is trained on pairwise preference data but deployed as a pointwise evaluator that produces rubric-grounded critiques and scalar scores for individual responses. Its core design combines a Preference-Aware Reward (PAR) mechanism, which converts relative pairwise preferences into rollout-level rewards for generative judgment traces, with a task-adaptive rubric system that mixes task-level primary criteria and instance-specific rubric generation. The framework is reported to improve RewardBench and RMBench performance by an average relative 4.7% across Qwen3-8B and Qwen3-14B, and to improve downstream RLHF performance on IFEval and InFoBench by an average 13.6% (Jian et al., 28 Oct 2025).

1. Definition and conceptual position

PaTaRM addresses a structural mismatch in reward modeling for RLHF. Standard pairwise reward models are trained on tuples of the form “chosen versus rejected” and are straightforward to optimize with Bradley–Terry-style objectives, but RLHF deployment typically requires pointwise rewards for single responses. Conversely, pointwise generative reward models can produce structured critiques and absolute scores, yet they usually depend on rubric-driven absolute labels that are expensive to obtain and often lack adaptability across tasks (Jian et al., 28 Oct 2025).

Within this framing, PaTaRM is defined by two simultaneous commitments. First, it treats pairwise preference data as the primary supervision source, thereby avoiding explicit pointwise labels. Second, it trains a generative reward model to behave as a pointwise evaluator at inference time, producing rubric-based explanations and a final numeric score for one response at a time. This suggests that PaTaRM is not merely a new loss function, but a unification of pairwise supervision, pointwise inference, and rubric-conditioned reasoning in a single reward-modeling pipeline (Jian et al., 28 Oct 2025).

The framework belongs to a broader lineage of reward learning from preference data. Earlier work in preference-based RL typically used binary comparisons between trajectory segments and Bradley–Terry likelihoods to recover a scalar reward model (Katz et al., 2021). Subsequent work explored richer reward representations, including dynamics-aware rewards for sample-efficient preference-based reinforcement learning (Metcalf et al., 2024), low-rank style adaptation of pre-trained reward functions under low-preference-data constraints (Marta et al., 14 Apr 2025), and generalized acquisition functions that target behavioral equivalence rather than exact reward-parameter identification (Ellis et al., 2024). PaTaRM differs from these approaches by centering the RLHF-specific problem of pairwise-to-pointwise reward transfer for generative evaluators (Jian et al., 28 Oct 2025).

2. Core architecture: PAR and task-adaptive rubrics

The first defining component of PaTaRM is the Preference-Aware Reward mechanism. For a prompt $x$ with a chosen response $y^c$ and a rejected response $y^r$, the generative reward model produces multiple judgment rollouts for each response. Each rollout contains a critique and a final score, from which numeric values $s_i^c$ and $s_j^r$ are extracted. The model then computes average scores

$$\bar{s}^c = \frac{1}{n}\sum_{i=1}^n s_i^c,\qquad \bar{s}^r = \frac{1}{n}\sum_{j=1}^n s_j^r.$$

PaTaRM enforces the desired inequality $\bar{s}^c > \bar{s}^r$ by assigning rollout-level rewards relative to the opposite group’s mean score (Jian et al., 28 Oct 2025).

For chosen-response rollouts, the margin is defined as $\delta_i^c = |s_i^c - \bar{s}^r|$; for rejected-response rollouts, $\delta_j^r = |s_j^r - \bar{s}^c|$. The PAR rewards are

$$R_{\text{PAR}}(y_i^c)=\mathbb{I}[s_i^c > \bar{s}^r]\cdot f(\delta_i^c),$$

$y^c$0

where $y^c$1 maps the score margin to a reward magnitude (Jian et al., 28 Oct 2025). Two instantiations are described. A graded version uses

$y^c$2

while a constant version uses $y^c$3 whenever $y^c$4. The graded form is reported to preserve discriminative capacity more effectively during RL training (Jian et al., 28 Oct 2025).

The second defining component is the task-adaptive rubric system. PaTaRM combines global task-level primary rubrics with instance-specific rubric generation. Primary rubrics are domain dependent: chat emphasizes usefulness; math emphasizes correctness and logic; code emphasizes correctness and algorithmic reasoning; safety emphasizes harm avoidance; instruction following emphasizes coverage and constraint adherence. Dynamic rubric generation adds one to three criteria tailored to the concrete prompt and response. This produces a combined rubric

$y^c$5

which structures both the generated critique and the final score (Jian et al., 28 Oct 2025).

A format reward is added to maintain structural validity of the output:

$y^c$6

The total rollout reward is then

$y^c$7

This is the signal used to optimize the generative reward model (Jian et al., 28 Oct 2025).

3. Training procedure and inference behavior

PaTaRM training proceeds in two stages. In the supervised fine-tuning stage, the generative reward model is initialized to produce rubric-based critiques and scores in a constrained output format such as <answer\>5</answer>. The supervision is pseudo-labeled by larger Qwen2.5 models, and the training objective is standard token-level language-modeling loss over the generated judgment text (Jian et al., 28 Oct 2025).

In the reinforcement-learning stage, the model is optimized with PAR-derived rollout rewards. For each pair $y^c$8, the model samples multiple judgment rollouts for the chosen and rejected responses, parses the scores, computes the PAR rewards and format penalties, and applies a PPO-style RL objective through GRPO, Reinforce++, or DAPO. The paper presents the conceptual objective as maximizing the expected sum of rollout rewards over chosen and rejected judgment traces, with GRPO supplying group-relative advantages for stabilization (Jian et al., 28 Oct 2025).

This training design is intended to close the pairwise–pointwise gap. Training still relies on pairwise preference tuples, but the model is always optimized in pointwise judgment mode: given one response, it must generate a critique and scalar score. A plausible implication is that PaTaRM transfers pairwise supervision into a pointwise scoring scale by repeatedly enforcing relative consistency across groups of generated evaluations rather than by fitting a separate scalar head to absolute labels.

At inference time, the trained model behaves as a pointwise reward evaluator. Given a prompt $y^c$9 and candidate response $y^r$0, it constructs a judge prompt with primary and, when used, dynamically generated rubrics; produces one or more judgment rollouts; extracts the scalar scores; and aggregates them, often by averaging:

$y^r$1

This score can be used directly as the reward in RLHF training or as a ranking signal in evaluation (Jian et al., 28 Oct 2025).

The article’s examples emphasize that PaTaRM’s outputs are not restricted to scalar values. It also emits rubric-conditioned reasoning, making it more interpretable than scalar reward models. In a qualitative example about cleaning a showerhead, the model generates criteria such as usefulness, simplicity and accessibility, and safety and precautions, then explains the response along these axes before producing an aggregate score (Jian et al., 28 Oct 2025).

4. Empirical results and downstream RLHF effects

PaTaRM is evaluated as a reward model on RewardBench and RMBench and as an RLHF component on IFEval and InFoBench. On RewardBench and RMBench, PaTaRM improves over the corresponding Qwen3 baselines for both 8B and 14B backbones. For Qwen3-8B, RewardBench overall increases from 78.1 to 82.5 and RMBench overall from 71.0 to 73.6. For Qwen3-14B, RewardBench overall increases from 81.9 to 86.2 and RMBench overall from 73.2 to 76.2. Averaged across Qwen3-8B and Qwen3-14B, the relative improvement over RewardBench and RMBench is reported as 4.7% (Jian et al., 28 Oct 2025).

The framework also improves downstream RLHF. On policy models including Qwen2.5-7B-Base and Qwen3-14B, GRPO using PaTaRM as the reward model outperforms SFT, DPO, and GRPO using Skywork-LLaMA-3.1-8B as the reward model. The paper reports an average improvement of 13.6% across IFEval and InFoBench benchmarks (Jian et al., 28 Oct 2025). On Qwen2.5-7B-Base, PaTaRM-based RL improves IFEval average from 40.1 to 49.2 and InFoBench from 66.7 to 84.3. On Qwen3-14B, PaTaRM-based RL improves IFEval average from 89.0 to 90.9 and InFoBench from 86.7 to 89.2 (Jian et al., 28 Oct 2025).

The framework is also evaluated in pairwise mode. On pairwise RewardBench, PaTaRM Qwen3-8B reaches 87.9 and PaTaRM Qwen3-14B reaches 88.6, matching or slightly surpassing strong pairwise generative reward-model baselines when dynamic rubric adaptation is included (Jian et al., 28 Oct 2025). This is important because it indicates that the rubric mechanism is not only useful for pointwise evaluation; it also remains beneficial in settings closer to classical pairwise judging.

Ablation results isolate two components. First, using task-adaptive rubrics outperforms using only primary rubrics or only generated rubrics. In pointwise settings, primary-only rubrics are reported to suffer from entropy collapse and reward-signal collapse, whereas the combination of primary and generated rubrics is the most robust configuration (Jian et al., 28 Oct 2025). Second, the graded margin function in PAR outperforms the constant margin version, and score margins between chosen and rejected responses are observed to shrink during RL training, especially for smaller models. The graded reward slows this shrinkage and maintains better discriminability (Jian et al., 28 Oct 2025).

5. Relation to adjacent reward-modeling research

PaTaRM sits at an intersection of several active lines of work on preference learning and reward-model adaptation. One adjacent direction improves sample efficiency or robustness in preference-based RL by changing the structure of the reward representation. For example, dynamics-aware reward functions based on self-supervised temporal consistency improve preference-based RL sample efficiency by an order of magnitude in locomotion domains and recover substantially more of ground-truth reward policy performance than existing approaches (Metcalf et al., 2024). FLoRA addresses low-preference-data adaptation by freezing a pre-trained reward model and learning low-rank reward adapters, mitigating catastrophic reward forgetting during robotic style adaptation (Marta et al., 14 Apr 2025). These approaches focus on control settings and scalar rewards rather than generative rubric-based RLHF, but they share PaTaRM’s concern with preserving useful reward structure under limited preference data.

A second adjacent direction concerns task or preference conditioning of reward models. PARM conditions an autoregressive reward model on a user preference vector for multi-objective test-time alignment, using Preference-Aware Bilinear Low-Rank Adaptation to control objective trade-offs during inference (Lin et al., 6 May 2025). In-context Reward Adaptation instead uses a transformer to infer a new reward function from a small set of preference demonstrations, and shows that adding response time as an auxiliary signal can remove asymptotic bias for unseen preference domains (Sun et al., 28 May 2026). These systems are task-adaptive in different senses than PaTaRM: PARM is conditioned on explicit objective vectors, while in-context adaptation conditions on few-shot demonstrations. PaTaRM conditions through rubric generation and preference-aware rollout rewards (Jian et al., 28 Oct 2025).

A third line focuses on robustness under distribution shift or reward-model co-adaptation. ReCouPLe augments preference labels with natural-language rationales and treats the rationale as a projection axis in a shared trajectory–language embedding space, yielding large gains under distribution shifts and novel-task transfer (Hwang et al., 5 Mar 2026). Mutual-Taught iteratively co-adapts policy and reward models in an EM-like loop, using policy-comparison pseudo-labels so that the reward model tracks the evolving policy distribution (Shi et al., 17 May 2025). APLOT introduces adaptive margins derived from optimal transport to focus BT-based reward models on semantically similar and weakly separated hard samples, improving both ID and OOD generalization (Li et al., 13 Oct 2025). PaTaRM differs from all three by targeting the pairwise-to-pointwise mismatch in generative reward models and by centering dynamic rubric generation (Jian et al., 28 Oct 2025).

These comparisons suggest that PaTaRM is best understood as one member of a broader movement away from static scalar reward heads toward richer, adaptive, and more structured preference models. Its specific contribution is the claim that robust pointwise generative reward modeling can be learned from pairwise data alone when relative preferences are converted into rollout-level rewards and interpreted through task-adaptive rubrics (Jian et al., 28 Oct 2025).

6. Interpretation, limitations, and significance

PaTaRM’s most visible interpretability advantage over scalar reward models is that it produces a textual rationale alongside the score. This makes it possible to inspect which criteria were used and how the final judgment was formed. The paper explicitly contrasts this with scalar reward models that output only a number and are more difficult to audit (Jian et al., 28 Oct 2025). In that sense, PaTaRM belongs to the broader class of generative reward models that treat evaluation as structured reasoning rather than scalar regression.

At the same time, the framework has limitations. It still depends on curated pairwise preference data and on pseudo-labeled pointwise SFT data produced by larger teacher models. Dynamic rubric generation can introduce noise when the model proposes irrelevant or inconsistent criteria. Generative evaluation is also more computationally expensive than scalar reward inference because the judge prompt is longer and voting over multiple rollouts is often beneficial. The paper further notes that score-margin collapse can occur during extended RL training, especially for smaller models, reducing discriminative power and motivating better margin-preserving objectives (Jian et al., 28 Oct 2025).

From an encyclopedic perspective, a common misconception would be to treat PaTaRM as a standard pairwise reward model with an explanatory prompt wrapper. That description is incomplete. The framework’s defining claim is stronger: pairwise data are used to optimize a pointwise generative evaluator through rollout-level rewards, so the pointwise score is not appended after training but emerges from the training objective itself (Jian et al., 28 Oct 2025). Another misconception would be to equate task adaptation with merely swapping static rubrics across domains. PaTaRM’s task adaptivity is dynamic and instance-conditioned, because it combines global rubrics with generated criteria tailored to the specific prompt and response (Jian et al., 28 Oct 2025).

The broader significance of PaTaRM is that it reframes reward modeling for RLHF as a problem of converting comparative human supervision into interpretable, task-conditioned pointwise judgments. This suggests a research direction in which reward models are expected not only to rank outputs, but also to expose the evaluative criteria that justify those rankings. Within that direction, PaTaRM provides a concrete synthesis of preference-aware optimization and task-adaptive rubric construction, with reported gains both on reward-model benchmarks and in downstream RLHF (Jian et al., 28 Oct 2025).