PersRM-R1: Personalized Reward Reasoning

• PersRM-R1 is a personalized reward modeling framework that treats stylistic evaluation as an explicit reasoning process, delivering both structured rationales and scalar scores from limited exemplars.
• It employs a two-stage optimization approach combining supervised fine-tuning for format fidelity and reinforcement fine-tuning for enhanced reasoning and generalization.
• The framework constructs synthetic, contrastive data to robustly learn transferable stylistic principles, enabling accurate evaluation across heterogeneous domains.

Searching arXiv for PersRM-R1 and closely related reward-modeling-as-reasoning papers.

PersRM-R1 is a personalized reward modeling framework that treats reward modeling as an explicit reasoning problem rather than as scalar preference estimation alone. Introduced as the first reasoning-based reward modeling framework specifically designed to identify and represent personal factors from only one or a few personal exemplars, it targets a setting in which a reward model must judge whether candidate outputs match an individual’s preferences, style, and communicative habits across domains such as email, news, essays, and blogs. The framework combines exemplar-conditioned synthetic contrastive data construction, reasoning-augmented supervised fine-tuning, and reinforcement fine-tuning to produce a generative reward model that outputs both structured rationales and scalar preference scores (Li et al., 12 Aug 2025).

1. Problem formulation and conceptual basis

Personalized reward modeling in PersRM-R1 is defined as scoring how well a model output aligns with an individual’s preferences, style, and communicative habits rather than with generic alignment criteria such as helpfulness, harmlessness, and honesty. The framework is motivated by two coupled constraints: per-user data is scarce, often limited to one or a few exemplars, and the resulting reward model must generalize across heterogeneous domains and unseen genres. Within this framing, the paper argues that conventional scalar reward models underperform because they do not expose explicit reasoning, while purely supervised personalization methods often overfit static patterns instead of learning transferable stylistic principles (Li et al., 12 Aug 2025).

PersRM-R1 therefore formulates personalized reward modeling as reasoning over text-embedded user traits. The central claim is that personal factors are better captured when the reward model explicitly extracts and decomposes stylistic signals, compares candidate responses against those signals, and produces an interpretable justification before issuing a judgment. The paper situates this approach alongside reasoning-enhanced generative reward models such as RM-R1 and RL-driven reasoners such as DeepSeek-R1, while specializing the paradigm to exemplar-conditioned personalization under extreme data scarcity (Chen et al., 5 May 2025, DeepSeek-AI et al., 22 Jan 2025).

A common misconception is that personalization in reward modeling can be reduced to a generic pairwise preference classifier trained on user-labeled comparisons. PersRM-R1 rejects that reduction. Its stated objective is not only to rank candidates, but to recover the latent criteria by which a user’s style manifests, including lexical choices, tone, narrative structure, and context-dependent rubric weighting. This suggests that the framework treats personal preference not as a fixed scalar target but as a structured comparative judgment process.

2. Generative reward model and reasoning representation

PersRM-R1 is an end-to-end personalized generative reward model that takes as input one or a few personal exemplars $e$, a query $x$, and a pair of candidate responses $(y^+, y^-)$. It outputs a reasoning-augmented evaluation tuple

$\mathcal{V} = (\tau, r^+, r^-),$

where $\tau$ is a structured, step-by-step rationale and $r^+, r^-$ are scalar scores reflecting stylistic compatibility with the exemplars, typically as integers in $[1,10]$. The model is implemented as a causal LLM fine-tuned to emit a standardized three-part format consisting of <criteria>, <eval>, and <scores> (Li et al., 12 Aug 2025).

The <criteria> section contains the evaluation rubrics selected by the model. These may include base rubrics such as Personal Style Adherence, Tone and Voice Consistency, Language Fluency and Coherence, and Relevance to Personal Preferences and Experiences, but the criteria set is not fixed. The model may also introduce sample-specific criteria discovered during training, including the case-study criterion “Argumentation Grounded in Personal Narrative.” The <eval> section provides a comparative analysis of the two candidates, and <scores> contains an integer pair [[x, y]] corresponding to $$(r^+, r^-)`.</p> <p>This output design is central to the framework’s interpretation of reward modeling as reasoning. The model is trained not merely to assign a preference label, but to identify relevant personal factors from one exemplar, weigh criteria dynamically according to context, and ensure that scalar scores remain consistent with the explanatory analysis. At inference time, multiple exemplars can be provided without special adapters or meta-learning modules; the model is described as aggregating stylistic information implicitly through the conditioning context$$e$$(<a href="/papers/2508.14076" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Li et al., 12 Aug 2025</a>).</p> <p>The architecture is instantiated with <a href="https://www.emergentmind.com/topics/qwen2-5-3b-instruct" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Qwen2.5-3B-Instruct</a> and <a href="https://www.emergentmind.com/topics/qwen2-5-7b-instruct" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Qwen2.5-7B-Instruct</a> backbones, producing PersRM-R1-3B and PersRM-R1-7B respectively. <a href="https://www.emergentmind.com/topics/qwen2-5-72b-instruct" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Qwen2.5-72B-Instruct</a> is used during data construction to generate synthetic responses and <a href="https://www.emergentmind.com/topics/reasoning-traces" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">reasoning traces</a>.</p> <h2 class='paper-heading' id='exemplar-conditioned-synthetic-data-construction'>3. Exemplar-conditioned synthetic data construction</h2> <p>Because personalized corpora are small, PersRM-R1 constructs a synthetic contrastive dataset conditioned on a limited user corpus$$D_{\text{expl}}=\{e_i\}$$. The synthetic pipeline has two stages: pairwise preference construction and reasoning trace generation with filtering. The resulting synthetic dataset is intended to span a curriculum from easy negatives to <a href="https://www.emergentmind.com/topics/livecodebench-hard" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">hard</a> confounding negatives, which the paper identifies as important for robustness and generalization (<a href="/papers/2508.14076" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Li et al., 12 Aug 2025</a>).</p> <p>In pairwise preference construction, each training item is a tuple$$(x,e,y^+,y^-)$$. Positive sampling uses intra-author retrieval and lexical perturbation. Intra-author retrieval reuses other writings by the same author to preserve authentic style adherence. Lexical perturbation minimally rewrites exemplar segments through 5–6 synonym substitutions while strictly preserving sentence structure and order. <a href="https://www.emergentmind.com/topics/negative-sampling" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Negative sampling</a> uses cross-author retrieval, random sampling, and confounding sampling. Cross-author retrieval selects writings from different authors with clear style divergence. Random sampling asks an LLM to generate loosely related or off-topic text, producing easy negatives. Confounding sampling asks an LLM to mimic superficial stylistic cues, creating strong adversarial negatives intended to force attention to fine-grained stylistic signals beyond surface imitation.</p> <p>The second stage uses a powerful external LLM, Qwen2.5-72B-Instruct, to generate reasoning tuples$$x$0 for each synthetic pair. The rationale is required to analyze stylistic alignment, tone, phrasing, and semantic intent in comparative, step-by-step form. A faithful reasoning filter then retains only outputs whose scores agree with the construction signal, meaning $x$1 should be more aligned than $x$2, and whose format matches the required schema. This yields the supervised dataset

$x$3

Representative prompt families are provided in the paper’s appendix under “Style Mimicking,” “Minor Replacement,” “Random Style,” and “Reasoning Trace Generation.” As a synthetic separability sanity check, the GPT-4o proxy evaluation reports the following scores for the generated categories: other writings by the target author, 9.41; minor replacement, 9.39; style mimicking, 5.89; writings by other authors, 3.86; style randomization, 2.41. Within the paper, these figures are used to support the claim that the synthetic categories are meaningfully separable for training (Li et al., 12 Aug 2025).

4. Two-stage optimization: supervised fine-tuning and reinforcement fine-tuning

PersRM-R1 is trained as a generative model over the reasoning tuple $x$4 conditioned on $x$5. The first stage, supervised fine-tuning, trains a model denoted PersRM-SFT with a causal next-token objective:

$x$6

This stage teaches the model to imitate faithful, well-formatted rationales and to emit scores consistent with the synthetic construction labels. The paper contrasts this generative objective with the canonical Bradley–Terry scalar reward-model loss and explicitly adopts the causal formulation in order to model reasoning traces (Li et al., 12 Aug 2025).

The second stage is reinforcement fine-tuning, which optimizes the reward model itself through RL rather than using the reward model only as a training signal for a separate policy. Given a sampled output $x$7, a format validator checks whether the output conforms to the required schema. The parsed scores define a sparse rule-based reward:

$x$8

The optimization target is

$x$9

For reinforcement fine-tuning, the framework uses GRPO, described as an efficient PPO variant. The appendix specifies group sampling, Monte Carlo advantage estimation, and the GRPO objective with importance-ratio clipping and KL regularization. The training recipe uses OpenRLHF SFTTrainer for SFT with batch size 64, 1 epoch, Adam learning rate $(y^+, y^-)$0, gradient checkpointing, FlashAttention, and optimizer offloading. RFT uses verl GRPO with FSDP, parameter/gradient/optimizer offloading, batch size 32, vLLM rollouts, temperature 1.0, top-$(y^+, y^-)$1 1.0, 8 candidates per prompt, KL coefficient $(y^+, y^-)$2, learning rate $(y^+, y^-)$3, no scheduler, and training on 1 node with $(y^+, y^-)$4 (Li et al., 12 Aug 2025).

A recurrent point in the paper is that SFT and RFT play different roles. SFT establishes format fidelity and initial reasoning skill; RFT is then used to explore and stabilize improved reasoning traces and comparison decisions. The paper’s training-curve discussion states that an SFT warm-start enables longer, higher-reward traces during RFT, whereas cold-start RFT is less effective.

5. Evaluation protocols, empirical performance, and observed behaviors

Evaluation is conducted on two writing-style corpora: CCAT, consisting of news articles from 50 authors, and CMCC, a multi-genre corpus containing emails, blogs, and essays from 21 authors. The splits are strictly author-disjoint to test generalization to unseen authors. A cross-domain test within CMCC further evaluates unseen genres including blogs, interviews, and chats. The reported split sizes are 45 CCAT plus 18 CMCC authors for training with approximately 17.2k pairwise-plus-reasoning samples, 2 CCAT plus 1 CMCC authors for validation with 200 samples, 3 CCAT plus 2 CMCC authors for a standard test with 334 samples, and 3 CMCC authors for a cross-domain test with 439 samples. The metric is pairwise preference accuracy, where random guessing yields 50% (Li et al., 12 Aug 2025).

Against scalar reward models, all reported baselines remain below 70% across the three test settings: Internlm2-7B-Reward achieves 67.8/69.2/64.3, RM-Mistral-7B achieves 65.7/68.1/62.8, and SR-Llama3.1-8B achieves 65.3/68.4/68.8 on CCAT, CMCC, and Cross-Domain respectively. Generative reward models scale more favorably: Qwen2.5-7B-Instruct reaches 77.6/75.8/72.4, Qwen2.5-32B-Instruct reaches 86.4/87.1/86.7, and Llama3.1-70B-Instruct reaches 94.3/94.3/93.7. Reasoning reward models are also strong, with RM-R1-7B at 89.8/88.2/89.7 and RRM-7B at 87.2/89.6/89.3. PersRM-R1 reports 91.8/92.2/89.7 for the 3B model and 93.8/94.6/92.3 for the 7B model, making the 7B variant state-of-the-art within the 7B class and competitive with substantially larger models (Li et al., 12 Aug 2025).

The paper highlights cross-domain generalization as a central result: PersRM-R1-7B attains 92.3% and PersRM-R1-3B 89.7% on the unseen-genre test. The authors interpret this as evidence that the model learns transferable stylistic principles rather than topic memorization. Training-paradigm ablations further separate the roles of the two stages. Starting from Qwen2.5-7B-Instruct, the base model yields 77.6/75.8/72.4; SFT only yields 86.1/87.7/82.2; RFT only yields 83.7/84.2/80.2; and SFT+RFT, i.e. PersRM-R1-7B, yields 93.8/94.6/92.3.

The paper also reports inference-time generalizability to additional exemplars. Using three exemplars improves Qwen2.5-7B-Instruct by +0.7/+3.4/+3.8, Llama3.1-70B-Instruct by +0.0/+0.3/+0.2, and PersRM-R1-7B by +0.3/+0.5/+0.3 across CCAT, CMCC, and Cross-Domain. Across training paradigms, SFT+RFT generalizes best to more exemplars.

Ablation and case-study analyses attribute several cognitive behaviors to the RFT stage, including Verification, Backtracking, Subgoal Setting, and Backward Chaining. The paper states that, under SFT-only training, these phenomena do not appear except subgoal setting. Task-specific behaviors include the discovery of novel, case-specific criteria and dynamic rubric prioritization. In the featured example, a user’s style depends on arguments grounded in personal stories; the model introduces “Argumentation Grounded in Personal Narrative” as the decisive criterion, backtracks from an initial hypothesis after deeper examination, verifies the logic, and emits scores [[9, 7]] consistent with the structured analysis. This example is presented as evidence that minimal exemplars can support nontrivial stylistic generalization (Li et al., 12 Aug 2025).

6. Position within reward modeling, practical use, and limitations

PersRM-R1 is explicitly positioned against several neighboring paradigms. Relative to standard scalar reward modeling trained with Bradley–Terry loss, it uses a generative reward model that produces explicit rationales and scores. Relative to vanilla RLHF, the reinforcement stage optimizes the reward model itself so that it can better reason and judge personalized preferences, rather than training only a downstream policy. Relative to DPO and KTO, which directly optimize policies from preferences, PersRM-R1 aims to build a reusable personalized reward model capable of evaluating arbitrary candidate-response pairs conditioned on exemplars. The paper presents this as an adaptation of the reasoning-reward-model line, exemplified by RM-R1, to the personalization setting through exemplar-conditioned synthetic contrasts, faithful rationale filtering, and a two-stage SFT+RFT pipeline suited to limited user data (Li et al., 12 Aug 2025, Chen et al., 5 May 2025).

In practical terms, the framework is intended to support personalization from 1–3 short writings that reflect a target style. The reported end-to-end recipe is to collect exemplars, synthesize contrastive pairs via intra-author retrieval or lexical perturbation for positives and cross-author retrieval, random style generation, and confounding sampling for negatives, generate reasoning tuples with a strong LLM, filter for format and label consistency, train a 3B or 7B generative reward model via SFT+RFT, and then use the resulting reward model to rank or score candidate outputs conditioned on the user’s exemplars. The released code and datasets are hosted at the repository specified by the paper.

The framework’s stated limitations are specific. It relies on synthetic data and format-constrained reasoning traces, so poor adherence to format by external LLMs can weaken supervision. Its empirical evaluation is limited to writing style, and extending it to other domains requires appropriate exemplars and domain-specific contrastive sampling. Extremely noisy or contradictory exemplars can confuse both data construction and filtering, degrading robustness. The ethical discussion emphasizes that personalization can reinforce harmful or biased stylistic preferences unless safeguards and filters are applied during exemplar and negative construction, and it explicitly raises consent and privacy concerns for personal corpora (Li et al., 12 Aug 2025).

The overall implication advanced by the paper is that personalized reward modeling can be made both more accurate and more interpretable by recasting it as exemplar-conditioned reasoning. That implication remains bounded by the paper’s experimental scope, but within that scope PersRM-R1 presents personalization as a setting in which explicit comparative analysis, structured rationales, and RL-induced reasoning behaviors materially improve reward-model generalization under one- or few-shot supervision.