Personalized Generative Reward Model (GRM)

• Personalized GRM is a framework that reformulates reward inference as a generative reasoning task by outputting explicit evaluation chains such as chain-of-thought or checklists.
• It employs dual-granularity test-time scaling by aggregating individual and prototype-level evaluations to robustly capture diverse user-specific preferences.
• The model enhances interpretability, transferability, and scalability in RLHF pipelines by generating structured intermediate evaluation data that mitigates reward hacking.

A Personalized Generative Reward Model (GRM) is an advanced framework in personalized alignment of LLMs, where the reward inference process is reformulated as a generative reasoning task producing explicit, structured evaluation traces—often in the form of chain-of-thought, checklists, or critique—with the dual goals of interpretability and adaptability to user-specific preferences. Unlike earlier scalar or static-contextual reward models, GRMs generate extensive intermediate structures that explicitly represent personalized evaluation criteria and provide enhanced robustness against reward hacking, superior faithfulness in alignment, and improved transfer and scalability. State-of-the-art systems such as P-GenRM and P-Check embody the generative reward modeling paradigm through dynamic evaluation chain construction, test-time user-based scaling, and multi-granular personalization (Zhang et al., 12 Feb 2026, Seo et al., 6 Jan 2026, Zhu et al., 21 Oct 2025).

1. Formal Definition and Problem Motivation

A GRM is a model that, for a given user $u$, query $q$, context or persona $P$, and candidate response $y$, generates a structured reasoning chain or checklist $S$ conditioned on user-specific history $H^{(u)}$, possibly dynamic evaluation rubrics, or critique content, from which scalar (or vector) rewards are then extracted or predicted. This extends traditional reward modeling by making the evaluation process explicit and query-adaptive rather than static, thus addressing major limitations in prior scalar-based and latent-context reward models which (1) compress evaluation into inflexible criteria and (2) cannot generalize robustly to new users or intra-user preference variability (Zhang et al., 12 Feb 2026, Seo et al., 6 Jan 2026). The GRM paradigm is motivated by the need for accurate, interpretable, user-specific feedback signals in RLHF pipelines for LLMs, especially where high preference diversity or open-ended response spaces arise.

2. Architectural Principles and Data Flow

Personalized GRMs, as exemplified by P-GenRM, are instantiated as generative LLMs with specialized input representations and structured outputs. The primary workflow includes:

• Input signals: the current user query $q_t$, implicit feedback history $$H_t^{(u)} = \{(q_\tau,y_\tau^+,y_\tau^-)\}_{\tau<t}$$ of size $\mathbf{h}$, and optionally explicit criteria $E^{(u)}$.
• Generation: The GRM $q$0 produces a composite evaluation chain $q$1, where $q$2 is a scenario-specific persona and $q$3 is a weighted rubric or checklist.
• Score extraction: From $q$4, one extracts scalar scores $q$5 for batch $q$6 candidate responses.
• Advanced derivations include dual output heads (for critique + multiple scores) as in (Zhu et al., 21 Oct 2025), and dynamic query-adaptive checklist generation (Seo et al., 6 Jan 2026).

Data flows from historical interaction through persona/rubric synthesis to explicit, scenario- and user-conditioned reward prediction. This design supports dynamic adjustment to both user- and scenario-level shifts in evaluation priorities and provides interpretability by surfacing the reasoning behind each reward.

3. Mathematical Formulation

The reward function in a personalized GRM directly incorporates the generative evaluation chain. For a given user $q$7 at turn $q$8: $q$9 The model outputs

$P$0

The total reward used in RL is a convex combination: $P$1 where $P$2 is the process-level reward (for chain/evaluation quality) and $P$3 is the outcome reward for correct response ranking. This is optimized using Generative Reward Policy Optimization (GRPO): $P$4 where $P$5 is a sampled chain, $P$6 the likelihood ratio, and $P$7 the advantage. The loss functions for checklist construction (cross-entropy), critique generation (NLL), and score regression (MSE) are used in other GRMs (Seo et al., 6 Jan 2026, Zhu et al., 21 Oct 2025).

4. Personalization and Dual-Granularity Test-time Scaling

P-GenRM introduces a novel dual-granularity scaling at test time:

• Individual-level scaling: Multiple ($P$8) evaluation chains are generated for a user and aggregated for robustness.
• Prototype-level scaling: The user is matched to $P$9 similar users (“prototypes”), and rubric/score information is shared by aggregating their outputs. Scores for each candidate response $y$0 are aggregated as: $y$1 Prototypes are initialized via K-means over persona embeddings and iteratively refined via a history-aware attentive mechanism: $y$2

$y$3

The pairwise loss and regularization terms encourage both per-instance discrimination and prototype stability. This scaling mechanism improves user-level consistency and OOD generalization capacity, mitigating preference estimation noise and information sparsity in unseen users (Zhang et al., 12 Feb 2026).

5. Training Paradigms and Losses

Training of GRMs occurs in staged regimes:

• Supervised Fine-Tuning (SFT): The model learns to generate evaluation chains from hybrid ground truth signals.
• Reinforcement Learning: Criteria-based reasoning enhancement using REINFORCE/PPO-style updates and GRPO, optimizing both chain quality and ranking accuracy.
• Curriculum Learning: Hard negative samples are introduced progressively, with process reward turned off for stricter outcome focus.
• For P-Check, the checklist generator is trained via cross-entropy on synthetic (persona, query, checklist) tuples, and reward prediction leverages a contrastive weighting of personalized criteria (Seo et al., 6 Jan 2026); in (Zhu et al., 21 Oct 2025), joint textual critique and score regression are optimized with balanced NLL + MSE losses.

6. Empirical Evaluation and Results

Extensive evaluation is reported on publicly available and proprietary personalized reward modeling benchmarks:

• Datasets: Chatbot Arena-Personalized (131 users), PRISM-Personalized (up to 720 users), LaMP-QA (OOD), BESPOKE-MetaEval.
• Metrics: Pairwise ranking accuracy, Spearman’s $y$4 for QA, best-of-N selection, direct preference optimization metrics (ROUGE-L, METEOR, BESPOKE-Eval).
• Baselines: In-context LLM-as-judge, Bradley-Terry, GPO/VPL/PAL, SynthesizeMe, OpenAI-o3.
• Results:
  • P-GenRM-8B: 72.68% (Arena) / 65.32% (PRISM), +2.77% average over SOTA; P-GenRM-70B: 73.42% / 66.21%, +1.99% over SOTA.
  • Test-time scaling yields ≈3% absolute improvements; LaMP-QA performance surpasses larger non-personalized models.
  • Ablations show 2–6% performance drops when omitting key training stages; SFT-only baseline yields ≈56%.
  • Prototype aggregation exhibits stable macro-accuracy across diverse persona distributions (Zhang et al., 12 Feb 2026, Seo et al., 6 Jan 2026, Zhu et al., 21 Oct 2025).

7. Qualitative Analyses, Limitations, and Future Directions

Qualitative case studies illustrate nuanced evaluation chains: e.g., a single user displaying radically different personas and rubric weights across “music recommendation” (pragmatic, inquisitive; high on helpfulness/factuality) versus “moral discussion” (concise, nuanced, philosophical) (Zhang et al., 12 Feb 2026). Test-time prototype scaling consistently reduces noisy or idiosyncratic reward estimates. Checklists and textual critiques in P-Check and Critique-Post-Edit frameworks surface explicit user criteria, increase downstream generation quality, and resist reward hacking (Seo et al., 6 Jan 2026, Zhu et al., 21 Oct 2025).

GRMs entail increased inference cost due to chain generation, demand several preference samples for robust persona extraction, and pipeline complexity that could limit deployment. Prospective research areas include lightweight chain distillation, dynamic prototype structures, hybridizing explicit and latent personalization signals, and more extensive integration of multimodal feedback (Zhang et al., 12 Feb 2026, Seo et al., 6 Jan 2026). A plausible implication is that more granular or ensemble-based prototypes might further enhance data efficiency and generalization in low-data or cold-start user regimes.

---

Key References:

• P-GenRM: Personalized Generative Reward Model with Test-time User-based Scaling (Zhang et al., 12 Feb 2026)
• P-Check: Advancing Personalized Reward Model via Learning to Generate Dynamic Checklist (Seo et al., 6 Jan 2026)
• Towards Faithful and Controllable Personalization via Critique-Post-Edit Reinforcement Learning (Zhu et al., 21 Oct 2025)