---
title: Customizable Reward Models Overview
url: https://www.emergentmind.com/topics/customizable-reward-models-crms
type: topic
---

# Customizable Reward Models Overview

Searching arXiv for the cited CRM-related papers to ground the article in current literature.
Customizable Reward Models (CRMs) are reward models that replace a single fixed notion of response quality with a configurable preference function conditioned on explicit criteria, user profiles, rubrics, dynamic checklists, in-context preference demonstrations, domain prompts, or safety configurations. Taken together, recent work treats CRMs as an answer to pluralistic alignment: general reward models capture broad signals such as correctness, helpfulness, and harmlessness, but they do not adequately represent subjective, domain-specific, or user-specific preferences, nor do they expose the decision basis in an interpretable form [2309.03126; 2508.09463; 2601.02986; 2602.08819; 2605.30487].

## 1. Conceptual scope and motivation

The central motivation for CRMs is that human preference is neither one-dimensional nor stable across users, domains, and scenarios. Early CRM work distinguished **general preferences**—preferences satisfying universal human values, including helpfulness and harmlessness—from **customized preferences** related to “a subgroup of people or a particular scenario,” and argued that a single general reward model is insufficient for pluralistic deployment settings [2309.03126]. Subsequent work strengthened this claim with empirical analyses of subjective data. The User-centric Subjective Leaderboard study analyzed **10,794 real preference comparisons** from LMArena and reported that subjective preferences exhibit “significant diversity and contradictions,” including an **Adjusted Rand Index = 0.001** for clustering chosen versus rejected criteria embeddings, indicating essentially no separable distribution of preference criteria [2508.09463].

A second motivation is personalization. Personalized RewardBench isolates user-specific preference by constructing chosen and rejected responses that are both high quality on general dimensions but differ sharply on user rubrics: human evaluation reports chosen responses at about **4.84–4.93** on personal rubrics versus **1.44–1.49** for rejected responses, while both remain high on correctness, relevance, and helpfulness [2604.07343]. This benchmark also shows that existing reward models still struggle with personalization, with the best reported accuracy peaking at **75.94%**. A plausible implication is that many failures attributed to “reward model weakness” are specifically failures of conditioning, not merely failures of scale.

Interpretability is a third motivation. Scalar or pairwise reward models rank responses but usually do not reveal why. Rubric-based and checklist-based CRMs address this by exposing explicit criteria; multi-agent and token-level reward decompositions address it by exposing partial signals or reward trajectories. This suggests that customization and interpretability are tightly coupled in the current literature: the same structural changes that make reward functions configurable often also make them inspectable.

## 2. Formal interfaces and conditioning mechanisms

At a formal level, CRMs preserve the standard pairwise reward-modeling substrate while modifying the conditioning variables. The baseline remains the Bradley–Terry or ranking-style objective over prompt–response pairs, but the reward function is no longer solely $r(x,y)$. Instead, it becomes criterion-conditioned, rubric-conditioned, user-conditioned, or context-conditioned. For example, the subjective CRM formulation conditions on an explicit preference criterion $c$; OpenRubrics represents a prompt-specific rubric as $\mathcal{R}(x)=\{c_i\}_{i=1}^k$; P-Check scores a candidate with a generated checklist $C_{u,q}$; Meta Reward Modeling expresses each user reward as a weighted combination of basis functions; ICRM conditions on in-context demonstrations $\mathcal{C}$; and CSRM conditions on a safety configuration $\mathbf{p}$ [2508.09463; 2510.07743; 2601.02986; 2601.18731; 2602.08819; 2605.30487].

| Conditioning interface | Representative form | Paper |
|---|---|---|
| Criterion-conditioned pairwise RM | $(c,q,o^{\rm A},o^{\rm B},y_c)$ | [2508.09463] |
| Rubric-conditioned RM | $r_\theta(x,\hat y_1,\hat y_2;\{c_i\}_{i=1}^k)$ | [2510.07743] |
| Personalized checklist RM | $r_{u,q}(y)=\mathbf{w}_{u,q}^{\top}\theta(GP_u,q,y,\hat C_{u,q})$ | [2601.02986] |
| Meta-personalized RM | $r_{w_i}(x,y)=\sum_{k=1}^K w_{i,k}\phi_k(x,y)$ | [2601.18731] |
| In-context steerable RM | $z=P(y_w \succ y_l\mid x,\mathcal C)$ | [2602.08819] |
| Configurable safety RM | input $(x,r,\mathbf p)$, output $(y,c)$ | [2605.30487] |

This conditioning shift changes what is meant by “the reward model.” In conventional RLHF, the model approximates a single latent preference function. In CRM formulations, the reward model instead approximates a *family* of preference functions indexed by a control variable such as a rubric, profile, or policy. A plausible implication is that the reward model becomes closer to a conditional evaluator or preference interpreter than to a static scalar judge.

The same shift also changes what counts as supervision. Some CRMs still rely on pairwise preference labels, but others introduce criterion extraction, rubric synthesis, checklist generation, strictness augmentation, or preference demonstrations as additional supervision channels. In practice, the literature treats the control variable as either supplied externally, inferred from data, or generated by a learned module.

## 3. Rubrics, checklists, and explicit criteria

One major CRM line replaces opaque scalar rewards with structured natural-language criteria. OpenRubrics defines a rubric as $\mathcal{R}(x)=\{c_i\}_{i=1}^k$ and distinguishes **Hard Rules**—explicit prompt requirements such as length limits, format constraints, required citations, or forbidden content—from **Principles** such as reasoning soundness, factuality, clarity, coherence, creativity, or stylistic quality. Its Contrastive Rubric Generation (CRG) derives rubrics from preference tuples $\mathcal{D}=\{(x_i,\hat y_i^+,\hat y_i^-)\}_{i=1}^N$, then filters them via rejection sampling so that the rubric is retained only when a rubric-guided judgment reproduces the original label. The resulting Rubric-RM reportedly surpasses strong same-scale baselines by **6.8% on average** across reward-modeling benchmarks and improves downstream policy optimization by about **2.9% on average** [2510.07743].

CDRRM extends rubric-based reward modeling with a **Contrast-then-Synthesis** pipeline. It first performs multi-dimensional contrastive profiling over chosen and rejected responses, anchors judgments to evidence spans, and only then synthesizes a compact, context-aware rubric. It retains a rubric only if a rubric-guided judge predicts the original preference correctly. On RewardBench, RM-Bench, and RMB, CDRRM-14B (SFT) reports **88.3 average**, compared with **83.5 average** for the best rubric-based baseline and **85.2 average** for the best generative RM; the paper also emphasizes that training the rubric generator on only **3k** samples enables a frozen judge to outperform fully fine-tuned baselines [2603.08035].

P-Check moves from static user conditioning to dynamic, query-specific evaluation criteria. Given user history $H_u$ and query $q$, it constructs a general profile $GP_u$, generates a checklist $C_{u,q}$, assigns criterion saliency through Preference-Contrastive Criterion Weighting, and aggregates criterion scores as $r_{u,q}(y)=\mathbf{w}_{u,q}^{\top}\theta(GP_u,q,y,\hat C_{u,q})$. Its reported average binary preference accuracy is **63.62%**, compared with **53.19%** for the Default baseline, and it improves downstream personalized generation under both Best-of-N and DPO [2601.02986].

A related domain-specific variant is ER²Score for radiology report evaluation. It uses a multi-reward head to output criterion-specific sub-rewards and defines the overall score as $\text{ER}^2\text{Score}=\sum_{j=1}^N r^{i,j}$. The method is trained with a **Margin Reward Enforcement (MRE) loss** over accepted and rejected reports generated with GPT-4 under either a **RadCliQ-style** 6-category rubric or an **MRScore-style** 7-item rubric. On ReXVal, ER²Score reports **Kendall’s Tau 0.751** and **Spearman 0.910** for total score, outperforming traditional metrics [2411.17301].

These rubric and checklist approaches share a common structural claim: reward customization is easier when the reward function is decomposed into named criteria rather than compressed into an undifferentiated scalar.

## 4. Personalized and meta-adaptive CRMs

Personalized CRMs aim to model not merely what humans prefer on average, but what a particular user prefers. Personalized RewardBench formalizes this as tuples $(q,u,y_c,y_r)$, where $u$ is a user profile built by retrieving top $k=10$ historical items with Contriever, and where $y_c=\text{LLM}(q,u,\mathcal R_u)$ and $y_r=\text{LLM}(q,u,\neg\mathcal R_u)$. Its evaluation considers **w/o profile**, **w/ profile**, and **w/ plan**, with the planner defined by $\tilde R_u=\mathcal P(q,u)$. The benchmark’s downstream validation reports stronger correlation with policy performance than prior personalized baselines, including **NDCG = 0.9180** and **Weighted $\tau = 0.3409$** for Best-of-N, and **NDCG = 0.9265** and **Weighted $\tau = 0.4793$** for PPO [2604.07343].

Meta Reward Modeling (MRM) treats each user as a meta-learning task. It represents user $i$’s reward as
$$
r_{w_i}(x,y)=\sum_{k=1}^K w_{i,k}\phi_k(x,y),
$$
with user-specific weights $w_i$ adapted from a learned initialization $w_0$ by a MAML-style inner loop, and with the **Robust Personalization Objective (RPO)** emphasizing high-loss users during meta-optimization. On PRISM and Reddit TLDR, MRM reports the best overall user-level accuracy in the reported settings, including **65.3%** on PRISM overall and **69.6% / 69.7%** on Reddit TLDR with 100 / 150 examples, while also improving performance on the worst **10%**, **20%**, and **50%** of users [2601.18731].

ICRM addresses personalization and customization through test-time steerability rather than per-user fine-tuning. It models the latent preference probability
$$
z \coloneqq P(y_w \succ y_l \mid x, \mathcal{C}) \in [0,1]
$$
under a Bradley–Terry mechanism with a conjugate Beta prior and a variational Beta posterior. Because the posterior is conditioned on in-context demonstrations $\mathcal C$, the same trained model can adapt to unseen preference distributions at inference time. The paper reports up to **34% accuracy gain** on SafeRLHF and **9% accuracy gain** on RM-Bench with more demonstrations, and a **4% gain in hypervolume** in helpfulness–refusal trade-off experiments [2602.08819].

Across these personalized lines, a recurring pattern is that raw user history is not treated as sufficient. Personalized RewardBench finds that naive profile injection can hurt; P-Check inserts an explicit checklist generator; MRM learns fast adaptation in a low-dimensional reward basis; ICRM uses demonstrations as online preference evidence. This suggests that personalization in reward modeling is increasingly implemented through an intermediate structure—planner output, checklist, basis weights, or posterior belief—rather than by simple concatenation of profile text.

## 5. Domain specialization, safety configuration, and training-free customization

A second major CRM line addresses domain-specific or policy-specific reward modeling. The 2023 DSP work introduced a **three-stage customized RM learning scheme** consisting of base LM training, **General RM Fine-tuning (GRFT)** on Helpful/Harmless, WebGPT, and GPT-4-LLM, and **Customized RM Fine-tuning (CRFT)** on a new Domain-Specific Preference dataset covering **Academy**, **Business**, **Entertainment**, and **Literature / Art**. It found that general preference enrichment and customized imitation learning, especially with **$\mu=0.1$**, improve preservation of general preference ability during customization [2309.03126].

DogeRM addresses domain customization without domain-specific pairwise preference labels. It merges a general reward model with a domain-specific supervised fine-tuned model via parameter interpolation:
$$
A^{\mathrm{MERGE}}_{\mathrm{trans}}=\lambda O^{\mathrm{SFT}}_{\mathrm{trans}}+(1-\lambda)O^{\mathrm{RM}}_{\mathrm{trans}},
$$
while preserving the original RM regression head. The paper reports RewardBench improvements such as **11.4%** and **17%** on math when merging with MetaMath-7B and MAmmoTH-7B, and Best-of-N reranking gains of about **5%** on GSM8K at best-of-16 and about **1.4%** on MBPP [2407.01470].

Dr.SoW, described in the details as customizable strong-over-weak density-ratio reward, constructs a training-free reward
$$
r(x,y)=\log \pi_{\text{strong}}(y\mid x)-\log \pi_{\text{weak}}(y\mid x),
$$
then customizes it by query domain through routing and prompt templates $\mathcal T(x)$. With a Mistral-7B strong/weak pair, the method reports **82.6 overall RewardBench**, with domain-specific gains such as **82.4 → 91.0** in Safety, **73.8 → 88.0** in Reasoning, and **60.4 → 69.7** in ChatHard; downstream SimPO tuning pushes Llama-3-8B-Instruct to **37.4%** win rate on ArenaHard and **40.7%** length-controlled win rate on AlpacaEval 2.0 [2411.02481].

CSRM extends customization to safety policy itself. It conditions on a natural-language safety configuration $\mathbf p$, takes input $(x,r,\mathbf p)$, and outputs both a safety label and a scalar reward. Training combines classification with reward modeling over paired strict and lenient configurations, supported by **configuration-targeted augmentation** and **strictness augmentation**. CSRM reports **94.6% F1** on CoSApien and **75.8% F1** on DynaBench, along with pairwise reward accuracies of **0.782** on BeaverTails-Aug, **0.763** on Safety-RLHF, and **0.648** on WildGuardMix-Aug, all **without requiring additional human annotation** [2605.30487].

These domain and safety variants indicate that CRM need not mean direct user personalization. In this literature, “customization” also includes post hoc domain adaptation, domain-routed reward construction, and test-time policy reconfiguration.

## 6. Evaluation, downstream effects, and adjacent developments

CRM research places unusual weight on evaluation methodology, because a customizable reward model is useful only if its benchmark performance predicts downstream policy behavior. Personalized RewardBench explicitly argues this point and measures correlation with **Best-of-N (BoN)** and **PPO**. OpenRubrics reports transfer from reward-model benchmarks to instruction-following and biomedical policy models. DogeRM evaluates not only RewardBench and Auto-J Eval but also Best-of-N reranking on GSM8K and MBPP. CSRM studies downstream safety alignment with Reward Distillation and REINFORCE++, reporting a dominant safety-helpfulness Pareto frontier in held-out CoSApien configurations [2604.07343; 2510.07743; 2407.01470; 2605.30487].

Several limitations recur across the literature. First, benchmark accuracy on generic quality is not enough: Personalized RewardBench shows that both chosen and rejected responses can remain high-quality in general while differing mainly in personal alignment [2604.07343]. Second, control variables must be structured. P-Check reports that explicit checklist generation outperforms static user conditioning; Personalized RewardBench finds that a planner converting profile to rubric aspects works better than raw profile injection; OpenRubrics and CDRRM both use filtering to remove noisy rubrics [2601.02986; 2604.07343; 2510.07743; 2603.08035]. Third, customization may introduce trade-offs: DogeRM notes possible drops in **chat-hard** and **safety** for some merges, and P-Check explicitly states that personalization should be used alongside safety constraints [2407.01470; 2601.02986].

The acronym “CRM” is also used in adjacent but not identical reward-modeling lines. “Conditional Reward Modeling for LLM Reasoning” defines step rewards as $r_t=\log(1-h(t))$ under a temporal-causal model linked to final correctness, improving Best-of-N, beam search, and RL robustness to reward hacking [2509.26578]. “Temporally Coherent Reward Modeling” adds lookahead and smoothness regularizers to make intermediate token scores approximate conditional expectations of final reward, raising middle-token pairwise accuracy from near chance to as high as **88.9%** while preserving final-token accuracy [2604.22981]. These are not user-configurable CRMs in the strict sense, but they suggest a broader tendency: reward modeling is moving away from a single terminal scalar toward structured, modular, and condition-dependent reward representations.

Taken together, current CRM research defines a shift in alignment methodology. The reward model is no longer merely a static proxy for average human judgment. It becomes a configurable mechanism for specifying *which* preference function should apply, *why* one response is preferred, and *how* that preference should transfer to search, reranking, or policy optimization.

Source: https://www.emergentmind.com/topics/customizable-reward-models-crms