Customizable Reward Models Overview

• Customizable Reward Models (CRMs) are configurable reward systems that replace a fixed quality metric with user-specific, explicit criteria such as rubrics, checklists, and safety configurations.
• They employ conditioning mechanisms—like criterion-, rubric-, and context-based methods—to generate interpretable, personalized reward functions that expose decision-making processes.
• CRMs enhance AI alignment by tailoring evaluation to domain-specific and subjective preferences, evidenced by improved performance metrics and downstream policy optimization in various studies.

Searching arXiv for the cited CRM-related papers to ground the article in current literature. Customizable Reward Models (CRMs) are reward models that replace a single fixed notion of response quality with a configurable preference function conditioned on explicit criteria, user profiles, rubrics, dynamic checklists, in-context preference demonstrations, domain prompts, or safety configurations. Taken together, recent work treats CRMs as an answer to pluralistic alignment: general reward models capture broad signals such as correctness, helpfulness, and harmlessness, but they do not adequately represent subjective, domain-specific, or user-specific preferences, nor do they expose the decision basis in an interpretable form (Cheng et al., 2023, Jia et al., 13 Aug 2025, Seo et al., 6 Jan 2026, Hong et al., 9 Feb 2026, Jiang et al., 28 May 2026).

1. Conceptual scope and motivation

The central motivation for CRMs is that human preference is neither one-dimensional nor stable across users, domains, and scenarios. Early CRM work distinguished general preferences—preferences satisfying universal human values, including helpfulness and harmlessness—from customized preferences related to “a subgroup of people or a particular scenario,” and argued that a single general reward model is insufficient for pluralistic deployment settings (Cheng et al., 2023). Subsequent work strengthened this claim with empirical analyses of subjective data. The User-centric Subjective Leaderboard study analyzed 10,794 real preference comparisons from LMArena and reported that subjective preferences exhibit “significant diversity and contradictions,” including an Adjusted Rand Index = 0.001 for clustering chosen versus rejected criteria embeddings, indicating essentially no separable distribution of preference criteria (Jia et al., 13 Aug 2025).

A second motivation is personalization. Personalized RewardBench isolates user-specific preference by constructing chosen and rejected responses that are both high quality on general dimensions but differ sharply on user rubrics: human evaluation reports chosen responses at about 4.84–4.93 on personal rubrics versus 1.44–1.49 for rejected responses, while both remain high on correctness, relevance, and helpfulness (Ma et al., 8 Apr 2026). This benchmark also shows that existing reward models still struggle with personalization, with the best reported accuracy peaking at 75.94%. A plausible implication is that many failures attributed to “reward model weakness” are specifically failures of conditioning, not merely failures of scale.

Interpretability is a third motivation. Scalar or pairwise reward models rank responses but usually do not reveal why. Rubric-based and checklist-based CRMs address this by exposing explicit criteria; multi-agent and token-level reward decompositions address it by exposing partial signals or reward trajectories. This suggests that customization and interpretability are tightly coupled in the current literature: the same structural changes that make reward functions configurable often also make them inspectable.

2. Formal interfaces and conditioning mechanisms

At a formal level, CRMs preserve the standard pairwise reward-modeling substrate while modifying the conditioning variables. The baseline remains the Bradley–Terry or ranking-style objective over prompt–response pairs, but the reward function is no longer solely $r(x,y)$. Instead, it becomes criterion-conditioned, rubric-conditioned, user-conditioned, or context-conditioned. For example, the subjective CRM formulation conditions on an explicit preference criterion $c$; OpenRubrics represents a prompt-specific rubric as $\mathcal{R}(x)=\{c_i\}_{i=1}^k$; P-Check scores a candidate with a generated checklist $C_{u,q}$; Meta Reward Modeling expresses each user reward as a weighted combination of basis functions; ICRM conditions on in-context demonstrations $\mathcal{C}$; and CSRM conditions on a safety configuration $\mathbf{p}$ (Jia et al., 13 Aug 2025, Liu et al., 9 Oct 2025, Seo et al., 6 Jan 2026, Cai et al., 26 Jan 2026, Hong et al., 9 Feb 2026, Jiang et al., 28 May 2026).

Conditioning interface	Representative form	Paper
Criterion-conditioned pairwise RM	$(c,q,o^{\rm A},o^{\rm B},y_c)$	(Jia et al., 13 Aug 2025)
Rubric-conditioned RM	$r_\theta(x,\hat y_1,\hat y_2;\{c_i\}_{i=1}^k)$	(Liu et al., 9 Oct 2025)
Personalized checklist RM	$$r_{u,q}(y)=\mathbf{w}_{u,q}^{\top}\theta(GP_u,q,y,\hat C_{u,q})$$	(Seo et al., 6 Jan 2026)
Meta-personalized RM	$r_{w_i}(x,y)=\sum_{k=1}^K w_{i,k}\phi_k(x,y)$	(Cai et al., 26 Jan 2026)
In-context steerable RM	$c$0	(Hong et al., 9 Feb 2026)
Configurable safety RM	input $c$1, output $c$2	(Jiang et al., 28 May 2026)

This conditioning shift changes what is meant by “the reward model.” In conventional RLHF, the model approximates a single latent preference function. In CRM formulations, the reward model instead approximates a family of preference functions indexed by a control variable such as a rubric, profile, or policy. A plausible implication is that the reward model becomes closer to a conditional evaluator or preference interpreter than to a static scalar judge.

The same shift also changes what counts as supervision. Some CRMs still rely on pairwise preference labels, but others introduce criterion extraction, rubric synthesis, checklist generation, strictness augmentation, or preference demonstrations as additional supervision channels. In practice, the literature treats the control variable as either supplied externally, inferred from data, or generated by a learned module.

3. Rubrics, checklists, and explicit criteria

One major CRM line replaces opaque scalar rewards with structured natural-language criteria. OpenRubrics defines a rubric as $c$3 and distinguishes Hard Rules—explicit prompt requirements such as length limits, format constraints, required citations, or forbidden content—from Principles such as reasoning soundness, factuality, clarity, coherence, creativity, or stylistic quality. Its Contrastive Rubric Generation (CRG) derives rubrics from preference tuples $c$4, then filters them via rejection sampling so that the rubric is retained only when a rubric-guided judgment reproduces the original label. The resulting Rubric-RM reportedly surpasses strong same-scale baselines by 6.8% on average across reward-modeling benchmarks and improves downstream policy optimization by about 2.9% on average (Liu et al., 9 Oct 2025).

CDRRM extends rubric-based reward modeling with a Contrast-then-Synthesis pipeline. It first performs multi-dimensional contrastive profiling over chosen and rejected responses, anchors judgments to evidence spans, and only then synthesizes a compact, context-aware rubric. It retains a rubric only if a rubric-guided judge predicts the original preference correctly. On RewardBench, RM-Bench, and RMB, CDRRM-14B (SFT) reports 88.3 average, compared with 83.5 average for the best rubric-based baseline and 85.2 average for the best generative RM; the paper also emphasizes that training the rubric generator on only 3k samples enables a frozen judge to outperform fully fine-tuned baselines (Liu et al., 9 Mar 2026).

P-Check moves from static user conditioning to dynamic, query-specific evaluation criteria. Given user history $c$5 and query $c$6, it constructs a general profile $c$7, generates a checklist $c$8, assigns criterion saliency through Preference-Contrastive Criterion Weighting, and aggregates criterion scores as $c$9. Its reported average binary preference accuracy is 63.62%, compared with 53.19% for the Default baseline, and it improves downstream personalized generation under both Best-of-N and DPO (Seo et al., 6 Jan 2026).

A related domain-specific variant is ER²Score for radiology report evaluation. It uses a multi-reward head to output criterion-specific sub-rewards and defines the overall score as $\mathcal{R}(x)=\{c_i\}_{i=1}^k$0. The method is trained with a Margin Reward Enforcement (MRE) loss over accepted and rejected reports generated with GPT-4 under either a RadCliQ-style 6-category rubric or an MRScore-style 7-item rubric. On ReXVal, ER²Score reports Kendall’s Tau 0.751 and Spearman 0.910 for total score, outperforming traditional metrics (Liu et al., 2024).

These rubric and checklist approaches share a common structural claim: reward customization is easier when the reward function is decomposed into named criteria rather than compressed into an undifferentiated scalar.

4. Personalized and meta-adaptive CRMs

Personalized CRMs aim to model not merely what humans prefer on average, but what a particular user prefers. Personalized RewardBench formalizes this as tuples $\mathcal{R}(x)=\{c_i\}_{i=1}^k$1, where $\mathcal{R}(x)=\{c_i\}_{i=1}^k$2 is a user profile built by retrieving top $\mathcal{R}(x)=\{c_i\}_{i=1}^k$3 historical items with Contriever, and where $\mathcal{R}(x)=\{c_i\}_{i=1}^k$4 and $\mathcal{R}(x)=\{c_i\}_{i=1}^k$5. Its evaluation considers w/o profile, w/ profile, and w/ plan, with the planner defined by $\mathcal{R}(x)=\{c_i\}_{i=1}^k$6. The benchmark’s downstream validation reports stronger correlation with policy performance than prior personalized baselines, including NDCG = 0.9180 and Weighted $\mathcal{R}(x)=\{c_i\}_{i=1}^k$7 for Best-of-N, and NDCG = 0.9265 and Weighted $\mathcal{R}(x)=\{c_i\}_{i=1}^k$8 for PPO (Ma et al., 8 Apr 2026).

Meta Reward Modeling (MRM) treats each user as a meta-learning task. It represents user $\mathcal{R}(x)=\{c_i\}_{i=1}^k$9’s reward as

$C_{u,q}$0

with user-specific weights $C_{u,q}$1 adapted from a learned initialization $C_{u,q}$2 by a MAML-style inner loop, and with the Robust Personalization Objective (RPO) emphasizing high-loss users during meta-optimization. On PRISM and Reddit TLDR, MRM reports the best overall user-level accuracy in the reported settings, including 65.3% on PRISM overall and 69.6% / 69.7% on Reddit TLDR with 100 / 150 examples, while also improving performance on the worst 10%, 20%, and 50% of users (Cai et al., 26 Jan 2026).

ICRM addresses personalization and customization through test-time steerability rather than per-user fine-tuning. It models the latent preference probability

$C_{u,q}$3

under a Bradley–Terry mechanism with a conjugate Beta prior and a variational Beta posterior. Because the posterior is conditioned on in-context demonstrations $C_{u,q}$4, the same trained model can adapt to unseen preference distributions at inference time. The paper reports up to 34% accuracy gain on SafeRLHF and 9% accuracy gain on RM-Bench with more demonstrations, and a 4% gain in hypervolume in helpfulness–refusal trade-off experiments (Hong et al., 9 Feb 2026).

Across these personalized lines, a recurring pattern is that raw user history is not treated as sufficient. Personalized RewardBench finds that naive profile injection can hurt; P-Check inserts an explicit checklist generator; MRM learns fast adaptation in a low-dimensional reward basis; ICRM uses demonstrations as online preference evidence. This suggests that personalization in reward modeling is increasingly implemented through an intermediate structure—planner output, checklist, basis weights, or posterior belief—rather than by simple concatenation of profile text.

5. Domain specialization, safety configuration, and training-free customization

A second major CRM line addresses domain-specific or policy-specific reward modeling. The 2023 DSP work introduced a three-stage customized RM learning scheme consisting of base LM training, General RM Fine-tuning (GRFT) on Helpful/Harmless, WebGPT, and GPT-4-LLM, and Customized RM Fine-tuning (CRFT) on a new Domain-Specific Preference dataset covering Academy, Business, Entertainment, and Literature / Art. It found that general preference enrichment and customized imitation learning, especially with $C_{u,q}$5, improve preservation of general preference ability during customization (Cheng et al., 2023).

DogeRM addresses domain customization without domain-specific pairwise preference labels. It merges a general reward model with a domain-specific supervised fine-tuned model via parameter interpolation:

$C_{u,q}$6

while preserving the original RM regression head. The paper reports RewardBench improvements such as 11.4% and 17% on math when merging with MetaMath-7B and MAmmoTH-7B, and Best-of-N reranking gains of about 5% on GSM8K at best-of-16 and about 1.4% on MBPP (Lin et al., 2024).

Dr.SoW, described in the details as customizable strong-over-weak density-ratio reward, constructs a training-free reward

$C_{u,q}$7

then customizes it by query domain through routing and prompt templates $C_{u,q}$8. With a Mistral-7B strong/weak pair, the method reports 82.6 overall RewardBench, with domain-specific gains such as 82.4 → 91.0 in Safety, 73.8 → 88.0 in Reasoning, and 60.4 → 69.7 in ChatHard; downstream SimPO tuning pushes Llama-3-8B-Instruct to 37.4% win rate on ArenaHard and 40.7% length-controlled win rate on AlpacaEval 2.0 (Xu et al., 2024).

CSRM extends customization to safety policy itself. It conditions on a natural-language safety configuration $C_{u,q}$9, takes input $\mathcal{C}$0, and outputs both a safety label and a scalar reward. Training combines classification with reward modeling over paired strict and lenient configurations, supported by configuration-targeted augmentation and strictness augmentation. CSRM reports 94.6% F1 on CoSApien and 75.8% F1 on DynaBench, along with pairwise reward accuracies of 0.782 on BeaverTails-Aug, 0.763 on Safety-RLHF, and 0.648 on WildGuardMix-Aug, all without requiring additional human annotation (Jiang et al., 28 May 2026).

These domain and safety variants indicate that CRM need not mean direct user personalization. In this literature, “customization” also includes post hoc domain adaptation, domain-routed reward construction, and test-time policy reconfiguration.

6. Evaluation, downstream effects, and adjacent developments

CRM research places unusual weight on evaluation methodology, because a customizable reward model is useful only if its benchmark performance predicts downstream policy behavior. Personalized RewardBench explicitly argues this point and measures correlation with Best-of-N (BoN) and PPO. OpenRubrics reports transfer from reward-model benchmarks to instruction-following and biomedical policy models. DogeRM evaluates not only RewardBench and Auto-J Eval but also Best-of-N reranking on GSM8K and MBPP. CSRM studies downstream safety alignment with Reward Distillation and REINFORCE++, reporting a dominant safety-helpfulness Pareto frontier in held-out CoSApien configurations (Ma et al., 8 Apr 2026, Liu et al., 9 Oct 2025, Lin et al., 2024, Jiang et al., 28 May 2026).

Several limitations recur across the literature. First, benchmark accuracy on generic quality is not enough: Personalized RewardBench shows that both chosen and rejected responses can remain high-quality in general while differing mainly in personal alignment (Ma et al., 8 Apr 2026). Second, control variables must be structured. P-Check reports that explicit checklist generation outperforms static user conditioning; Personalized RewardBench finds that a planner converting profile to rubric aspects works better than raw profile injection; OpenRubrics and CDRRM both use filtering to remove noisy rubrics (Seo et al., 6 Jan 2026, Ma et al., 8 Apr 2026, Liu et al., 9 Oct 2025, Liu et al., 9 Mar 2026). Third, customization may introduce trade-offs: DogeRM notes possible drops in chat-hard and safety for some merges, and P-Check explicitly states that personalization should be used alongside safety constraints (Lin et al., 2024, Seo et al., 6 Jan 2026).

The acronym “CRM” is also used in adjacent but not identical reward-modeling lines. “Conditional Reward Modeling for LLM Reasoning” defines step rewards as $\mathcal{C}$1 under a temporal-causal model linked to final correctness, improving Best-of-N, beam search, and RL robustness to reward hacking (Zhang et al., 30 Sep 2025). “Temporally Coherent Reward Modeling” adds lookahead and smoothness regularizers to make intermediate token scores approximate conditional expectations of final reward, raising middle-token pairwise accuracy from near chance to as high as 88.9% while preserving final-token accuracy (Nikulkov, 24 Apr 2026). These are not user-configurable CRMs in the strict sense, but they suggest a broader tendency: reward modeling is moving away from a single terminal scalar toward structured, modular, and condition-dependent reward representations.

Taken together, current CRM research defines a shift in alignment methodology. The reward model is no longer merely a static proxy for average human judgment. It becomes a configurable mechanism for specifying which preference function should apply, why one response is preferred, and how that preference should transfer to search, reranking, or policy optimization.