Principle-Constrained Reward Modeling (PCRM)

• Principle-Constrained Reward Modeling (PCRM) is a framework that structures reward signals using explicit principles, rubrics, or constraints to ensure interpretability and alignment.
• The methodology employs two-stage learning and principled reasoning—as seen in models like WebArbiter and Orcust—to mitigate reward hacking and address spurious correlations.
• By integrating explicit evaluative criteria, PCRM improves bias mitigation and scaling efficiency, resulting in robust reward models across diverse application domains.

Searching arXiv for papers on principle-constrained reward modeling and closely related reward-modeling paradigms. Searching for "Principle-Constrained Reward Modeling" and the listed instantiations. Principle-Constrained Reward Modeling (PCRM) denotes a family of reward-modeling approaches in which reward judgments are structured by explicit principles, rubrics, or constraints rather than by opaque scalar heuristics or purely outcome-based labels. In contemporary usage, this family includes principle-guided process reward models for web, GUI, and search agents; rubric-conditioned judges for open-ended preference modeling; constrained optimization schemes that keep component reward models within their “thresholds of usefulness”; and a specific prior constraints-based reward-model training method that regulates reward-score margins through similarity-derived constraints (Zhang et al., 29 Jan 2026, Lu et al., 22 Sep 2025, Moskovitz et al., 2023, Zhou et al., 2024). Across these variants, the common objective is to make reward signals more interpretable, more robust to reward hacking and spurious correlations, and better aligned with task progress, correctness, safety, or other normative desiderata.

1. Conceptual scope and terminological usage

PCRM emerged against a background in which conventional reward models were often trained as scalar or pairwise predictors over preference data and then optimized by RLHF or related post-training schemes. Multiple papers identify recurrent pathologies in that regime: sparse and delayed supervision in long-horizon tasks, poor credit assignment, brittleness under layout or semantic changes, bias toward superficial features such as verbosity or response position, and reward overoptimization when proxy scores are pushed beyond the region where they track true evaluation (Zhang et al., 29 Jan 2026, Lu et al., 22 Sep 2025, Moskovitz et al., 2023, Liu et al., 9 Mar 2026). In response, PCRM shifts reward modeling from unconstrained scoring toward explicit evaluative structure.

A notable feature of the literature is terminological plurality. In WebArbiter and Orcust, PCRM is used in the now-common sense of explicit principle-guided reward modeling: the reward model articulates criteria such as correctness, task progress, risk, goal alignment, or efficiency, and then reasons under those criteria before issuing a verdict (Zhang et al., 29 Jan 2026, Lu et al., 22 Sep 2025). In rubric-based work such as OpenRubrics and CDRRM, the same underlying idea appears as rubrics-as-rewards or rubric-guided judging, with hard rules and principles conditioning the evaluator (Liu et al., 9 Oct 2025, Liu et al., 9 Mar 2026). By contrast, "Prior Constraints-based Reward Model Training for Aligning LLMs" uses the acronym PCRM for a distinct but related method: prior constraints on reward-score margins, based on length ratio or cosine similarity, to control reward scaling during reward-model training (Zhou et al., 2024). Constrained RLHF for composite reward models contributes another strand, treating each component reward model as a proxy with a proxy point $\theta_i^\star$ and enforcing constraints through a CMDP and Lagrange multipliers (Moskovitz et al., 2023).

Taken together, these works indicate that PCRM is best understood as a design family rather than a single algorithm. The unifying commitment is that reward modeling should be governed by explicit criteria, whether those criteria are textual principles, hard rules, rubric items, similarity-derived margin bounds, or optimization constraints on component reward magnitudes.

2. Representations of principles and constraints

The most direct PCRM formulation represents principles as natural-language evaluation criteria. WebArbiter requires outputs with <State>, <Criteria>, <Analysis>, and <Answer> sections; the <Criteria> section names context-specific evaluative dimensions such as “Clarity and helpfulness,” “Correctness of reasoning,” “Task progress,” “Situational awareness,” “Risk,” “Goal alignment,” “Strategic reasoning,” and “Efficiency / avoiding detours” (Zhang et al., 29 Jan 2026). OpenRubrics and CDRRM further split such criteria into Hard Rules, which are explicit and verifiable constraints, and Principles, which are broader qualitative criteria such as reasoning soundness, factuality, clarity, or originality (Liu et al., 9 Oct 2025, Liu et al., 9 Mar 2026). Eval-Skill generalizes this representation from per-query rubrics to domain-level “skills” composed of a high-level Workflow and low-level Principles, both injected into the judge context (Yue et al., 5 Jun 2026).

A second formulation treats principles as hybrid symbolic and learned checks. Orcust defines a dual-source principle set

$$\mathcal{P} = \left\{ p^{\text{human}}_j \right\}_{j=1}^{m} \cup \left\{ p^{\text{GPT}}_i \right\}_{i=1}^{n},$$

with deterministic predicates over state, action, and sometimes reasoning, $p_k(s_t,a_t,c_t)\in\{\text{True},\text{False}\}$, and a stepwise reward

$$R(\tau)=\sum_{t=1}^{T}\big[r^{\text{env}}(s_t,a_t\mid\mathcal{P})+r^{\text{critique}}(c_t\mid\mathcal{P})\big].$$

Here environment-verifiable principles impose non-negotiable checks, while LLM-derived principles critique coherence, planning, efficiency, and safety reasoning (Lu et al., 22 Sep 2025). Principle Process Reward (PPR) uses a similar hybridization for search agents: a principle-based process reward model evaluates each intermediate step using explicit principles such as correct extraction from information, appropriate search queries, and correct decisions about whether to search, then combines that with a verifiable outcome reward (Xu et al., 29 Sep 2025).

A third formulation encodes principles as explicit probabilistic or optimization constraints rather than textual rubrics. Conditional Reward Modeling (CRM) models reasoning through a hidden first-wrong-step variable $z$, conditional hazards $h(t)$, and survival probabilities

$$S(T)=\prod_{t=1}^{T}(1-h(t)), \qquad r_t=\log(1-h(t)),$$

so that per-step rewards are constrained to factorize the probability of a correct final answer (Zhang et al., 30 Sep 2025). Constrained RLHF over composite reward models formulates alignment as

$$\max_{\pi}\; v_0^\pi \quad \text{s.t.} \quad v_i^\pi \ge \theta_i,$$

or related upper-bound or equality constraints, where $\theta_i$ are proxy points estimated from evaluation surfaces over component rewards (Moskovitz et al., 2023). The prior constraints-based PCRM method instead constrains pairwise reward margins through

$$\Delta^*(y_1,y_2,x)=\frac{\beta_1}{\mathrm{Sim}(y_1,y_2,x)+\beta_2}+\beta_3,$$

with $$\mathcal{P} = \left\{ p^{\text{human}}_j \right\}_{j=1}^{m} \cup \left\{ p^{\text{GPT}}_i \right\}_{i=1}^{n},$$0 instantiated by length ratio or cosine similarity, thereby regulating absolute reward magnitudes and score margins during reward-model training (Zhou et al., 2024).

These representations differ in mechanism, but they share a structural property: reward is no longer a free scalar function of $$\mathcal{P} = \left\{ p^{\text{human}}_j \right\}_{j=1}^{m} \cup \left\{ p^{\text{GPT}}_i \right\}_{i=1}^{n},$$1. It is conditioned, bounded, or decomposed by explicit evaluative structure.

3. Learning pipelines and operational mechanisms

A recurrent pattern in PCRM is two-stage learning: first induce or distill principled reasoning, then align the resulting judgments against verifiable or preference-supervised correctness. WebArbiter exemplifies this. It formulates reward modeling as conditional text generation,

$$\mathcal{P} = \left\{ p^{\text{human}}_j \right\}_{j=1}^{m} \cup \left\{ p^{\text{GPT}}_i \right\}_{i=1}^{n},$$2

where the justification $$\mathcal{P} = \left\{ p^{\text{human}}_j \right\}_{j=1}^{m} \cup \left\{ p^{\text{GPT}}_i \right\}_{i=1}^{n},$$3 ends with a preference verdict. Stage 1 performs reasoning distillation from a stronger teacher that generates <Criteria>-based structured justifications; Stage 2 applies RL with binary reward and KL regularization, optimized via GRPO, to correct teacher biases while preserving the principled reasoning format (Zhang et al., 29 Jan 2026). The paper’s own interpretation identifies this as “principle-constrained reasoning,” not merely reasoning-based reward modeling.

Orcust integrates PCRM directly into reinforcement fine-tuning. After an SFT phase on curated GUI interaction trajectories, Online VM-Grounded Trajectory Construction (OVTC) generates fresh rollouts in instrumented virtual machines, logs screenshots, DOM snapshots, and input events, and computes dense stepwise rewards from environment-verifiable principle checks and a generative critique model. The GRPO update maximizes expected PCRM reward over trajectories, with the paper reporting that a 4-step reward horizon is the best reward interval for AndroidControl-Low/High (Lu et al., 22 Sep 2025).

Rubric-centered approaches emphasize principle synthesis and quality control. OpenRubrics introduces Contrastive Rubric Generation (CRG), which starts from preference tuples $$\mathcal{P} = \left\{ p^{\text{human}}_j \right\}_{j=1}^{m} \cup \left\{ p^{\text{GPT}}_i \right\}_{i=1}^{n},$$4, extracts explicit requirements from the prompt, analyzes discriminative differences between preferred and rejected responses, and then abstracts these into [Hard Rule] and [Principle] rubric items. Reliability is enforced by preference-label consistency: a rubric is retained only if re-evaluating the pair under that rubric reproduces the original preference label (Liu et al., 9 Oct 2025). CDRRM pushes this further with a Contrast-then-Synthesis pipeline: multi-dimensional contrastive profiling over a taxonomy $$\mathcal{P} = \left\{ p^{\text{human}}_j \right\}_{j=1}^{m} \cup \left\{ p^{\text{GPT}}_i \right\}_{i=1}^{n},$$5, evidence anchoring to instruction and response spans, rubric synthesis from the contrast $$\mathcal{P} = \left\{ p^{\text{human}}_j \right\}_{j=1}^{m} \cup \left\{ p^{\text{GPT}}_i \right\}_{i=1}^{n},$$6, and preference-consistency filtering before training a rubric generator and a rubric-grounded judge (Liu et al., 9 Mar 2026).

Generalist RM work broadens the space of principles and inference procedures. DeepSeek-GRM uses Self-Principled Critique Tuning (SPCT), in which the reward model itself samples principles $$\mathcal{P} = \left\{ p^{\text{human}}_j \right\}_{j=1}^{m} \cup \left\{ p^{\text{GPT}}_i \right\}_{i=1}^{n},$$7, then generates critiques conditioned on those principles, with rule-based RL rewards that depend on whether extracted scores match ground-truth rankings (Liu et al., 3 Apr 2025). Eval-Skill removes per-query rubric generation altogether: it evolves reusable domain-level skills through exploration and selection over 100 cases per domain, first generating workflows and then principles, and finally injects the selected skill directly into the judge context (Yue et al., 5 Jun 2026). PPR, in turn, demonstrates that even for non-verifiable intermediate steps, a principle-based process judge can be trained and then coupled to outcome correctness through ReNorm: $$\mathcal{P} = \left\{ p^{\text{human}}_j \right\}_{j=1}^{m} \cup \left\{ p^{\text{GPT}}_i \right\}_{i=1}^{n},$$8 which forces process rewards to be non-positive on failed trajectories and non-negative on successful ones (Xu et al., 29 Sep 2025).

4. Representative instantiations across domains

In web navigation, WebArbiter is a fully worked-out PCRM instantiation. It replaces scalar or checklist-based WebPRMs with a “reasoning-first, principle-inducing” process reward model whose outputs explicitly enumerate evaluation criteria, compare candidate actions under those criteria, and then select the preferred action (Zhang et al., 29 Jan 2026). WebPRMBench, introduced alongside it, spans four environments—Mind2Web, WebArena, AssistantBench, and WorkArena—and contains 1,150 step-level preference instances with one positive action and four negatives matched on action type. On this benchmark, WebArbiter-7B achieves 89.19% average pairwise accuracy and 74.60% average Best-of-$$\mathcal{P} = \left\{ p^{\text{human}}_j \right\}_{j=1}^{m} \cup \left\{ p^{\text{GPT}}_i \right\}_{i=1}^{n},$$9 accuracy, compared with 82.13% and 65.50% for GPT-5, and it improves reward-guided trajectory search on WebArena-Lite by up to 7.2 points over the best prior WebPRM (Zhang et al., 29 Jan 2026).

In GUI control, Orcust formulates PCRM as dense, interpretable stepwise reward for long chain-of-thought reasoning. Its principles include both explicit operational constraints—such as cursor bounds, confirmation before destructive actions, or mandatory field completion—and LLM-derived principles such as decomposition into sub-goals, milestone marking, and avoidance of unnecessary clicks (Lu et al., 22 Sep 2025). With PCRM plus OVTC, Orcust reports improvements of 22.2% on ScreenSpot and 23.9% on ScreenSpot-Pro over the base model Qwen2.5-VL-7B, and the combined EVP+LDP reward outperforms EVP-only or LDP-only ablations on AndroidControl-Low/High (Lu et al., 22 Sep 2025).

For non-verifiable tool-using agents, PPR and CRM represent two different constraint philosophies. PPR builds a principle-based process judge for search trajectories and couples it to exact-match outcome verification through reward normalization, yielding average EM 0.375 for 3B and 0.400 for 7B on the reported suite, with ablations showing that removing principles or normalization degrades performance (Xu et al., 29 Sep 2025). CRM instead encodes causal and probabilistic constraints over reasoning trajectories; it does not rely on verifiable step labels at inference, but constrains per-step rewards through the survival factorization of final correctness. Experiments across Best-of-$p_k(s_t,a_t,c_t)\in\{\text{True},\text{False}\}$0, beam search, and RL show consistent gains over existing reward models, including improved robustness to reward hacking and stable downstream improvements without relying on verifiable rewards derived from ground truth (Zhang et al., 30 Sep 2025).

In open-ended LLM evaluation, OpenRubrics, DeepSeek-GRM, Eval-Skill, and CDRRM collectively define a broad rubric- and principle-driven PCRM regime. OpenRubrics constructs a large-scale dataset of $p_k(s_t,a_t,c_t)\in\{\text{True},\text{False}\}$1 pairs and reports that Rubric-RM surpasses strong size-matched baselines by 6.8%, with gains transferring to policy models on instruction-following and biomedical benchmarks (Liu et al., 9 Oct 2025). DeepSeek-GRM demonstrates that self-generated principles, critiques, and inference-time voting can yield strong generalist reward modeling, reaching 72.8 overall with meta-RM-guided voting@32 (Liu et al., 3 Apr 2025). Eval-Skill shows that reusable domain-level evaluation skills can improve vanilla judging by +13.44% for Qwen3-8B and 18.51% for DeepSeek-V4-Flash on RewardBench 2 (Yue et al., 5 Jun 2026). CDRRM, finally, reports 88.3 average accuracy for CDRRM-14B (SFT) across RewardBench, RM-Bench, and RMB, and 83.4 on RM-Bench Hard, explicitly attributing its robustness to contrast-driven, compact, consistency-filtered rubrics (Liu et al., 9 Mar 2026).

5. Empirical properties: interpretability, scaling, and robustness

A central empirical property of PCRM systems is interpretability. WebArbiter’s <Criteria> and <Analysis> fields, OpenRubrics’ [Hard Rule] and [Principle] tags, CDRRM’s rubric JSON schema, and PPR’s per-principle analyses all expose the evaluative basis of a reward judgment (Zhang et al., 29 Jan 2026, Liu et al., 9 Oct 2025, Liu et al., 9 Mar 2026, Xu et al., 29 Sep 2025). This does not by itself guarantee faithfulness, but it makes failure analysis and debugging substantially more tractable than with scalar scores alone. CDRRM’s case studies on verbosity bias illustrate the point: when a direct judge preferred a longer but truncated “executive summary,” rubric-grounded evaluation selected the concise complete answer by applying hard rules about completeness and the absence of unrequested structure (Liu et al., 9 Mar 2026).

PCRM also exhibits several forms of scaling. Inference-time scaling is explicit in WebArbiter, DeepSeek-GRM, and Eval-Skill. WebArbiter’s Appendix G reports monotonic improvement in Pairwise and BoN accuracy as the number of sampled reward evaluations $p_k(s_t,a_t,c_t)\in\{\text{True},\text{False}\}$2 increases (Zhang et al., 29 Jan 2026). DeepSeek-GRM uses parallel sampling of principles and critiques, aggregates scores by voting, and then filters trajectories with a meta RM; SPCT-GRM-27B improves from 69.9 overall in greedy mode to 72.8 with meta-RM-guided voting@32 (Liu et al., 3 Apr 2025). Eval-Skill studies “Skill Evolution-Time Scaling” and finds that parallel exploration over workflows and principles improves test accuracy up to a plateau, whereas sequential refinement of a single skill often plateaus quickly and can fluctuate (Yue et al., 5 Jun 2026).

Another recurring empirical theme is data efficiency. CDRRM shows that training the rubric generator on only 3k high-quality samples can empower a frozen pre-trained judge model to outperform fully fine-tuned baselines, and scaling studies indicate that the rubric generator saturates quickly, with performance around 85.6 average score at 1k samples and 86.0 at 12k (Liu et al., 9 Mar 2026). OpenRubrics similarly argues that rubrics provide scalable alignment signals that narrow the gap between costly human evaluation and automated reward modeling (Liu et al., 9 Oct 2025). This suggests that explicit evaluative structure can substitute for part of the sample complexity ordinarily borne by opaque reward-model parameterization.

Bias mitigation is an especially prominent claim. CDRRM targets verbosity and position bias and reports large gains on RM-Bench Hard, a subset designed to stress subtle discrepancies and bias traps (Liu et al., 9 Mar 2026). Eval-Skill diagnoses online rubric generation failures as dominated by “Missing Criteria” and “Misaligned or Rigid,” showing that naively adding rubrics can reduce performance below vanilla judging (Yue et al., 5 Jun 2026). Constrained RLHF over composite reward models provides a complementary robustness perspective: component reward models can overoptimize beyond their proxy points, and the joint proxy point is shifted by correlation between rewards, so static scalarization is insufficient (Moskovitz et al., 2023). The prior constraints-based PCRM method addresses a related failure mode at reward-model training time by constraining score margins for similar outputs and reports improvements in reward-model accuracy, GPT-4 win rate, and DPO variants relative to unconstrained baselines (Zhou et al., 2024).

6. Limitations, controversies, and open directions

PCRM does not eliminate the classic failure modes of reward modeling; it relocates and sometimes makes them more inspectable. Teacher quality is a recurrent bottleneck. WebArbiter notes that reasoning distillation inherits teacher hallucinations, brittle heuristics, and environment-specific shortcuts, with RL correcting but not fully eliminating them (Zhang et al., 29 Jan 2026). Orcust’s LLM-derived principle critiques are powerful but computationally expensive, and the framework remains dependent on simulated VMs and instrumented environments (Lu et al., 22 Sep 2025). OpenRubrics and CDRRM rely on synthetic rubrics generated by strong LLMs, then filtered for consistency; this improves reliability, but the principles may still reflect upstream biases or imperfect abstractions (Liu et al., 9 Oct 2025, Liu et al., 9 Mar 2026).

A second limitation concerns formal guarantees. Many principle-guided systems enforce constraints softly through prompts, structured outputs, SFT targets, or KL regularization rather than through hard executable guarantees. WebArbiter explicitly frames its principle constraints as “soft, representational constraints” induced through prompt design, structured outputs, and teacher traces (Zhang et al., 29 Jan 2026). Eval-Skill’s domain-level skills can substantially improve performance, but mixed-domain skills can confuse weaker backbones, and complex workflows may exceed the model’s instruction-following capacity (Yue et al., 5 Jun 2026). This suggests that explicit principles are necessary but not sufficient; capacity to follow them remains a separate variable.

A third issue is objective mismatch between local process quality and global outcome quality. PPR is explicit that optimizing raw process reward can diverge from final correctness, motivating ReNorm to enforce sign consistency between process and outcome (Xu et al., 29 Sep 2025). CRM addresses the same problem via probabilistic factorization and potential-based reward shaping, but it assumes an irrecoverable first wrong step, an approximation that may fail in domains with recovery or self-correction (Zhang et al., 30 Sep 2025). Constrained RLHF further shows that even when multiple reward components correspond to meaningful principles, the location of their useful operating region depends on correlation structure; individual proxy points are not sufficient for the composite setting (Moskovitz et al., 2023).

These limitations motivate several active directions. One direction is stronger formalization of principle sets and their relationship to task specifications, including explicit rule encoding, consistency losses, or multi-trajectory stability constraints (Zhang et al., 29 Jan 2026). Another is broader domain transfer: OpenRubrics, Eval-Skill, and CDRRM all show that domain-specific principles matter, but mixed-domain composition remains brittle (Liu et al., 9 Oct 2025, Yue et al., 5 Jun 2026, Liu et al., 9 Mar 2026). A third is integration with online RLHF and RLVR, where rubric- or principle-conditioned judges would provide process supervision or constraint signals directly to policy optimization (Xu et al., 29 Sep 2025, Liu et al., 9 Mar 2026). A plausible implication is that the field is moving toward a layered reward stack: explicit principles or skills for interpretability, probabilistic or CMDP-style constraints for objective control, and inference-time aggregation or evolution mechanisms for robustness under distribution shift (Moskovitz et al., 2023, Liu et al., 3 Apr 2025, Yue et al., 5 Jun 2026).

Under that broader interpretation, PCRM is less a single method than an organizing principle for modern reward modeling: reward should be grounded in explicit normative structure, and optimization should respect the limited validity, causal semantics, or procedural requirements of the signals being used.