Contrastive Rubric Generation (CRG)

• Contrastive Rubric Generation (CRG) is a method that leverages contrastive signals from paired model outputs to generate structured rubrics with both hard rules and implicit principles.
• It overcomes traditional scalar and pairwise reward modeling limitations by offering interpretable, multi-dimensional criteria that enhance language model alignment.
• CRG uses rejection sampling to ensure rubric consistency, with improved accuracy validated across benchmarks such as RewardBench and HealthBench.

Contrastive Rubric Generation (CRG) is a methodology for scalable rubric synthesis in reward modeling, introduced in the OpenRubrics framework. It leverages contrastive signals from paired model outputs—specifically, “preferred” and “rejected” responses—to generate structured, multi-aspect rubrics comprising both explicit hard rules and implicit principles. This approach addresses limitations of traditional scalar or pairwise preference-based reward models by promoting richer, interpretable, and principle-driven alignment signals for LLMs (Liu et al., 9 Oct 2025).

1. Motivation and Problem Formulation

Traditional reward modeling in reinforcement learning from human feedback (RLHF) is predominantly based on pointwise scalar scores $r(x, \hat{y})$ or pairwise preferences between outputs $\hat{y}^+$ and $\hat{y}^-$. Such supervision paradigms inevitably collapse multifaceted quality criteria—such as factuality, style, or adherence to instructions—into a single undifferentiated number, thereby obscuring which aspects of the output drive preference. The “Rubrics-as-Rewards” (RaR) paradigm proposes instead to evaluate outputs against $k$ explicit, human-readable criteria: $\mathcal{R}(x) = \{c_1, \ldots, c_k\},$ where each $c_i$ denotes a distinct evaluation dimension. Generating such rubrics at scale, reliably and without extensive manual authoring, is a core challenge that CRG aims to solve (Liu et al., 9 Oct 2025).

2. Mathematical Formalism of CRG

CRG operates over a dataset of annotated preference tuples

$$\mathcal{D} = \left\{(x_i, \hat{y}_i^+, \hat{y}_i^-, \ell_i)\right\}_{i=1}^N, \quad \ell_i \in \{"\text{+ preferred}", "\text{– preferred}"\},$$

where $x_i$ is a prompt, $\hat{y}_i^+$ and $\hat{y}_i^-$ are respective preferred and rejected responses, and $\ell_i$ records the preference label.

A pre-trained LLM $h_\psi$ is enlisted to synthesize a rubric conditioned on the tuple: $$\mathcal{R}(x_i) \sim h_\psi(x_i, \hat{y}_i^+, \hat{y}_i^-, \ell_i),$$ where the resulting rubric $\mathcal{R}(x_i) = \{c_{i,1}, \ldots, c_{i,k_i}\}$ is partitioned into: $$\underbrace{\{c_{i,j}\}_{j\in\mathcal{H}_i}}_{\text{Hard Rules}} \;\cup\; \underbrace{\{c_{i,j}\}_{j\in\mathcal{P}_i}}_{\text{Principles}}.$$

• Hard rules encode explicit constraints (e.g., length, required elements) from the prompt.
• Principles capture implicit qualitative distinctions justifying the preferred response (e.g., clarity, creativity).

To ensure rubrics are consistent with human preference, CRG applies a rejection sampling step: a second LLM call predicts which response is preferred under the rubric,

$$\widehat{\ell}_i = h_\psi(x_i, \mathcal{R}(x_i), \hat{y}_i^+, \hat{y}_i^-),$$

retaining only those rubric–tuple pairs for which $\widehat{\ell}_i = \ell_i$. The final corpus is: $$\mathcal{D}_\text{rubric} = \left\{ (x_i, \hat{y}_i^+, \hat{y}_i^-, \mathcal{R}^*(x_i)) \right\}_{i=1}^M,$$ with

$$\mathcal{R}^*(x_i) = \begin{cases} \mathcal{R}(x_i) & \text{if } \widehat{\ell}_i = \ell_i, \ \emptyset & \text{otherwise}. \end{cases}$$

3. Algorithmic Structure

The CRG workflow comprises three principal stages. The process is formalized as follows:

for (x, y^+, y^-, ℓ) in D: R = SAMPLE_RUBRIC(h_ψ, x, y^+, y^-, ℓ) ℓ̂ = PREDICT_LABEL(h_ψ, x, R, y^+, y^-) if ℓ̂ == ℓ: keep (x, y^+, y^-, R) in D_rubric else: discard Train g_θ on D_rubric with cross-entropy: L_SFT^rubric = –E_{(x,y^+,y^-,R)} ∑_{t} log p_θ(R_t | x, y^+, y^-, R_{<t}) Build D_rm = {(x, y^+, y^-, R, ℓ)} using R from D_rubric Train r_φ with: L_SFT^rm = –E_{(x,y^+,y^-,R,ℓ)} ∑_{t} log p_φ(ℓ_t | x, y^+, y^-, R, ℓ_{<t}) Given (x, y^A, y^B): R̂ = g_θ(x, y^A, y^B) l̂ = argmax_{label∈{A,B}} p_φ(label | x, y^A, y^B, R̂) return l̂

Rejection sampling in Stage 1 implements the preference–label consistency filter, ensuring robust downstream supervision (Liu et al., 9 Oct 2025).

4. Rubric Anatomy and Illustrative Instances

CRG-generated rubrics manifest as numbered lists, each entry formatted “The response ... [Hard Rule/Principle].” Hard rules function as strict gatekeepers (e.g., “written in fewer than two paragraphs”), while principles represent generalized, subjective aspects such as “employs sensory details” or “demonstrates originality.”

Examples:

• RewardBench Chat-Hard:

1. The response uses strong imagery and creative language to create a vivid and unique character description. [Hard Rule]
2. The response is written in fewer than two paragraphs. [Hard Rule]
3. The response presents distinctive and memorable traits. [Principle]
4. The response employs sensory details to enhance the reader’s mental image. [Principle]
5. The response demonstrates originality to avoid clichés. [Principle]
6. The response balances detail and conciseness. [Principle]

• FollowBench:

1. The response must incorporate a quote from a recent news article or paper. [Hard Rule]
2. The response must mention the publication date of the referenced source. [Hard Rule]
3. The response must concisely summarize the quoted source. [Hard Rule]
4. The response must discuss economic implications based on the source. [Hard Rule]
5. The response is written in a clear and understandable manner. [Principle]
6. The response is well-organized and easy to follow. [Principle]

Explicit labeling of criteria enables downstream reward models to enforce hard constraints prior to evaluating under softer, principle-based axes.

5. Experimental Results and Benchmarks

CRG and the consequent Rubric-RM reward models were evaluated on a comprehensive slate of benchmarks: RewardBench (Chat, Chat-Hard), RM-Bench, FollowBench, InfoBench, IFBench, RewardBench2 (Precise-IF, Focus), and HealthBench. The principal evaluation metric is pairwise win-rate accuracy.

Model Variant	Average Accuracy (%)	Comparative Baseline Range (%)
Rubric-RM-4B	65.6	53.8–61.7
Rubric-RM-8B	68.5	baseline +6.8 points
Rubric-RM-8B-voting@5	71.2	~14B model equivalence

Further results indicate:

• Policy fine-tuning with Direct Preference Optimization (DPO) using Rubric-RM as the reward results in an average gain of 2.9 percentage points on IFEval/InfoBench.
• On HealthBench, Rubric-RM achieves downstream policy performance of 23.8 versus 22.7–22.5 for alternative judges.
• Ablation studies: removing the preference–label consistency filter reduced reward model accuracy by approximately 2–3 percentage points; omission of either principles or hard rules degrades downstream policy performance by up to 4 percentage points.

A naïve pipeline (“prompt→rubric then judge” without fine-tuning) achieved only 58.9% accuracy, underscoring the necessity of rubric consistency filtering and end-to-end reward model tuning.

6. Significance, Limitations, and Future Directions

CRG constitutes a principled approach that bridges the gap between expensive, manually-authored rubrics and oversimplified scalar rewards, yielding interpretable and scalable supervision signals for RLHF. Hard rules enforce strict adherence to explicit instructions, reducing common model faults such as verbosity bias or hallucinated references. Principles enable richer, multi-facet judgments that guide policy models towards high-level, generalizable qualities (Liu et al., 9 Oct 2025).

The two-stage, contrastive methodology is robust to correlational noise in LLM outputs via rejection sampling. Rubrics can be precomputed and cached to amortize synthesis cost.

Notable limitations include the reliance on a high-quality LLM $h_\psi$ for rubric synthesis and an exclusive focus on pairwise preferences; generalization to $n$-way output comparisons remains an open question. Integrating rubrics in-the-loop during RLHF, rather than as a post-hoc judge, is identified as a key future direction for enabling principle-driven policy exploration. Human calibration or semi-automated refinement of rubrics also remains an area for further investigation.

7. Context within LLM Alignment and Reward Modeling

CRG exemplifies an evolution in reward modeling strategies for LLM alignment—moving from opaque, collapsed scalar and pairwise scores to interpretable, structured supervision. Compared to scalar reward models, CRG’s rubric-based models (“Rubric-RM”) offer substantially improved accuracy (e.g., +6.8 points over size-matched baselines) and transfer gains to downstream policy fine-tuning. This suggests potential for narrowing the gap between automated reward modeling and gold-standard human evaluation, catalyzing a principle-oriented alignment paradigm for future LLM development (Liu et al., 9 Oct 2025).