---
title: Contrastive Rubric Generation (CRG) Algorithm
url: https://www.emergentmind.com/topics/contrastive-rubric-generation-crg-algorithm
type: topic
---

# Contrastive Rubric Generation (CRG) Algorithm

OpenRubrics is a modular system for scalable synthetic rubric generation, designed to support reward modeling and alignment in large language models (LLMs). The architecture is optimized to elicit, structure, filter, and utilize multi-criteria natural language rubrics, with the aim of replacing or augmenting scalar/pairwise evaluations that inadequately capture the multifaceted nature of response quality. The system demonstrates significant gains in both the accuracy of reward models and downstream policy alignment on diverse instruction-following and biomedical tasks, substantiating the principle-driven paradigm for rubric-based LLM alignment [2510.07743].

## 1. Dataset Construction Pipeline

OpenRubrics synthesizes its foundational dataset from heterogeneous open-ended dialogue and instruction-following corpora. Data sources include UltraFeedback (Evol-Instruct, UltraChat, ShareGPT, TruthfulQA), Tulu 2.5 (AlpacaFarm, Chatbot Arena, Capybara, StackExchange, Nectar, SHP, HH-RLHF, HelpSteer, Orca), HelpSteer 3, Skywork-Preference, Tulu 3-IF, MegaScience, and Medical-o1. 

Preference pairs are constructed through three primary mechanisms: selection of highest and lowest scoring responses (UltraFeedback), response generation and ranking using open-source reward models (Tulu3-IF/MegaScience/Medical-o1; Athene-RM-8B, Skywork-Reward-V2), and programmatic verifiability checks (verifiable-IF). These pairs undergo preprocessing steps that include discarding trivial or format-violating pairs, token truncation/canonicalization, and deduplication by prompt-response fingerprints. The result is a dataset $\mathcal{D}=\{(x_i,\hat y_i^+,\hat y_i^-,\ell_i)\}_{i=1}^N$, where $\ell_i \in\{+1,-1\}$ encodes the direction of preference.

## 2. Contrastive Rubric Generation (CRG)

To capture discriminative and interpretable evaluation signals, CRG operates by contrasting a chosen ($\hat y_i^+$) and a rejected ($\hat y_i^-$) response to the same prompt $x_i$, producing a set of rubric criteria. Each CRG invocation consists of (1) extracting non-negotiable hard rules from $x_i$, (2) identifying concrete differences between responses, and (3) abstracting those differences into domain-agnostic principles. A margin-based contrastive loss (optional) is used to encourage the generation of criteria that effectively separate positive from negative responses:
\[
\mathcal L_{\mathrm{CRG}} = \sum_{i=1}^N\sum_{j=1}^{K_i} \left[ -\log\sigma\big(s_\psi(c_{i,j},x_i,\hat y_i^+) - s_\psi(c_{i,j},x_i,\hat y_i^-)\big) \right]
\]
where $s_\psi(c,x,y)$ denotes the compatibility score between criterion and response. The generator backbone is a pretrained instruction-tuned LLM $h_\psi$.

## 3. Rejection Sampling for Preference-Label Consistency

To enforce rubric reliability, OpenRubrics implements rejection sampling based on preference-label consistency. Each rubric $\mathcal R(x_i)$ generated from a triplet $(x_i, \hat y_i^+, \hat y_i^-, \ell_i)$ is retained only if the generator $h_\psi$ correctly predicts the original preference label $\ell_i$ when given the rubric as context. This filtering yields a reliable set $\mathcal D_{\mathrm{rubric}} = \{(x_i,\hat y^+_i,\hat y^-_i,\mathcal R^*(x_i))\}_{i=1}^M, M \le N$.

## 4. Rubric-RM Reward Model Architecture

OpenRubrics centers its modeling around two supervised modules:

**Rubric Generator $g_\theta$:**  
— Backbone: Qwen-3 (4B/8B)  
— Input: "<PROMPT>…<RESPONSE+>…<RESPONSE−>…"  
— Output: Structured rubric $\mathcal R(x)$.  
— Training: Next-token cross-entropy  
\[
\mathcal L_{\mathrm{SFT}^{\mathrm{rubric}}} = -\mathbb E_{(x,y^+,y^-,\mathcal R^*)}\sum_{t=1}^{|\mathcal R^*|}\log p_\theta(\mathcal R^*_t | x, y^+, y^-, \mathcal R^*_{<t})
\]

**Rubric-Conditioned Judge $r_\phi$:**  
— Backbone: Qwen-3 (same scale)  
— Input: $[x;y^+;y^-;\mathcal R(x)]$  
— Output: Label in $\{\text{“A is better”}, \text{“B is better”}\}$  
— Training: Next-token cross-entropy  
\[
\mathcal L_{\mathrm{SFT}^{\mathrm{rm}}} = -\mathbb E_{(x,y^+,y^-,R^*,\ell)}\sum_{t=1}^{|\ell|}\log p_\phi(\ell_t | x, y^+, y^-, R^*, \ell_{<t})
\]

Key hyperparameters for Rubric-RM-8B include batch size 64, learning rate $5\times10^{-6}$, number of epochs 2, and $6144$ maximum tokens.

## 5. End-to-End Integration and Inference Workflow

The complete OpenRubrics workflow is as follows:
1. Triplet collection: $\{(x,\hat y^+,\hat y^-,\ell)\}$.
2. CRG plus rejection sampling yields reliable rubrics $\mathcal R^*(x)$.
3. Supervised fine-tuning of $g_\theta$ yields a rubric generator.
4. Supervised fine-tuning of $r_\phi$ yields a rubric-conditioned judge.
5. For new input $(x, y^A, y^B)$, $g_\theta$ generates a rubric $\hat{\mathcal R}(x)$, then $r_\phi$ predicts which of $y^A$ or $y^B$ is preferable given $\hat{\mathcal R}(x)$.

## 6. Scalability, Efficiency, and Benchmarks

Rubric-RM displays competitive performance across reward-modeling and downstream alignment tasks. On RewardBench, RM-Bench, IFBench, Rubric-RM-4B achieves 65.6% accuracy, Rubric-RM-8B reaches 68.5%, and Rubric-RM-8B-voting@5 attains 71.2%, approaching larger proprietary models. In policy fine-tuning (DPO), Rubric-RM leads to +3–4 points improvement over standard RLHF policies on IFEval and InfoBench, and achieves top open-source win rates (50–57%) on Arena-Hard and AlpacaEval. For biomedical (HealthBench), Rubric-RM-8B yields 68.3% versus RRM-7B's 63.3%. In large-scale scoring, Rubric-RM-8B is substantially faster (130s for 100 pairs) than RRM-7B (203s) and RM-R1-14B (322–382s), with rubric amortization enabling further acceleration.

## 7. Significance and Implications

The OpenRubrics architecture synthesizes contrastively generated, consistency-filtered rubrics as interpretable, multi-criterion scaffolds for reward modeling. This paradigm narrows the gap between human evaluation and automated RLHF, offering scalable and principle-driven alignment for LLMs. A plausible implication is that rubrics provide more granular, transparent reward signals, advancing beyond scalar/pairwise judges and enabling reliable deployment in both generalist and domain-specific (e.g., biomedical) contexts [2510.07743].

Source: https://www.emergentmind.com/topics/contrastive-rubric-generation-crg-algorithm