---
title: OpenRubrics Dataset Overview
url: https://www.emergentmind.com/topics/openrubrics-dataset
type: topic
---

# OpenRubrics Dataset Overview

OpenRubrics is a large-scale, diverse dataset for scalable synthetic rubric generation, developed to address the limitations of scalar or pairwise judgments in reward modeling for reinforcement learning from human feedback (RLHF). By providing a corpus of structured, multi-dimensional rubrics aligned to various prompt domains, OpenRubrics enables the training and evaluation of rubric-based reward models, with demonstrable gains in alignment and downstream policy optimization for large language models (LLMs) [2510.07743].

## 1. Dataset Composition and Statistics

OpenRubrics comprises 53,398 unique prompt–rubric pairs. The dataset’s domain distribution is as follows:

| Domain                | # Prompt–Rubric Pairs | Percentage    |
|-----------------------|----------------------|--------------|
| Instruction-following | 23,184               | 43.4%        |
| Biomedical            | 12,817               | 24.0%        |
| Open-domain QA        | 10,560               | 19.8%        |
| Coding/Math           | 6,837                | 12.8%        |

Rubrics are decomposed into two types:
- **Hard rules:** Explicit, often verifiable constraints (e.g., “The response must use only information present in the passage”).
- **Principles:** Implicit, qualitative criteria (e.g., “The response should demonstrate clarity and logical structure”).

Each rubric specifies between 3 and 8 dimensions (mean = 5.2 per rubric), with domains such as biomedical and coding exhibiting more multi-faceted rubrics than general instruction-following tasks.

## 2. Data Format and Access

All OpenRubrics data is released in JSON format. Each sample consists of a `prompt` field, a `rubric` list, and associated metadata. The file structure is standardized as follows:

```json
{
  "prompt": "<input prompt>",
  "domain": "<domain label>",
  "rubric": [
    {
      "type": "hard_rule" | "principle",
      "description": "<criterion text>"
    },
    ...
  ],
  "metadata": {
    "rubric_dimensions": <int>,
    "source": "synthetic" | "human-authored",
    "preference_label_consistency": <float>
  }
}
```

- **JSON fields:** `prompt`, `domain`, `rubric` (list of objects with `type` and `description`), `metadata` (containing `rubric_dimensions`, `source`, `preference_label_consistency`).
- **Repository & access:** The dataset and related code are available under the CC BY 4.0 License at https://github.com/OpenRubrics/OpenRubrics and https://huggingface.co/datasets/OpenRubrics/OpenRubrics.
- **Licensing:** CC BY 4.0 (academic and commercial use; attribution required).

## 3. Generation Methodology

OpenRubrics employs the Contrastive Rubric Generation (CRG) framework. CRG decomposes rubric synthesis into the following steps:

- **Response Contrasting:** For each prompt, a preferred and a rejected response are selected (using preexisting preference datasets or synthetic LLM outputs).
- **Component Extraction:** CRG prompts an LLM to analyze the contrast between preferred and rejected responses to explicitly enumerate both:
  - **Hard rules** (directly violated in the rejected response but satisfied in the preferred one)
  - **Principles** (qualities more subtly present or absent, e.g., relevance, informativeness)
- **Rubric Assembly:** The set of hard rules and principles is combined to form a comprehensive multi-dimensional rubric for the given prompt.

The CRG loss function is defined as:

\[
\mathcal{L}_{\text{CRG}} = \mathbb{E}_{(x, y^+, y^-)} \left[ - \sum_{k=1}^D \log P( r_k^+ > r_k^- \mid x ) \right]
\]

where $x$ is the prompt, $y^+$/$y^-$ are the preferred/rejected responses, $r_k^+$/$r_k^-$ represent rubric satisfaction on dimension $k$, and $D$ is the number of rubric dimensions per pair.

## 4. Quality Control and Reliability

To maximize the reliability of rubrics and prevent alignment drift or ambiguity, OpenRubrics employs a preference-label consistency framework:

- **Preference-Label Consistency:** For each generated rubric, agreement is measured between the relative scorings of preferred and rejected responses. This is formalized as:

\[
\text{Consistency}(r) = \frac{1}{N} \sum_{i=1}^N \mathbb{I}\left[ r(x_i, y_i^+) > r(x_i, y_i^-) \right]
\]

where $r(\cdot,\cdot)$ is the rubric-based score function, and $N$ is the number of evaluated prompt–response pairs.

- **Rejection Sampling:** Rubrics exhibiting preference-label consistency under a threshold (set at 95%) are filtered out, enforcing that the induced scoring function reliably prefers designated preferred responses.

This results in a measured label consistency of 98.2% across the dataset, with detailed rejection logs released as part of the metadata.

## 5. Benchmarking and Applications

OpenRubrics is used to train Rubric-RM, a rubric-based reward model designed for reward modeling and LLM alignment. The following are key results:

- **Reward-Modeling Benchmarks:** Rubric-RM trained on OpenRubrics demonstrates a mean improvement of 6.8% over strong size-matched baselines on standard preference and reward modeling benchmarks.
- **Alignment Improvement:** Rubric-based signals enable model alignment with nuanced human-like standards, outperforming scalar judgment regimes.
- **Transfer to Policy Models:** Rubric-RM is incorporated for policy fine-tuning by using rubric-derived rewards in reinforcement learning protocols.

Performance transfer results include:
- **Instruction-Following Evaluation:** On instruction-following benchmarks, Rubric-RM aligned models achieve a 4.2% absolute gain in human agreement metrics over scalar reward modeling baselines.
- **Biomedical QA:** In biomedical answer generation, rubric-trained models register a 9.5% incremental improvement in F1-based utility, evidencing rubric-derived rewards' efficacy in specialized domains.

## 6. Example Entries

Representative prompt–rubric pairs from OpenRubrics illustrate the diversity and granularity of alignment signals:

**Example 1:** (Instruction-Following)
```json
{
  "prompt": "Summarize the key implications of the given research abstract.",
  "domain": "instruction-following",
  "rubric": [
    {"type": "hard_rule", "description": "Do not introduce information not present in the abstract."},
    {"type": "principle", "description": "Summary must prioritize main contributions over background details."},
    {"type": "principle", "description": "Explanation should use precise, field-appropriate terminology."},
    {"type": "principle", "description": "Response should be concise, no longer than 3 sentences."}
  ],
  "metadata": {"rubric_dimensions": 4, "source": "synthetic", "preference_label_consistency": 1.0}
}
```

**Example 2:** (Biomedical)
```json
{
  "prompt": "What are potential differential diagnoses for a 45-year-old with chest pain and shortness of breath?",
  "domain": "biomedical",
  "rubric": [
    {"type": "hard_rule", "description": "All listed diagnoses must be medically plausible given the symptoms."},
    {"type": "hard_rule", "description": "Do not suggest over-the-counter remedies as primary management."},
    {"type": "principle", "description": "Responses should cover both cardiac and non-cardiac etiologies."},
    {"type": "principle", "description": "Order differential by acuity and probability."},
    {"type": "principle", "description": "Mention red-flag symptoms justifying urgent evaluation."}
  ],
  "metadata": {"rubric_dimensions": 5, "source": "synthetic", "preference_label_consistency": 0.98}
}
```

**Example 3:** (Coding/Math)
```json
{
  "prompt": "Given the following function, explain its computational complexity.",
  "domain": "coding/math",
  "rubric": [
    {"type": "hard_rule", "description": "Explanation must specify both time and space complexity."},
    {"type": "principle", "description": "Provide concrete examples illustrating complexity bounds."},
    {"type": "principle", "description": "Relate complexity to relevant data structures, if any."}
  ],
  "metadata": {"rubric_dimensions": 3, "source": "synthetic", "preference_label_consistency": 1.0}
}
```

## References

- "OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment" [2510.07743]

Source: https://www.emergentmind.com/topics/openrubrics-dataset