Distilled Generative Reward Model

• The model integrates generative reward prediction with explicit rationale synthesis, enabling interpretable label decisions.
• A self-training loop augments labeled data with pseudo-labels refined via a preference-proving module using Bayesian selection.
• Empirical evaluations show improved ranking accuracy and robustness in RLHF, supporting broad downstream applications.

A distilled generative reward model is a paradigm in which generative models—often with explicit probabilistic structure—produce reward signals or reward-aligned outputs, frequently incorporating explicit reasoning or structured rationales. These models are trained not only to assign (or maximize) preference labels but also to synthesize interpretable reward rationales, leveraging self-training methodologies that exploit large-scale unlabeled data. The result is a foundation reward model capable of generalizing across a wide spectrum of tasks and supporting various downstream applications such as response ranking and reinforcement learning from human feedback (RLHF). The following sections provide an in-depth analysis of the GRAM-R² framework as presented in "GRAM-R$^2$: Self-Training Generative Foundation Reward Models for Reward Reasoning" (Wang et al., 2 Sep 2025).

1. Architectural Overview and Self-Training Paradigm

GRAM-R² comprises two tightly integrated modules:

• Generative Reward Model: Given a prompt, this model generates both a preference label (indicating, for instance, which of two responses is superior according to implicit or explicit preference signals) and a detailed reward rationale. The rationale elucidates the reasoning underlying the label decision.
• Preference-Proving Model: Trained on a modest corpus of rationale-annotated data, this auxiliary model converts rationale-free labeled data into structured preference proofs (rationales), enabling the augmentation of existing datasets with synthesized explanations.

The core training loop adheres to a self-training scheme:

1. Start from a model initially trained using available labeled data (with synthesized rationales, if necessary).
2. Use the generative reward model to generate pseudo-labels (labels plus rationales) for a vast pool of unlabeled data.
3. Apply the preference-proving model to enhance the generated rationales, constructing higher-quality proofs from initially rationale-free (pseudo-)labeled data.
4. Aggregate these enhanced pseudo-labeled samples with the labeled set and perform further training, thereby progressively reinforcing the model's reward reasoning ability.

This iterative process allows GRAM-R² to substantially grow its training data and to iteratively improve both label accuracy and rationale fidelity by mining and refining annotations from unlabeled corpora.

2. Generative Objective and Modeling Details

The loss for the generative reward model incorporates explicit rationale generation:

$$\mathcal{L}_g = - \mathbb{E}_{(c,x,y_a,y_b,l,z)\sim D_p} \Bigl[ \log \pi_\phi(z|s) + \log \pi_\phi(w=l|s,z) \Bigr]$$

where:

• $c$ is the (optional) context or task description,
• $x$ is the user prompt,
• $y_a, y_b$ are candidate responses for comparison,
• $l$ is the preference label,
• $z$ is the reward rationale (i.e., a structured explanation or proof),
• $\pi_\phi$ denotes the generative model’s conditional distribution,
• $s$ encodes the (prompt, candidates) tuple, and
• $w$ is the preference label token.

This two-stage generation—first producing a rationale, then a label conditioned on the rationale—enables the model to represent reasoning dependencies and makes upstream reward modeling more interpretable and flexible.

3. Preference-Proof Selection via Bayesian Reasoning

To ensure rationale quality, the preference-proving module implements a selection scheme that is theoretically grounded in Bayesian estimation. For a given input $s$ and label $l$, the best proof $\hat{z}$ is chosen to maximize:

$$\log \pi_\psi(\hat{z}|s,l) - \log \pi_\psi(\hat{z})$$

where $\pi_\psi$ is the probability under a parametric model initialized or fine-tuned on rationale-annotated data. This form penalizes generic, uninformative rationales (common in large models) and promotes informative, label-relevant explanations.

These best proofs are then incorporated into the training set in subsequent iterations, yielding further improvements in rationale quality and in downstream preference modeling.

4. Empirical Performance and Generalization Capabilities

GRAM-R² is designed to support a variety of downstream tasks:

• Response Ranking (Pairwise and Listwise): The generative framework, through both direct scoring and synthesized rationales, produces superior ranking accuracies compared to discriminative and earlier generative baselines.
• Task Adaptation: Applications in specialized domains, e.g., STEM or code generation, are enabled by incorporating small in-domain annotated datasets with high-quality rationales, allowing the model to adapt with only minimal additional supervision.
• RLHF/Alignment: As a reward model, GRAM-R² is used for downstream reinforcement learning, where the combined label-plus-rationale outputs help mitigate common failure modes such as overoptimization, i.e., the model overfitting to reward signals without understanding underlying task criteria.

Empirical results on benchmarks such as RM-Bench and JudgeBench consistently show that GRAM-R² outperforms strong discriminative and generative baselines, providing both higher ranking accuracy and increased robustness against reward-overoptimization artifacts.

5. Comparison to Other Paradigms and Methodological Implications

GRAM-R² fundamentally extends earlier work by explicitly combining preference prediction and reward rationale formation—an architectural distinction from standard discriminative reward models, which focus on scalar predictions alone. Unlike models with only supervised fine-tuning from limited labeled data, self-training on unlabeled corpora with explicit rationale synthesis enables stronger generalization and data efficiency.

The approach also explicitly addresses and mitigates limitations in overoptimization: the rationale-centric training provides a richer signal for policy alignment, preventing the model from exploiting shallow or adversarial solution spaces.

A summary table of comparative features is presented below:

Model Class	Preference Label	Rationale Generation	Self-Training on Unlabeled Data	Application Paradigms
Discriminative RM	✔	✗	✗	Ranking, RLHF
Early Generative RM	✔	Limited/✗	✗	Ranking, RLHF
GRAM-R²	✔	✔	✔	Ranking, RLHF, Task Adaptation

6. Limitations and Future Directions

While GRAM-R² sets a new standard for reward reasoning in generative models, several extensions are suggested:

• Scaling to Larger and More Diverse Unlabeled Data: Additional unlabeled datasets could further enhance generalization capabilities and allow more domain-specific adaptation.
• Preference Proof Refinement: Improvements in the proof selection criterion (e.g., leveraging better uncertainty quantification or richer reference proof corpora) could increase rationale informativeness and fidelity.
• Multi-task and Multi-modal Extensions: Exploring integration with multi-task reward tuning and adaptation to multi-modal input spaces represents a natural direction for expanding the impact and applicability of distilled generative reward models.
• Alignment Robustness: Continued research is required to more deeply address edge cases and guard against degenerate policy behaviors that may arise in poorly defined or ambiguous feedback settings.

7. Significance and Broader Implications

GRAM-R² exemplifies the maturation of distilled generative reward modeling: producing both task-aligned preference signals and explicit reward rationales via a self-training generative loop. This development supports the emergence of reward foundation models—generalist, interpretable reward models that can be rapidly adapted for a broad spectrum of alignment, ranking, and RLHF applications with minimal additional supervision, thus advancing both the practical and theoretical frontiers of alignment methodology. The explicit separation of reward reasoning from pure label prediction reduces the risk of reward hacking and enhances transparency, providing critical scaffolding for the next generation of aligned AI systems.