R3: Robust Rubric-Agnostic Reward Models (2505.13388v2)

Abstract: Reward models are essential for aligning LLM outputs with human preferences, yet existing approaches often lack both controllability and interpretability. These models are typically optimized for narrow objectives, limiting their generalizability to broader downstream tasks. Moreover, their scalar outputs are difficult to interpret without contextual reasoning. To address these limitations, we introduce R3, a novel reward modeling framework that is rubric-agnostic, generalizable across evaluation dimensions, and provides interpretable, reasoned score assignments. R3 enables more transparent and flexible evaluation of LLMs, supporting robust alignment with diverse human values and use cases. Our models, data, and code are available as open source at https://github.com/rubricreward/r3

Analysis of $black$: Robust Rubric-Agnostic Reward Models

In the field of LLM evaluation, reward models are crucial for aligning outputs with human preferences by assigning scalar scores to responses. However, current approaches often suffer from limitations in controllability and interpretability due to optimization over narrow objectives and opaque scalar outputs. This paper introduces a novel framework, $black$, designed to overcome these difficulties by being rubric-agnostic, generalizable across various evaluation dimensions, and capable of providing interpretable score assignments with explicit reasoning.

Overview

$black$ proposes a task-agnostic framework that utilizes fine-grained rubrics to facilitate controllable and interpretable reward evaluations. These rubrics can be handcrafted by humans or generated by LLMs, enhancing transparency and flexibility in evaluation. The framework supports diverse data types and task formats, including point-wise, pair-wise, and binary evaluations, enabling robust alignment with human values across various use cases.

Dataset Curation and Model Training

The paper outlines a meticulous process for dataset curation involving initial data gathering from publicly available sources, diversity sampling, and quality filtering stages. The curated dataset comprises 4K and 14K examples enriched with rubrics and explanation traces, facilitating efficient training. Training is conducted using both full fine-tuning and compute-efficient techniques like LoRA, ensuring scalable deployment across different model sizes.

Experimental Results

The paper presents comprehensive evaluations on several benchmarks, including RM-Bench, RewardBench, FeedbackBench, BBH, and MMLU-STEM. $black$ models consistently outperform existing reward models across these benchmarks, demonstrating robust and versatile performance. Particularly notable is the success of smaller $black$ models, which achieve competitive results under stringent resource constraints, emphasizing the efficacy of the framework and dataset.

Implications and Future Directions

The rubric-agnostic nature of $black$ marks a significant step towards more transparent and adaptable reward models, enhancing trust in evaluations across diverse applications. The method's demonstrated scalability and effectiveness under resource limitations suggest potential for broader deployment in real-world scenarios. Future research could explore integrating $black$ as a reinforcement learning signal for improving model performance, further solidifying its role in advancing AI alignment.

Conclusion

This paper highlights the innovative approach of the $black$ framework in addressing the inherent limitations of existing reward models. By allowing for rubric-free evaluation grounded in interpretable reasoning, $black$ sets a new standard for generality and transparency in reward modeling, paving the way for more trustworthy interaction between LLMs and human users.