Whether and how RLRR alleviates reward over-optimization

Determine whether and how reinforcement learning from rubrics-based reward (RLRR) mitigates reward over-optimization in large language model post-training, where reward over-optimization refers to a policy exploiting misspecified proxy rewards to achieve high scores while true output quality deteriorates.

Background

The paper studies rubric-based rewards as an approach to post-training LLMs and contrasts them with traditional Bradley–Terry preference-based reward models, which are known to suffer from reward over-optimization. In RLRR, each prompt is associated with explicit criteria and weights, and a verifier assesses whether a response satisfies each criterion, producing a weighted reward.

Despite growing interest in RLRR, the authors note that it is not yet established whether rubric-based rewards actually alleviate the reward over-optimization phenomenon in practice, motivating their theoretical and empirical investigation into high-reward region accuracy and rubric construction methods.