When and how to incorporate additional Recursive Reward Modeling steps beyond critique

Ascertain the appropriate point at which to introduce further steps of Recursive Reward Modeling (RRM) beyond a single critique stage for supervising large language models and determine whether the critique method can be effectively used within RRM.

Background

The authors position critique as a first step toward scalable oversight and relate it to Recursive Reward Modeling (RRM), which iteratively decomposes evaluation and supervision tasks. They explicitly state uncertainty about when to add additional RRM steps and whether critique is suitable for RRM, highlighting a gap in guidance for multi-step oversight workflows.

Resolving this question would inform the design of scalable oversight systems that leverage critiques as building blocks within RRM, potentially improving training signal quality for advanced models.