RRPO: Reference-Relative Policy Optimization with Stratified Conditional Rollouts

Abstract: Group Relative Policy Optimization (GRPO) has shown strong effectiveness in reinforcement learning from verifiable feedback, where sampled rollouts can be compared within a group using task-provided correctness signals. However, extending group-relative optimization beyond verifiable settings is challenging because success in many tasks is not captured by a single correctness criterion. We propose \textbf{Reference-Relative Policy Optimization (RRPO)}, which generalizes GRPO by replacing direct correctness-based advantage construction with reference-relative contrastive comparisons. RRPO first uses \emph{stratified conditional rollouts} to construct positive and negative anchor sets, and then trains a metric projection head with a set-contrastive objective to compare candidate rollouts against these anchors. The resulting alignment scores directly define contrastive advantages: during policy optimization, the projection head is frozen, and the scores are centered within each rollout group in a standard group-relative objective. We evaluate RRPO using anchor-based contrastive advantages throughout policy optimization, without relying on task ground-truth verifiers. Across verifiable reasoning, open-ended generation, and post-SFT settings, RRPO remains competitive with verifier-based optimization, improves over weakly supervised baselines, and provides additional gains after supervised fine-tuning.

• The paper introduces RRPO, a method that extends GRPO by leveraging stratified conditional rollouts and a contrastive framework to optimize policies in weak supervision settings.
• It precomputes positive and negative anchor sets based on weak conditions, enabling reference-relative scoring without relying on absolute correctness signals.
• Experimental results on GSM8K, ELI5, and SAMSum demonstrate that RRPO delivers competitive reasoning performance and stable improvements over baseline weak supervision methods.

"RRPO: Reference-Relative Policy Optimization with Stratified Conditional Rollouts" (2607.18470)

Introduction

The paper "RRPO: Reference-Relative Policy Optimization with Stratified Conditional Rollouts" proposes a method to extend Group Relative Policy Optimization (GRPO) into settings where success criteria are not strictly verifiable. While GRPO is effective in environments with task-provided correctness signals, it is limited when tasks like dialogue summarization or open-ended QA require judgment beyond binary correctness. The proposed Reference-Relative Policy Optimization (RRPO) addresses these limitations by constructing advantages based on stratified conditional rollouts. Positively and negatively stratified rollouts serve as reference anchors against which candidate rollouts are scored, thereby facilitating policy optimization in weakly supervised environments.

Methodology

RRPO introduces two main innovations: stratified conditional rollouts and a reference-relative contrastive learning framework. For any input $x$, RRPO generates positive and negative anchor sets with stratified rollouts, which serve as proxies for more and less desirable behaviors respectively. These anchor sets are not generated on the fly but are rather precomputed based on weak conditions specific to the task domain, and cached for repeated use during training. This structure promotes comparison-based scoring rather than relying on absolute correctness signals.

An offline-trained metric projection head is employed to evaluate the similarity of candidate rollouts to the anchor sets using a set-contrastive objective. This objective ensures that policy rollouts align more with positive anchors than with negative ones. During policy optimization, the projection head is kept static to produce consistent reference-relative alignment scores, thereby defining the basis for advantage computation.

Figure 1: Stratified conditional rollouts in RRPO. For each input, weak positive and negative conditions generate instance-specific anchor sets that emphasize desirable and undesirable reference behaviors. An offline-trained metric head compares policy rollouts against these anchors to produce reference-relative alignment scores.

The loss function comprising these reference-relative scores is used in a Dr.GRPO-style clipped objective, ensuring the advantages are centered within each group of rollouts. This maintains GRPO's core optimization principles while adapting it for weak supervision contexts.

Figure 2: Loss function calculation during training process

Experimental Evaluation

The effectiveness of RRPO was examined across several benchmark datasets, including GSM8K for verifiable reasoning, and ELI5 and SAMSum for open-ended generation tasks. The modeling framework spans multiple model scales, comparing RRPO against verifier-based methods like Dr.GRPO and weak supervision baselines such as Pseudo-SFT.

Verifiable Setting (GSM8K): RRPO demonstrated competitive performance with verifier-based methods without leveraging exact answers for training rewards. It outperformed both baseline models and Dr.GRPO on smaller-scale models across various pass@k metrics.

Open-Ended Setting (ELI5 and SAMSum): In these contexts, RRPO showed considerable stable improvements over Pseudo-SFT, indicating the robustness and effectiveness of reference-relative rollouts as a weak supervision signal. RRPO maintained or improved performance metrics like ROUGE and BLEU while achieving a more stable training process.

Figure 3: Learning curves during RRPO training. Evaluation performance improves over training steps on both SAMSum and GSM8K, indicating that the reference-relative training signal provides useful optimization feedback for open-ended generation and verifiable reasoning tasks.

Post-SFT Improvements: Additionally, RRPO applied to models already fine-tuned using supervised data (after SFT) showed modest but consistent gains, affirming that reference-relative advantages can complement conventional supervised fine-tuning processes.

Implications and Future Directions

RRPO's weakly supervised methodology broadens the applicability of policy optimization to tasks without direct verifier signals, such as long-form, context-driven tasks. This methodology provides a stable optimization framework that circumvents the need for absolute correctness criteria, fostering improvements in both reasoning tasks and creative open-ended generation.

Future developments could explore further refinement of weak condition formulations, the impact of larger model scales, and broader empirical validations across diverse datasets. Additionally, there is potential for extending RRPO to incorporate dynamic adjustments in anchor strategies or enhancing the scope of metrics training to adaptively refine the projection head while maintaining consistent reward evaluations.

Conclusion

RRPO provides a significant step forward in generalizing GRPO for weakly supervised tasks by emphasizing relative comparisons over absolute correctness. Its ability to deliver competitive results across both verifier-bound and verifier-free tasks illustrates the versatility and efficacy of reference-relative optimization, highlighting a promising avenue for further research and development in reinforcement learning paradigms for OpenAI models beyond classical rigid benchmarks.