Extract useful training signal from dropped rollouts

Determine whether dropped rollouts excluded by RaRE’s drop-on-disagreement mask can provide a reusable training signal through methods more effective than group-mean reward substitution.

Background

RaRE removes high-disagreement rollouts from the GRPO update by substituting the unmasked group mean, thereby assigning them zero normalized advantage. The paper reports that an exploratory retrospective off-policy GRPO variant did not improve over the baseline and that no reusable signal was extracted from the dropped rollouts, but explicitly leaves further investigation unresolved.

References

When applying RaRE's drop-on-disagreement mask, we discard the dropped rollouts entirely rather than extracting any partial signal from them. Initial exploratory attempts in this direction (Appendix~\ref{app:retro}) did not yield any improvement over our baseline methods, but further exploration is left to future work.

— Judging the Judges: Confidence and Disagreement in Rubrics as Rewards  (2610.21179 - Liu et al., 15 Oct 2026) in Section Limitations; Appendix, Section Retro-GRPO-aux