Effectiveness of GRPODropout with very small rollout groups

Evaluate the effectiveness of GRPODropout when policy updates use very small groups of sampled rollouts, including whether its reliance on within-prompt differences in advantages and probabilities remains beneficial in that regime.

Background

GRPODropout selects and recenters rollouts using differences in their advantages and old-policy probabilities within each prompt. Consequently, the method is designed for multi-rollout groups and does not directly apply to single-rollout training.

The paper explicitly leaves unresolved how the method behaves when the group contains very few rollouts. This is important because smaller groups provide less within-prompt variation for estimating entropy-related contributions and for making deletion decisions, potentially affecting both the method’s validity and its empirical benefits.

References

Finally, GRPODropout relies on within-prompt differences in advantages and probabilities across multiple rollouts. It therefore does not directly apply to single-rollout training, while its effectiveness with very small groups remains to be evaluated.

— GRPODropout: Less is More for Online Reinforcement Learning Rollouts  (2610.11854 - Deng et al., 8 Oct 2026) in Section 5, Conclusion, paragraph “Limitations”