Cause of train–test reward mismatch in RLER training

Investigate whether the observed discrepancy between high training rewards and lower downstream evaluation performance in DR Tulu results from mismatches between training-time tasks, rubric designs, and evaluation protocols, and characterize conditions under which such mismatches occur.

Background

During development, the authors observed that higher training rewards did not consistently translate to better external benchmark scores. They hypothesize that differences between training-time judges and benchmark judges, as well as rubrics and task setups, may lead to reward hacking or misalignment.

Validating this conjecture would inform how to align training and evaluation environments, including judge choice and rubric construction, to ensure that improvements measured during training predict external performance.

References

We conjecture that this stems from a mismatch between the tasks, rubrics, and evaluation setups of the external benchmarks vs.~what we used for training. We defer a deeper investigation of this mismatch phenomenon to future work, as addressing it would help improve the effectiveness of rubrics for RL training.

— DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research  (2511.19399 - Shao et al., 24 Nov 2025) in Discussion and Future Work, subsection "The train-test mismatch challenge"

A robot verifier is optimized against in the same way, and the mechanism therefore applies on the robot side, where the corresponding measurement has yet to be run.

— No Free Checker: A Survey of Verifiers for Robot Policies  (2609.09250 - Wan et al., 8 Sep 2026) in Section 7.2, subsection "Measuring a Verifier by the Policy It Trains"

We don't understand why this gap is larger, but are investigating this in future work.

— Debate Training Reduces Reward Hacking in RLAIF  (2608.17776 - Kenton et al., 18 Aug 2026) in Section 4, subsection “Debate reduces reward hacking,” paragraph “RL with an LLM judge has a smaller reward train/validation gap than RLVR”

The converse question, when an internal mismatch must incur nonvanishing task loss, remains open.

— How Wrong Can a Good Predictor Be? Diverging Updates with Vanishing Predictive KL  (2609.11132 - Wen et al., 10 Sep 2026) in Section Limitations

Two failure modes remain open. The first is a dense-gain fade. In some runs, the dense-inference gain peaks and then decays, while the reward of the routed training rollouts keeps rising. The generator then improves its routed transitions faster than its dense ones. Centered routed steps keep the rollouts near the dense path and reduce this split on Stable Diffusion 3.5, but the mechanism is not fully isolated. The second is reward hacking against the frozen preference model. This risk is general to online RL for generative models and is not specific to RPO. Both failure modes suggest that a scalar terminal reward is a coarse supervisor for a routed exploration action.

— Routing Policy Optimization for Online Diffusion Reinforcement Learning  (2610.22282 - Shen et al., 15 Oct 2026) in Appendix, Section "Extended discussion," paragraph "Known failure modes"