Effect of alternative OPD objectives on pass@k coverage and RL interaction

Investigate whether alternative on-policy distillation objectives—including reverse KL over the full next-token distribution or a top-$K$ truncation, forward KL, and Jensen–Shannon divergence—expand pass@$k$ coverage more effectively than the token-level reverse-KL objective and determine how the resulting cold starts interact with subsequent reinforcement learning.

Background

The OPD stage in the study uses an on-policy reverse-KL objective estimated token by token on student-generated rollouts. The authors point out that other divergence choices can differ in mode coverage and generation diversity, potentially producing different initializations for the later RL stage.

The unresolved questions concern both the capability-expansion effect of these alternatives—specifically whether they improve pass@kk coverage more effectively—and the downstream consequences for RL when each alternative supplies the cold start. The paper explicitly characterizes these as open questions.

References

Our analysis does not independently vary teacher distributional alignment and retained uncertainty, leaving their individual contributions and interaction unresolved.

— RL Starts before RL: On Policy Distillation for Better Reinforcement Learning  (2609.28145 - Dong et al., 23 Sep 2026) in Section 6, “Discussion and Limitations”

A natural direction for future work is to investigate alternative OPD objectives that may yield a different cold-start for RL, including reverse KL evaluated over the full next-token distribution (or top-$K$), as well as forward KL or Jensen-Shannon divergence (JSD) objectives, which differ from reverse KL in their mode-coverage behavior and effect on generation diversity~\citep{eopd}. Whether these objectives expand pass@$k$ coverage more effectively, and how the resulting cold-start interacts with RL, remain open questions.

— Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVR  (2609.04108 - Li et al., 3 Sep 2026) in Limitations, paragraph “Choice of OPD objective”

Interactions between axes, their dependence on the task, and their side effects therefore remain largely unknown, and, in our opinion, deserve systematic evaluation.

— Disentangling Self-Distillation: Measuring and Modeling Acquisition and Retention  (2609.39494 - Zuin et al., 30 Sep 2026) in Section 2, “Seeing a forest from the trees”

Finally, our coverage results are based on a finite number of samples. Although we use additional sampling and manual inspection to check apparently new solutions, we cannot rule out that larger sampling budgets or different decoding settings would uncover additional correct responses from the initial model.

— Gains and Collapse in On-Policy Distillation:A Reinforcement Learning Perspective  (2610.03185 - Cui et al., 2 Oct 2026) in Limitations section

We leave a systematic study of the effect of $K$ and the resulting top-$K$ approximation error to future work.

— Learning What to Imitate: Entropy-Aware Distribution Mixing  (2610.06671 - Giraldo et al., 5 Oct 2026) in Appendix, Section 'Implementation Details', paragraph 'Teacher support'

As noted earlier, we leave a systematic study of alternative divergence objectives to future work.

— Learning What to Imitate: Entropy-Aware Distribution Mixing  (2610.06671 - Giraldo et al., 5 Oct 2026) in Appendix, Section 'Results: Reverse-KL Distillation with OPD'

For instance, an interesting open question would be to understand if the diversity improvements of our teacher selection scheme can be used to create a better warm start for RL post-OPD.

— Compatibility Matters: A Framework for Token-Level Teacher Selection in On-Policy Distillation  (2610.21234 - Wei et al., 15 Oct 2026) in Section 6, Conclusion