Transferability of CROP Across Models and Domains

Determine the extent to which the findings for CROP transfer across model families and domains beyond the two Qwen teacher–student settings and mathematical training prompts.

Background

The paper evaluates Counterfactual Relevance for On-Policy Distillation (CROP) in only two Qwen teacher–student configurations using mathematical training prompts. CROP ranks response tokens by comparing student-distribution sensitivity to a condition-changing counterfactual against sensitivity to a meaning-preserving paraphrase. The authors explicitly identify the unresolved issue of whether the observed benefits generalize beyond these model families and the mathematical domain. They also note that future evaluations could extend the method to coding and open-domain reasoning, but the explicitly stated open problem is the broader transferability of the findings.

References

Our experiments currently cover two Qwen teacher--student settings and mathematical training prompts, so the extent to which the findings transfer across model families and domains remains open.

— CROP: Task Relevance via Counterfactuals for Selective On-Policy Distillation  (2608.13387 - Li et al., 13 Aug 2026) in Section Limitations, Appendix

Our MOPD runs use three domains with one teacher each, leaving open how far the diversity result extends as teachers are added.

— Rethinking On-Policy Distillation of Large Language Models II: One Training Example  (2609.04172 - Fu et al., 3 Sep 2026) in Section 9, Conclusion, Limitations