Transferability of CROP Across Models and Domains

Determine the extent to which the findings for CROP transfer across model families and domains beyond the two Qwen teacher–student settings and mathematical training prompts.

Background

The paper evaluates Counterfactual Relevance for On-Policy Distillation (CROP) in only two Qwen teacher–student configurations using mathematical training prompts. CROP ranks response tokens by comparing student-distribution sensitivity to a condition-changing counterfactual against sensitivity to a meaning-preserving paraphrase. The authors explicitly identify the unresolved issue of whether the observed benefits generalize beyond these model families and the mathematical domain. They also note that future evaluations could extend the method to coding and open-domain reasoning, but the explicitly stated open problem is the broader transferability of the findings.