Extend MetaOPD to broader models and tasks

Extend the empirical evaluation of MetaOPD, a bilevel on-policy distillation framework that learns token weights from post-update validation loss, to broader model families and task domains.

Background

MetaOPD is evaluated with a Qwen3-8B teacher and Qwen3-0.6B-Base or Qwen3-1.7B-Base students on mathematical-reasoning benchmarks, with additional out-of-domain evaluation on GPQA-Diamond, MMLU-Pro, and AlpacaEval 2.0. The paper explicitly identifies extending these findings to broader models and tasks as an unresolved direction, leaving open whether the observed benefits generalize beyond the evaluated architectures, scales, and datasets.

References

Extending these findings to broader models and tasks, relating local update quality to long-horizon learning, and reducing meta-update cost remain open directions.

— MetaOPD: Meta-Learned Token Weighting for On-Policy Distillation  (2610.11989 - Wang et al., 8 Oct 2026) in Section Conclusion