Identify effective token-importance signals and position-selection policies for selective KD in autoregressive LLMs

Determine which token-level importance signals most reliably identify positions that benefit from logit-based knowledge distillation in autoregressive large language models, and characterize how different position-selection policies interact with these signals to yield an effective distillation curriculum.

Background

Knowledge distillation for autoregressive LLMs typically matches the teacher’s next-token distribution at every position, but recent work suggests that selectively supervising a subset of positions can improve performance. A variety of token-importance signals (such as entropy, cross-entropy, and teacher–student divergence) and position-selection policies (such as top-k selection, global-level selection, curriculum scheduling, and stochastic sampling) have been proposed, yet their comparative effectiveness and interactions remain uncertain.

This paper frames selective knowledge distillation across multiple axes and focuses on the position axis, emphasizing the need to rigorously determine which uncertainty or discrepancy signals best identify high-value tokens for distillation and how selection policies shape the resulting training curriculum. The authors explicitly note that this uncertainty motivates their systematic analysis and the development of student-entropy-guided selection methods.

References

Yet, it remains unclear which token-importance signals most reliably identify positions that benefit from logit-based distillation in LLMs, and how different position-selection policies interact with these signals to shape an effective distillation curriculum.

— Rethinking Selective Knowledge Distillation  (2602.01395 - Tavor et al., 1 Feb 2026) in Section 1 (Introduction), second paragraph

Despite the consistent performance gains of GMTS in RLVR, several limitations remain and point to directions for future research. First, although extensive empirical evaluations support the effectiveness of GMTS, its theoretical foundation remains only partially understood. In this work, we provide preliminary evidence on the relationship between token entropy and gradient magnitude, but a rigorous theoretical justification for why GMTS outperforms ETS, along with a deeper analytical characterisation of its behaviour, remains an important direction for future work.

— GMTS: Gradient Magnitude-based Token Selection Improves RLVR Training for LLM Reasoning  (2608.30632 - Lv et al., 31 Aug 2026) in Limitations section

It remains unclear which tokens most strongly reshape the student's shared representation; predictive entropy may simply serve as a proxy for these influential updates rather than their underlying cause.

— Learning What to Imitate: Entropy-Aware Distribution Mixing  (2610.06671 - Giraldo et al., 5 Oct 2026) in Section 'Limitations and open questions' in the Conclusion

The strong Sparse RKL also raises a further question: could two directions benefit from different position or allocations that change during training? DrOPD provides a simple effective allocation, while jointly adapting the placement and relative strength of RKL and FKL remains for future work.

— Divergence-Routed On-Policy Distillation for Language Models  (2610.19572 - Peng et al., 14 Oct 2026) in Section 4.6, page 7

Selective supervision thus raises a question of reliability: how can useful teacher knowledge be retained without also reinforcing its errors?

— Anvil: Scaling Embodied Spatial Intelligence through Knowledge Recirculation  (2610.21882 - Ruan et al., 15 Oct 2026) in Section 2, Related Work, paragraph 'Knowledge distillation'

Finally, our analysis relates the token-level guidance to the gains of BSD only through correlations across students. It characterizes where the guidance concentrates but does not establish which tokens it helps, or why, which we leave for future work.

— Bridged Self-Distillation for On-Policy Learning  (2610.26397 - Chen et al., 19 Oct 2026) in Limitations, third paragraph