Efficiently recovering post-training capabilities after hybrid distillation

Develop an efficient method to recover the instruction-following and alignment capabilities introduced by post-training in the original Transformer base models after they are converted via distillation into RNN–attention hybrid architectures using pre-training-style data.

Background

The paper presents HALO, a distillation pipeline that converts pre-trained Transformer models into RNN–attention hybrid architectures (HypeNet) using primarily pre-training-style data. While this conversion yields strong long-context efficiency and competitive performance, it can diminish capabilities that are typically introduced during post-training, such as instruction-following and alignment.

The authors note that this degradation is a common shortcoming across existing distillation-based conversions to hybrid models, not just their own approach. They explicitly identify the efficiency-focused recovery of these post-training capabilities in the converted hybrid models as an open question.

References

How to efficiently recover the base models' capabilities remains an open question.

— Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contexts  (2601.22156 - Chen et al., 29 Jan 2026) in Limitations

The aggregate statistics leave recovery success and the retention of specific teacher behaviors unresolved.

— Faynt: Scaling and Optimizing Policies for Competitive Melee  (2610.02144 - Janati et al., 1 Oct 2026) in Appendix E, Section “Scope of the Comparisons”