Generalization of feature-distillation behavior beyond structured-pruned initialization
Determine whether the observed pattern—that projection-based and CKA feature distillation fail to uniformly improve compact multilingual automatic speech recognition, with CKA benefiting non-English languages while harming English—also holds when the student model is initialized randomly rather than by structured pruning of the teacher.
References
Whether the same pattern holds under random initialization is left to future work.
— CAFE: Confining Feature Distillation to Language-Routed Adapters for Compact Multilingual Automatic Speech Recognition
(2610.19134 - Liu et al., 13 Oct 2026) in Section*{Limitations}