Generalization of feature-distillation behavior beyond structured-pruned initialization

Determine whether the observed pattern—that projection-based and CKA feature distillation fail to uniformly improve compact multilingual automatic speech recognition, with CKA benefiting non-English languages while harming English—also holds when the student model is initialized randomly rather than by structured pruning of the teacher.

Background

The study initializes its 254M multilingual ASR student through structured pruning of the Parakeet-TDT-0.6B-v3 teacher and explicitly limits its conclusions to this training regime. Its central empirical pattern is that feature distillation behaves differently across languages: projection-based distillation contributes little, whereas CKA improves non-English performance at the expense of English unless its gradient is confined to language-routed adapters. The unresolved question is whether this behavior is a consequence of the structured-pruned initialization or persists when the compact student is trained from a random initialization.

References

Whether the same pattern holds under random initialization is left to future work.