Explain why teacher mistakes benefit codistillation

Explain why the mistakes made by a teacher network can help a student network in codistillation for click-through-rate prediction.

Background

The paper distinguishes self-ensemble distillation, in which members differ through masks or subnetworks, from distillation among independently trained networks. It notes that codistillation can improve CTR models even when the networks share one architecture, but that the existing explanation attributes the benefit mainly to the networks observing different data subsets.

The unresolved issue is why distillation remains beneficial when the teacher makes prediction errors. The paper does not resolve this question; its own analysis instead focuses on disagreement arising from random seeds and data orders.

References

The authors attribute the gain mainly to the copies seeing different subsets of the data and leave open why a teacher's mistakes help the student.

— FLMNet: Self-Ensemble Distillation from Masks to Seeds for Single-Tower CTR Prediction  (2610.26143 - Wang et al., 19 Oct 2026) in Section 1, Introduction, paragraph beginning “Codistillation shows that distillation between networks”