Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

m2mKD: Module-to-Module Knowledge Distillation for Modular Transformers (2402.16918v3)

Published 26 Feb 2024 in cs.LG and cs.CV

Abstract: Modular neural architectures are gaining attention for their powerful generalization and efficient adaptation to new domains. However, training these models poses challenges due to optimization difficulties arising from intrinsic sparse connectivity. Leveraging knowledge from monolithic models through techniques like knowledge distillation can facilitate training and enable integration of diverse knowledge. Nevertheless, conventional knowledge distillation approaches are not tailored to modular models and struggle with unique architectures and enormous parameter counts. Motivated by these challenges, we propose module-to-module knowledge distillation (m2mKD) for transferring knowledge between modules. m2mKD combines teacher modules of a pretrained monolithic model and student modules of a modular model with a shared meta model respectively to encourage the student module to mimic the behaviour of the teacher module. We evaluate m2mKD on two modular neural architectures: Neural Attentive Circuits (NACs) and Vision Mixture-of-Experts (V-MoE). Applying m2mKD to NACs yields significant improvements in IID accuracy on Tiny-ImageNet (up to 5.6%) and OOD robustness on Tiny-ImageNet-R (up to 4.2%). Additionally, the V-MoE-Base model trained with m2mKD achieves 3.5% higher accuracy than end-to-end training on ImageNet-1k. Code is available at https://github.com/kamanphoebe/m2mKD.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (43)
  1. Do deep nets really need to be deep? In Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., and Weinberger, K., editors, Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc.
  2. Language models are few-shot learners. arXiv preprint arXiv: 2005.14165.
  3. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255.
  4. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv:2010.11929 [cs].
  5. Efficient knowledge distillation from an ensemble of teachers. In Interspeech.
  6. Coordination among neural modules through a shared global workspace. ArXiv, abs/2103.01197.
  7. DEMix layers: Disentangling domains for modular language modeling. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5557–5576, Seattle, United States. Association for Computational Linguistics.
  8. Dynamic neural networks: A survey. IEEE Transactions on Pattern Analysis & Machine Intelligence, 44(11):7436–7456.
  9. Fastermoe: Modeling and optimizing training of large-scale dynamic pre-trained models. In Proceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, PPoPP ’22, page 120–134, New York, NY, USA. Association for Computing Machinery.
  10. The many faces of robustness: A critical analysis of out-of-distribution generalization.
  11. The many faces of robustness: A critical analysis of out-of-distribution generalization. ICCV.
  12. Distilling the knowledge in a neural network.
  13. Parameter-efficient transfer learning for nlp. arXiv preprint arXiv: 1902.00751.
  14. Lora: Low-rank adaptation of large language models. International Conference On Learning Representations.
  15. Tutel: Adaptive mixture-of-experts at scale.
  16. Scaling up visual and vision-language representation learning with noisy text supervision. In Meila, M. and Zhang, T., editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 4904–4916. PMLR.
  17. Paraphrasing complex network: network compression via factor transfer. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 2765–2774.
  18. Sparse upcycling: Training mixture-of-experts from dense checkpoints. International Conference on Learning Representations.
  19. Krizhevsky, A. (2009). Learning multiple layers of features from tiny images. pages 32–33.
  20. Adaptive knowledge distillation based on entropy. In ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7409–7413.
  21. Gshard: Scaling giant models with conditional computation and automatic sharding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
  22. Prefix-tuning: Optimizing continuous prompts for generation. Annual Meeting Of The Association For Computational Linguistics.
  23. Module-wise adaptive distillation for multimodality foundation models.
  24. Improved knowledge distillation via teacher assistant: Bridging the gap between student and teacher. arXiv preprint arXiv:1902.03393.
  25. Multimodal contrastive learning with limoe: the language-image mixture of experts.
  26. Deep Incubation: Training Large Models by Divide-and-Conquering. arXiv:2212.04129 [cs].
  27. Evomoe: An evolutional mixture-of-experts training framework via dense-to-sparse gate.
  28. Modular deep learning. arXiv preprint arXiv: Arxiv-2302.11529.
  29. Contextual parameter generation for universal neural machine translation. arXiv preprint arXiv: 1808.08493.
  30. Efficient parametrization of multi-domain deep neural networks. arXiv preprint arXiv: 1803.10082.
  31. Scaling vision with sparse mixture of experts. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W., editors, Advances in Neural Information Processing Systems, volume 34, pages 8583–8595. Curran Associates, Inc.
  32. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.
  33. Moduleformer: Learning modular large language models from uncurated data. arXiv preprint arXiv: 2306.04640.
  34. Densely guided knowledge distillation using multiple teacher assistants. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9375–9384.
  35. Training data-efficient image transformers & distillation through attention. In Meila, M. and Zhang, T., editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 10347–10357. PMLR.
  36. The Caltech-UCSD Birds-200-2011 Dataset.
  37. Neural Attentive Circuits.
  38. Bert-of-theseus: Compressing bert by progressive module replacing. In Conference on Empirical Methods in Natural Language Processing.
  39. Deep model reassembly. arXiv preprint arXiv: 2210.17409.
  40. Reinforced multi-teacher selection for knowledge distillation. Proceedings of the AAAI Conference on Artificial Intelligence, 35(16):14284–14291.
  41. Moefication: Transformer feed-forward layers are mixtures of experts. FINDINGS.
  42. Knowledge Distillation via Module Replacing for Automatic Speech Recognition with Recurrent Neural Network Transducer. In Proc. Interspeech 2022, pages 4436–4440.
  43. St-moe: Designing stable and transferable sparse expert models.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (8)
  1. Ka Man Lo (5 papers)
  2. Yiming Liang (22 papers)
  3. Wenyu Du (21 papers)
  4. Yuantao Fan (8 papers)
  5. Zili Wang (52 papers)
  6. Wenhao Huang (98 papers)
  7. Lei Ma (195 papers)
  8. Jie Fu (229 papers)
Citations (2)
View on Semantic Scholar