Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Logit Standardization in Knowledge Distillation (2403.01427v1)

Published 3 Mar 2024 in cs.CV

Abstract: Knowledge distillation involves transferring soft labels from a teacher to a student using a shared temperature-based softmax function. However, the assumption of a shared temperature between teacher and student implies a mandatory exact match between their logits in terms of logit range and variance. This side-effect limits the performance of student, considering the capacity discrepancy between them and the finding that the innate logit relations of teacher are sufficient for student to learn. To address this issue, we propose setting the temperature as the weighted standard deviation of logit and performing a plug-and-play Z-score pre-process of logit standardization before applying softmax and Kullback-Leibler divergence. Our pre-process enables student to focus on essential logit relations from teacher rather than requiring a magnitude match, and can improve the performance of existing logit-based distillation methods. We also show a typical case where the conventional setting of sharing temperature between teacher and student cannot reliably yield the authentic distillation evaluation; nonetheless, this challenge is successfully alleviated by our Z-score. We extensively evaluate our method for various student and teacher models on CIFAR-100 and ImageNet, showing its significant superiority. The vanilla knowledge distillation powered by our pre-process can achieve favorable performance against state-of-the-art methods, and other distillation variants can obtain considerable gain with the assistance of our pre-process.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. Variational information distillation for knowledge transfer. In CVPR, 2019.
  2. Revisiting label smoothing and knowledge distillation compatibility: What was missing? In ICML, 2022.
  3. Online knowledge distillation with diverse peers. In AAAI, 2020.
  4. Knowledge distillation with the reused teacher classifier. In CVPR, 2022a.
  5. Distilling knowledge via knowledge review. In CVPR, 2021.
  6. Dearkd: Data-efficient early knowledge distillation for vision transformers. In CVPR, 2022b.
  7. On the efficacy of knowledge distillation. In ICCV, 2019.
  8. On calibration of modern neural networks. In ICML, 2017.
  9. Jia Guo. Reducing the teacher-student gap via adaptive temperatures, 2022.
  10. Class attention transfer based knowledge distillation. In CVPR, 2023.
  11. Deep residual learning for image recognition. In CVPR, 2016.
  12. A comprehensive overhaul of feature distillation. In ICCV, 2019.
  13. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.
  14. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.
  15. Knowledge distillation from a stronger teacher. NeurIPS, 2022.
  16. Edwin T Jaynes. Information theory and statistical mechanics. Physical review, 106(4):620, 1957.
  17. Multi-level logit distillation. In CVPR, 2023.
  18. Learning multiple layers of features from tiny images. 2009.
  19. Knowledge distillation for object detection via rank mimicking and prediction-guided feature imitation. In AAAI, 2022a.
  20. Locality guidance for improving vision transformers on tiny datasets. In ECCV, 2022b.
  21. Automated knowledge distillation via monte carlo tree search. In ICCV, 2023a.
  22. Online knowledge distillation via multi-branch diversity enhancement. In ACCV, 2020.
  23. Online knowledge distillation for efficient pose estimation. In ICCV, 2021.
  24. Curriculum temperature for knowledge distillation. In AAAI, 2023b.
  25. Knowledge distillation via the target-aware transformer. In CVPR, 2022.
  26. Meta knowledge distillation. arXiv preprint arXiv:2202.07940, 2022.
  27. Structured knowledge distillation for semantic segmentation. In CVPR, 2019.
  28. Improved knowledge distillation via teacher assistant. In AAAI, 2020.
  29. Relational knowledge distillation. In CVPR, 2019.
  30. Correlation congruence for knowledge distillation. In ICCV, 2019.
  31. Fitnets: Hints for thin deep nets. ICLR, 2015.
  32. Imagenet large scale visual recognition challenge. IJCV, 2015.
  33. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018.
  34. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
  35. Densely guided knowledge distillation using multiple teacher assistants. In ICCV, 2021.
  36. On the importance of initialization and momentum in deep learning. In ICML, 2013.
  37. Contrastive representation distillation. ICLR, 2020.
  38. Training data-efficient image transformers & distillation through attention. In ICML, 2021.
  39. Similarity-preserving knowledge distillation. In ICCV, 2019.
  40. Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 2008.
  41. Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks. IEEE TPAMI, 2021.
  42. Tinyvit: Fast pretraining distillation for small vision transformers. In ECCV, 2022.
  43. Knowledge distillation via softmax regression representation learning. In ICLR, 2021.
  44. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In CVPR, 2017.
  45. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016.
  46. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. ICLR, 2017.
  47. Do not blindly imitate the teacher: Using perturbed loss for knowledge distillation. arXiv preprint arXiv:2305.05010, 2023.
  48. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In CVPR, 2018a.
  49. Deep mutual learning. In CVPR, 2018b.
  50. Decoupled knowledge distillation. In CVPR, 2022.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Shangquan Sun (6 papers)
  2. Wenqi Ren (67 papers)
  3. Jingzhi Li (64 papers)
  4. Rui Wang (996 papers)
  5. Xiaochun Cao (177 papers)
Citations (30)
View on Semantic Scholar