Papers
Topics
Authors
Recent
2000 character limit reached

Bridging Cross-task Protocol Inconsistency for Distillation in Dense Object Detection (2308.14286v2)

Published 28 Aug 2023 in cs.CV

Abstract: Knowledge distillation (KD) has shown potential for learning compact models in dense object detection. However, the commonly used softmax-based distillation ignores the absolute classification scores for individual categories. Thus, the optimum of the distillation loss does not necessarily lead to the optimal student classification scores for dense object detectors. This cross-task protocol inconsistency is critical, especially for dense object detectors, since the foreground categories are extremely imbalanced. To address the issue of protocol differences between distillation and classification, we propose a novel distillation method with cross-task consistent protocols, tailored for the dense object detection. For classification distillation, we address the cross-task protocol inconsistency problem by formulating the classification logit maps in both teacher and student models as multiple binary-classification maps and applying a binary-classification distillation loss to each map. For localization distillation, we design an IoU-based Localization Distillation Loss that is free from specific network structures and can be compared with existing localization distillation losses. Our proposed method is simple but effective, and experimental results demonstrate its superiority over existing methods. Code is available at https://github.com/TinyTigerPan/BCKD.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (56)
  1. Variational information distillation for knowledge transfer. In Proc. CVPR, pages 9163–9171, 2019.
  2. Tide: A general toolbox for identifying object detection errors. In Proc. ECCV, pages 558–573, 2020.
  3. Cascade r-cnn: Delving into high quality object detection. In Proc. CVPR, pages 6154–6162, 2018.
  4. Pkd: General distillation framework for object detectors via pearson correlation coefficient. arXiv preprint arXiv:2207.02039, 2022.
  5. Learning efficient object detection models with knowledge distillation. In Proc. NeurIPS, pages 16468–16480, 2017.
  6. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155, 2019.
  7. Ranking consistency for image matching and object retrieval. Pattern Recognition, 47(3):1349–1360, 2014.
  8. MMRazor Contributors. Openmmlab model compression toolbox and benchmark. https://github.com/open-mmlab/mmrazor, 2021.
  9. General instance distillation for object detection. In Proc. CVPR, pages 7842–7851, 2021.
  10. Elastic knowledge distillation by learning from recollection. IEEE Transactions on Neural Networks and Learning Systems, 2021.
  11. Born again neural networks. In Proc. ICML, pages 1607–1616. PMLR, 2018.
  12. Ota: Optimal transport assignment for object detection. In Proc. CVPR, pages 303–312, 2021.
  13. Ross Girshick. Fast r-cnn. In Proc. ICCV, pages 1440–1448, 2015.
  14. Distilling object detectors via decoupled features. In Proc. CVPR, pages 2154–2164, 2021.
  15. Distilling image classifiers in object detectors. Advances in Neural Information Processing Systems, 34:1036–1047, 2021.
  16. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.
  17. Learning multi-level density maps for crowd counting. IEEE transactions on neural networks and learning systems, 31(8):2705–2715, 2019.
  18. Instance-conditional knowledge distillation for object detection. In Proc. NeurIPS, pages 16468–16480, 2021.
  19. Knowledge distillation for object detection via rank mimicking and prediction-guided feature imitation. In Proc. AAAI, volume 36, pages 1306–1313, 2022.
  20. Graph mode-based contextual kernels for robust svm tracking. In 2011 international conference on computer vision, pages 1156–1163. IEEE, 2011.
  21. Reskd: Residual-guided knowledge distillation. IEEE Trans. Image Process., 30:4735–4746, 2021.
  22. Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detection. In Proc. NeurIPS, pages 21002–21012, 2020.
  23. Focal loss for dense object detection. In Proc. ICCV, pages 2980–2988, 2017.
  24. Microsoft coco: Common objects in context. In Proc. ECCV, pages 740–755. Springer, 2014.
  25. Ssd: Single shot multibox detector. In Proc. ECCV, pages 21–37. Springer, 2016.
  26. Rtmdet: An empirical study of designing real-time object detectors. arXiv preprint arXiv:2212.07784, 2022.
  27. Improving object detection by label assignment distillation. In "Proc. WACV, pages 1005–1014, 2022.
  28. Relational knowledge distillation. In Proc. CVPR, pages 3967–3976, 2019.
  29. Towards understanding knowledge distillation. In Proc. ICML, pages 5142–5151. PMLR, 2019.
  30. You only look once: Unified, real-time object detection. In Proc. CVPR, pages 779–788, 2016.
  31. Faster r-cnn: Towards real-time object detection with region proposal networks. In Proc. NeurIPS, pages 6906–6919, 2015.
  32. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014.
  33. Does knowledge distillation really work? In Proc. NeurIPS, pages 6906–6919, 2021.
  34. Recent advances and trends in multimodal deep learning: A review. arXiv preprint arXiv:2105.11087, 2021.
  35. Fcos: Fully convolutional one-stage object detection. In Proc. ICCV, pages 9627–9636, 2019.
  36. YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. arXiv preprint arXiv:2207.02696, 2022.
  37. Progressive blockwise knowledge distillation for neural network acceleration. In IJCAI, pages 2769–2775, 2018.
  38. Distilling object detectors with fine-grained feature imitation. In Proc. CVPR, pages 4933–4942, 2019.
  39. Boosting detection in crowd analysis via underutilized output features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15609–15618, 2023.
  40. Rethinking knowledge distillation via cross-entropy. arXiv preprint arXiv:2208.10139, 2022.
  41. Focal and global knowledge distillation for detectors. In Proc. CVPR, pages 4643–4652, 2022.
  42. Masked generative distillation. In Proc. ECCV, pages 53–69. Springer, 2022.
  43. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In Proc. CVPR, pages 4133–4141, 2017.
  44. Improve object detection with feature-based knowledge distillation: Towards accurate and efficient detectors. In Proc. ICLR, 2021.
  45. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proc. ICCV, pages 3713–3722, 2019.
  46. Lgd: label-guided self-distillation for object detection. In Proc. AAAI, volume 36, pages 3309–3317, 2022.
  47. Bridging the gap between anchor-based and anchor-free detection via adaptive training sample selection. In Proc. CVPR, pages 9759–9768, 2020.
  48. Single-shot refinement neural network for object detection. In Proc. CVPR, pages 4203–4212, 2018.
  49. Decoupled knowledge distillation. In Proc. CVPR, pages 11953–11962, 2022.
  50. Mgsvf: Multi-grained slow vs. fast framework for few-shot class-incremental learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
  51. Memory-efficient class-incremental learning for image classification. IEEE Transactions on Neural Networks and Learning Systems, 33(10):5966–5977, 2021.
  52. Rbc: Rectifying the biased context in continual semantic segmentation. In European Conference on Computer Vision, pages 55–72. Springer, 2022.
  53. Localization distillation for dense object detection. In Proc. CVPR, pages 9407–9416, 2022.
  54. Towards defending against adversarial examples via attack-invariant features. In International Conference on Machine Learning, pages 12835–12845. PMLR, 2021.
  55. Modeling adversarial noise for adversarial training. In International Conference on Machine Learning, pages 27353–27366. PMLR, 2022.
  56. Teach less, learn more: On the undistillable classes in knowledge distillation. In Advances in Neural Information Processing Systems, 2022.
Citations (10)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
Lightbulb Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up