Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Exploring Inconsistent Knowledge Distillation for Object Detection with Data Augmentation (2209.09841v3)

Published 20 Sep 2022 in cs.CV

Abstract: Knowledge Distillation (KD) for object detection aims to train a compact detector by transferring knowledge from a teacher model. Since the teacher model perceives data in a way different from humans, existing KD methods only distill knowledge that is consistent with labels annotated by human expert while neglecting knowledge that is not consistent with human perception, which results in insufficient distillation and sub-optimal performance. In this paper, we propose inconsistent knowledge distillation (IKD), which aims to distill knowledge inherent in the teacher model's counter-intuitive perceptions. We start by considering the teacher model's counter-intuitive perceptions of frequency and non-robust features. Unlike previous works that exploit fine-grained features or introduce additional regularizations, we extract inconsistent knowledge by providing diverse input using data augmentation. Specifically, we propose a sample-specific data augmentation to transfer the teacher model's ability in capturing distinct frequency components and suggest an adversarial feature augmentation to extract the teacher model's perceptions of non-robust features in the data. Extensive experiments demonstrate the effectiveness of our method which outperforms state-of-the-art KD baselines on one-stage, two-stage and anchor-free object detectors (at most +1.0 mAP). Our codes will be made available at \url{https://github.com/JWLiang007/IKD.git}.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (61)
  1. Adversarial training for probabilistic spiking neural networks. In 2018 IEEE 19th International Workshop on Signal Processing Advances in Wireless Communications (SPAWC). IEEE, 1–5.
  2. Improving vision transformers by revisiting high-frequency components. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXIV. Springer, 1–18.
  3. Learning efficient object detection models with knowledge distillation. Advances in neural information processing systems 30 (2017).
  4. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155 (2019).
  5. Gridmask data augmentation. arXiv preprint arXiv:2001.04086 (2020).
  6. Jang Hyun Cho and Bharath Hariharan. 2019. On the efficacy of knowledge distillation. In Proceedings of the IEEE/CVF international conference on computer vision. 4794–4802.
  7. A comprehensive survey on model compression and acceleration. Artificial Intelligence Review 53, 7 (2020), 5113–5155.
  8. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. Ieee, 248–255.
  9. Terrance DeVries and Graham W Taylor. 2017. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552 (2017).
  10. Shortcut learning in deep neural networks. Nature Machine Intelligence 2, 11 (2020), 665–673.
  11. Simple copy-paste is a strong data augmentation method for instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2918–2928.
  12. Generative adversarial nets. Advances in neural information processing systems 27 (2014).
  13. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv:1708.06733 (2017).
  14. A Comprehensive Evaluation Framework for Deep Model Robustness. Pattern Recognition (2023).
  15. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision. 2961–2969.
  16. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770–778.
  17. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 2, 7 (2015).
  18. Adversarial examples are not bugs, they are features. Advances in neural information processing systems 32 (2019).
  19. High accuracy and high fidelity extraction of neural networks. In Proceedings of the 29th USENIX Conference on Security Symposium. 1345–1362.
  20. Prior-guided adversarial initialization for fast adversarial training. In European Conference on Computer Vision. Springer, 567–584.
  21. LAS-AT: adversarial training with learnable attack strategy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13398–13408.
  22. Huaizu Jiang and Erik Learned-Miller. 2017. Face detection with the faster R-CNN. In 2017 12th IEEE international conference on automatic face & gesture recognition (FG 2017). IEEE, 650–657.
  23. Data augmentation for object detection: A review. In 2021 IEEE International Midwest Symposium on Circuits and Systems (MWSCAS). IEEE, 537–543.
  24. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems 25 (2012).
  25. Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. arXiv preprint arXiv:2110.05208 (2021).
  26. Neural attention distillation: Erasing backdoor triggers from deep neural networks. arXiv preprint arXiv:2101.05930 (2021).
  27. A Large-Scale Multiple-objective Method for Black-box Attack Against Object Detection. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part IV. Springer, 619–636.
  28. Imitated Detectors: Stealing Knowledge of Black-box Object Detectors. In Proceedings of the 30th ACM International Conference on Multimedia. 4839–4847.
  29. Efficient adversarial attacks for visual object tracking. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVI 16. Springer, 34–50.
  30. Parallel rectangle flip attack: A query-based black-box attack against object detection. arXiv preprint arXiv:2201.08970 (2022).
  31. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision. 2980–2988.
  32. Microsoft coco: Common objects in context. In European conference on computer vision. Springer, 740–755.
  33. X-adv: Physical adversarial object attacks against x-ray prohibited item detection.
  34. Spatiotemporal attacks for embodied agents. In ECCV.
  35. Perceptual-sensitive gan for generating adversarial patches. In AAAI.
  36. Training robust deep neural networks via adversarial noise propagation. TIP (2021).
  37. Exploring the Relationship between Architecture and Adversarially Robust Generalization. In CVPR.
  38. Bias-based universal adversarial patch attack for automatic check-out. In ECCV.
  39. Harnessing Perceptual Adversarial Patches for Crowd Counting. In ACM CCS.
  40. Poisoning attack against estimating from pairwise comparisons. IEEE Transactions on Pattern Analysis and Machine Intelligence 44, 10 (2021), 6393–6408.
  41. A Tale of HodgeRank and Spectral Method: Target Attack Against Rank Aggregation is the Fixed Point of Adversarial Game. IEEE Transactions on Pattern Analysis and Machine Intelligence 45, 4 (2022), 4090–4108.
  42. Virtual big data for GAN based data augmentation. In 2019 IEEE International Conference on Big Data (Big Data). IEEE, 1478–1487.
  43. Channel-wise knowledge distillation for dense prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 5311–5320.
  44. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).
  45. Fabio Henrique Kiyoiti dos Santos Tanaka and Claus Aranha. 2019. Data augmentation using GANs. arXiv preprint arXiv:1904.09135 (2019).
  46. Robustart: Benchmarking robustness on architecture design and training techniques. ArXiv (2021).
  47. High-frequency component helps explain the generalization of convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 8684–8694.
  48. Dual attention suppression attack: Generate adversarial camouflage in physical world. In CVPR.
  49. Distilling object detectors with fine-grained feature imitation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 4933–4942.
  50. Transferable adversarial attacks for image and video object detection. arXiv preprint arXiv:1811.12641 (2018).
  51. Backdoorbench: A comprehensive benchmark of backdoor learning. Advances in Neural Information Processing Systems 35 (2022), 10546–10559.
  52. Improving transferability of adversarial examples with input diversity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2730–2739.
  53. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1492–1500.
  54. Focal and global knowledge distillation for detectors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 4643–4652.
  55. Masked Generative Distillation. arXiv preprint arXiv:2205.01529 (2022).
  56. Reppoints: Point set representation for object detection. In Proceedings of the IEEE/CVF international conference on computer vision. 9657–9666.
  57. A fourier perspective on model robustness in computer vision. Advances in Neural Information Processing Systems 32 (2019).
  58. A review of machine learning and IoT in smart transportation. Future Internet 11, 4 (2019), 94.
  59. Interpreting and Improving Adversarial Robustness of Deep Neural Networks with Neuron Sensitivity. IEEE Transactions on Image Processing (2021).
  60. Linfeng Zhang and Kaisheng Ma. 2020. Improve object detection with feature-based knowledge distillation: Towards accurate and efficient detectors. In International Conference on Learning Representations.
  61. Random erasing data augmentation. In Proceedings of the AAAI conference on artificial intelligence, Vol. 34. 13001–13008.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Jiawei Liang (8 papers)
  2. Siyuan Liang (73 papers)
  3. Aishan Liu (72 papers)
  4. Ke Ma (75 papers)
  5. Jingzhi Li (64 papers)
  6. Xiaochun Cao (177 papers)
Citations (13)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now