Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Enhance DNN Adversarial Robustness and Efficiency via Injecting Noise to Non-Essential Neurons (2402.04325v1)

Published 6 Feb 2024 in cs.LG, cs.AI, and cs.CR

Abstract: Deep Neural Networks (DNNs) have revolutionized a wide range of industries, from healthcare and finance to automotive, by offering unparalleled capabilities in data analysis and decision-making. Despite their transforming impact, DNNs face two critical challenges: the vulnerability to adversarial attacks and the increasing computational costs associated with more complex and larger models. In this paper, we introduce an effective method designed to simultaneously enhance adversarial robustness and execution efficiency. Unlike prior studies that enhance robustness via uniformly injecting noise, we introduce a non-uniform noise injection algorithm, strategically applied at each DNN layer to disrupt adversarial perturbations introduced in attacks. By employing approximation techniques, our approach identifies and protects essential neurons while strategically introducing noise into non-essential neurons. Our experimental results demonstrate that our method successfully enhances both robustness and efficiency across several attack scenarios, model architectures, and datasets.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (53)
  1. Achlioptas, D. Database-friendly random projections. In Proceedings of the twentieth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems, pp.  274–281. ACM, 2001.
  2. The fast johnson–lindenstrauss transform and approximate nearest neighbors. SIAM Journal on computing, 39(1):302–322, 2009.
  3. Deep learning algorithm for autonomous driving using googlenet. In 2017 IEEE intelligent vehicles symposium (IV), pp.  89–96. IEEE, 2017.
  4. Deep multi-sensor lane detection. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp.  3102–3109. IEEE, 2018.
  5. Random projection in dimensionality reduction: applications to image and text data. In Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining, pp.  245–250, 2001.
  6. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp), pp.  39–57. Ieee, 2017.
  7. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning, pp.  2206–2216. PMLR, 2020.
  8. Stochastic activation pruning for robust adversarial defense. arXiv preprint arXiv:1803.01442, 2018.
  9. Adversarial robustness via random projection filters. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  4077–4086, 2023.
  10. Boosting adversarial attacks with momentum. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  9185–9193, 2018.
  11. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635, 2018.
  12. Drawing robust scratch tickets: Subnetworks with inborn robustness are found within randomly initialized networks. Advances in Neural Information Processing Systems, 34:13059–13072, 2021.
  13. Deep learning-based image recognition for autonomous driving. IATSS research, 43(4):244–252, 2019.
  14. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
  15. Combating adversarial attacks using sparse representations. arXiv preprint arXiv:1803.03880, 2018.
  16. Model compression with adversarial robustness: A unified optimization framework. Advances in Neural Information Processing Systems, 32, 2019.
  17. Sparse dnns with improved adversarial robustness. Advances in neural information processing systems, 31, 2018.
  18. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015.
  19. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  770–778, 2016.
  20. Parametric noise injection: Trainable randomness to improve deep neural network robustness against adversarial attack. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  588–597, 2019.
  21. Exploring architectural ingredients of adversarially robust deep neural networks. Advances in Neural Information Processing Systems, 34:5545–5559, 2021.
  22. Instructors Sham Kakade, G. Cmsc 35900 (spring 2009) large scale learning lecture: 2 random projections. 2009.
  23. Learn2perturb: an end-to-end feature perturbation learning to improve adversarial robustness. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  1241–1250, 2020.
  24. Johnson, W. B. Extensions of lipshitz mapping into hilbert space. In Conference modern analysis and probability, 1984, pp.  189–206, 1984.
  25. Kim, H. Torchattacks: A pytorch repository for adversarial attacks. arXiv preprint arXiv:2010.01950, 2020.
  26. Learning multiple layers of features from tiny images. 2009.
  27. Adversarial examples in the physical world, 2016.
  28. Certified robustness to adversarial examples with differential privacy. In 2019 IEEE symposium on security and privacy (SP), pp.  656–672. IEEE, 2019.
  29. Very sparse random projections. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp.  287–296. ACM, 2006.
  30. Dynamic sparse graph for efficient deep learning. In International Conference on Learning Representations, 2019.
  31. Boosting deep neural network efficiency with dual-module inference. In International Conference on Machine Learning, pp.  6205–6215. PMLR, 2020.
  32. Towards robust neural networks via random self-ensemble. In Proceedings of the European Conference on Computer Vision (ECCV), pp.  369–385, 2018.
  33. Adversarial neural pruning with latent vulnerability suppression. In International Conference on Machine Learning, pp.  6575–6585. PMLR, 2020.
  34. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017.
  35. Deepfool: a simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  2574–2582, 2016.
  36. Patdnn: Achieving real-time dnn execution on mobile devices with pattern-based weight pruning. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems, pp.  907–922, 2020.
  37. Nvidia. Nvidia a100 tensor core gpu architecture., 2020. URL https://images.nvidia.com/aem-dam/en-zz/Solutions/data-center/nvidia-ampere-architecture-whitepaper.pdf.
  38. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
  39. Theoretical evidence for adversarial robustness through randomization. Advances in neural information processing systems, 32, 2019.
  40. Overfitting in adversarially robust deep learning. In International Conference on Machine Learning, pp.  8093–8104. PMLR, 2020.
  41. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics, 9:53–68, 2021.
  42. Hydra: Pruning adversarially robust neural networks. Advances in Neural Information Processing Systems, 33:19655–19666, 2020.
  43. Adversarial training for free! Advances in Neural Information Processing Systems, 32, 2019.
  44. Vu, K. K. Random projection for high-dimensional optimization. PhD thesis, Université Paris Saclay (COmUE), 2016.
  45. Fast is better than free: Revisiting adversarial training. arXiv preprint arXiv:2001.03994, 2020.
  46. Adversarial weight perturbation helps robust generalization. Advances in Neural Information Processing Systems, 33:2958–2969, 2020.
  47. Enhancing adversarial defense by k-winners-take-all. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=Skgvy64tvr.
  48. Mitigating adversarial effects through randomization. arXiv preprint arXiv:1711.01991, 2017.
  49. Adversarial robustness vs. model compression, or both? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  111–120, 2019.
  50. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016.
  51. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283–17297, 2020.
  52. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning, pp.  7472–7482. PMLR, 2019.
  53. Attacks which do not kill training make adversarial learning stronger. In International conference on machine learning, pp.  11278–11287. PMLR, 2020.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Zhenyu Liu (63 papers)
  2. Garrett Gagnon (2 papers)
  3. Swagath Venkataramani (14 papers)
  4. Liu Liu (190 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets