Papers
Topics
Authors
Recent
Search
2000 character limit reached

Relationship between Model Compression and Adversarial Robustness: A Review of Current Evidence

Published 27 Nov 2023 in cs.LG and cs.CV | (2311.15782v1)

Abstract: Increasing the model capacity is a known approach to enhance the adversarial robustness of deep learning networks. On the other hand, various model compression techniques, including pruning and quantization, can reduce the size of the network while preserving its accuracy. Several recent studies have addressed the relationship between model compression and adversarial robustness, while some experiments have reported contradictory results. This work summarizes available evidence and discusses possible explanations for the observed effects.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (67)
  1. I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” in International Conference on Learning Representations (ICLR), 2015.
  2. C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” in International Conference on Learning Representations (ICLR), 2014.
  3. A. Shafahi, M. Najibi, M. A. Ghiasi, Z. Xu, J. Dickerson, C. Studer, L. S. Davis, G. Taylor, and T. Goldstein, “Adversarial training for free!” in Advances in Neural Information Processing Systems (NIPS), 2019.
  4. T. Pang, X. Yang, Y. Dong, H. Su, and J. Zhu, “Bag of Tricks for Adversarial Training,” in International Conference on Learning Representations (ICLR), 2021.
  5. P. Maini, E. Wong, and J. Z. Kolter, “Adversarial robustness against the union of multiple perturbation models,” in International Conference on Machine Learning (ICML), 2020.
  6. L. Schott, J. Rauber, M. Bethge, and W. Brendel, “Towards the first adversarially robust neural network model on mnist,” in International Conference on Learning Representations (ICLR), 2018.
  7. A. Athalye, L. Engstrom, A. Ilyas, and K. Kwok, “Synthesizing Robust Adversarial Examples,” in International Conference on Machine Learning (ICML), 2018.
  8. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. S. Bernstein, A. C. Berg, and L. Fei-Fei, “Imagenet large scale visual recognition challenge,” Int. J. Comput. Vis., 2015.
  9. H. Salman, A. Ilyas, L. Engstrom, A. Kapoor, and A. Madry, “Do adversarially robust imagenet models transfer better?” in Advances in Neural Information Processing Systems (NIPS), 2020.
  10. C. Xie, M. Tan, B. Gong, J. Wang, A. L. Yuille, and Q. V. Le, “Adversarial examples improve image recognition,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  11. M. Andriushchenko and N. Flammarion, “Understanding and improving fast adversarial training,” in Advances in Neural Information Processing Systems (NIPS), 2020.
  12. A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards Deep Learning Models Resistant to Adversarial Attacks,” in International Conference on Learning Representations (ICLR), 2018.
  13. L. Rice, E. Wong, and J. Z. Kolter, “Overfitting in adversarially robust deep learning,” in International Conference on Machine Learning (ICML), 2020.
  14. H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf, “Pruning filters for efficient convnets,” in International Conference on Learning Representations (ICLR), 2017.
  15. S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding,” in International Conference on Learning Representations (ICLR), 2016.
  16. P. Stock, A. Joulin, R. Gribonval, B. Graham, and H. Jégou, “And the bit goes down: Revisiting the quantization of neural networks,” in International Conference on Learning Representations (ICLR), 2020.
  17. A. Galloway, G. W. Taylor, and M. Moussa, “Attacking binarized neural networks,” in International Conference on Learning Representations (ICLR), 2018.
  18. A. S. Rakin, J. Yi, B. Gong, and D. Fan, “Defend deep neural networks against adversarial examples via fixed and dynamic quantized activation functions,” arXiv preprint arXiv:1807.06714, 2018.
  19. A. W. Wijayanto, J. J. Choong, K. Madhawa, and T. Murata, “Towards robust compressed convolutional neural networks,” in IEEE International Conference on Big Data and Smart Computing (BigComp), 2019.
  20. J. Lin, C. Gan, and S. Han, “Defensive quantization: When efficiency meets robustness,” in International Conference on Learning Representations (ICLR), 2019.
  21. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proc. IEEE, 1998.
  22. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  23. N. Carlini and D. Wagner, “Towards evaluating the robustness of neural networks,” in IEEE Symposium on Security and Privacy, 2017.
  24. S. Zagoruyko and N. Komodakis, “Wide residual networks,” in British Machine Vision Conference (BMVC), 2016.
  25. A. Krizhevsky, G. Hinton et al., “Learning multiple layers of features from tiny images,” 2009.
  26. N. Papernot, P. McDaniel, S. Jha, M. Fredrikson, Z. B. Celik, and A. Swami, “The limitations of deep learning in adversarial settings,” in IEEE European symposium on security and privacy (EuroS&P), 2016.
  27. P. Chen, H. Zhang, Y. Sharma, J. Yi, and C. Hsieh, “ZOO: zeroth order optimization based black-box attacks to deep neural networks without training substitute models,” in ACM Workshop on Artificial Intelligence and Security, 2017.
  28. A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” CoRR, vol. abs/1704.04861, 2017.
  29. A. Zhou, A. Yao, Y. Guo, L. Xu, and Y. Chen, “Incremental network quantization: Towards lossless cnns with low-precision weights,” in International Conference on Learning Representations (ICLR), 2017.
  30. Y. Guo, A. Yao, and Y. Chen, “Dynamic network surgery for efficient dnns,” in Advances in Neural Information Processing Systems (NIPS), 2016.
  31. A. Kurakin, I. J. Goodfellow, and S. Bengio, “Adversarial examples in the physical world,” in International Conference on Learning Representations (ICLR) - Workshops, 2017.
  32. K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in International Conference on Learning Representations (ICLR), 2015.
  33. Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng, “Reading digits in natural images with unsupervised feature learning,” 2011.
  34. W. Xu, D. Evans, and Y. Qi, “Feature squeezing: Detecting adversarial examples in deep neural networks,” in Proceedings 2018 Network and Distributed System Security Symposium, 2018.
  35. M. Gorsline, J. Smith, and C. Merkel, “On the adversarial robustness of quantized neural networks,” in Proceedings of the 2021 on Great Lakes Symposium on VLSI, 2021.
  36. S. Varghese, C. Hümmer, A. Bär, F. Hüger, and T. Fingscheidt, “Joint optimization for dnn model compression and corruption robustness,” Deep Neural Networks and Data for Automated Driving, 2022.
  37. L. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder-decoder with atrous separable convolution for semantic image segmentation,” in European Conference on Computer Vision (ECCV), 2018.
  38. M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  39. T. Stauner, F. Blank, M. Fürst, J. Günther, K. Hagn, P. Heidenreich, M. Huber, B. Knerr, T. Schulik, and K. Leiß, “Synpeds: A synthetic dataset for pedestrian detection in urban traffic scenes,” in Computer Science in Cars Symposium, CSCS, 2022.
  40. L. Wang, G. W. Ding, R. Huang, Y. Cao, and Y. C. Lui, “Adversarial robustness of pruned neural networks,” Preprint, 2018.
  41. Y. Guo, C. Zhang, C. Zhang, and Y. Chen, “Sparse dnns with improved adversarial robustness,” Advances in Neural Information Processing Systems (NIPS), vol. 31, 2018.
  42. K. Neklyudov, D. Molchanov, A. Ashukha, and D. P. Vetrov, “Structured bayesian pruning via log-normal multiplicative noise,” Advances in Neural Information Processing Systems (NIPS), vol. 30, 2017.
  43. S.-M. Moosavi-Dezfooli, A. Fawzi, and P. Frossard, “Deepfool: A simple and accurate method to fool deep neural networks,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  44. A. Jordao and H. Pedrini, “On the effect of pruning on adversarial robustness,” in International Conference on Computer Vision (ICCV), 2021.
  45. M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “Mobilenetv2: Inverted residuals and linear bottlenecks,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  46. H. Zhang, M. Cissé, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” in 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings, 2018.
  47. T. DeVries and G. W. Taylor, “Improved regularization of convolutional neural networks with cutout,” arXiv preprint arXiv:1708.04552, 2017.
  48. S. Yun, D. Han, S. J. Oh, S. Chun, J. Choe, and Y. Yoo, “Cutmix: Regularization strategy to train strong classifiers with localizable features,” in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 6023–6032.
  49. J. T. C. Min and M. Motani, “Dropnet: reducing neural network complexity via iterative pruning,” in Proceedings of the 37th International Conference on Machine Learning, 2020, pp. 9356–9366.
  50. M. Lin, R. Ji, Y. Wang, Y. Zhang, B. Zhang, Y. Tian, and L. Shao, “Hrank: Filter pruning using high-rank feature map,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  51. J.-H. Luo and J. Wu, “Neural network pruning with residual-connections and limited-data,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  52. A. Jordao, F. Yamada, and W. R. Schwartz, “Deep network compression based on partial least squares,” Neurocomputing, 2020.
  53. D. Hendrycks and T. G. Dietterich, “Benchmarking neural network robustness to common corruptions and perturbations,” in International Conference on Learning Representations (ICLR), 2019.
  54. N. Liao, S. Wang, L. Xiang, N. Ye, S. Shao, and P. Chu, “Achieving adversarial robustness via sparsity,” Machine Learning, 2022.
  55. G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  56. S. Gui, H. Wang, H. Yang, C. Yu, Z. Wang, and J. Liu, “Model compression with adversarial robustness: A unified optimization framework,” Advances in Neural Information Processing Systems (NIPS), 2019.
  57. A. Sinha, H. Namkoong, and J. C. Duchi, “Certifying some distributional robustness with principled adversarial training,” in International Conference on Learning Representations (ICLR), 2018.
  58. S. Ye, K. Xu, S. Liu, H. Cheng, J.-H. Lambrechts, H. Zhang, A. Zhou, K. Ma, Y. Wang, and X. Lin, “Adversarial robustness vs. model compression, or both?” in International Conference on Computer Vision (ICCV), 2019.
  59. S. Han, J. Pool, J. Tran, and W. Dally, “Learning both weights and connections for efficient neural network,” Advances in Neural Information Processing Systems (NIPS), vol. 28, 2015.
  60. V. Sehwag, S. Wang, P. Mittal, and S. Jana, “Hydra: Pruning adversarially robust neural networks,” Advances in Neural Information Processing Systems (NIPS), 2020.
  61. T. Hu, T. Chen, H. Wang, and Z. Wang, “Triple wins: Boosting accuracy, robustness and efficiency together by enabling input-adaptive inference,” in International Conference on Learning Representations (ICLR), 2020.
  62. Z. Huang and N. Wang, “Data-driven sparse structure selection for deep neural networks,” in European Conference on Computer Vision (ECCV), 2018.
  63. T. Weng, H. Zhang, P. Chen, J. Yi, D. Su, Y. Gao, C. Hsieh, and L. Daniel, “Evaluating the robustness of neural networks: An extreme value theory approach,” in International Conference on Learning Representations (ICLR), 2018.
  64. Y. Carmon, A. Raghunathan, L. Schmidt, J. C. Duchi, and P. S. Liang, “Unlabeled data improves adversarial robustness,” Advances in Neural Information Processing Systems (NIPS), vol. 32, 2019.
  65. Y. Kaya, S. Hong, and T. Dumitras, “Shallow-deep networks: Understanding and mitigating network overthinking,” in International Conference on Machine Learning (ICML), 2019.
  66. J. Frankle and M. Carbin, “The lottery ticket hypothesis: Finding sparse, trainable neural networks,” in International Conference on Learning Representations (ICLR), 2019.
  67. Z. Liu, M. Sun, T. Zhou, G. Huang, and T. Darrell, “Rethinking the value of network pruning,” in International Conference on Learning Representations (ICLR), 2019.
Citations (3)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Summarize

Paper to Video (Beta)

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up