Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
166 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Tight Certification of Adversarially Trained Neural Networks via Nonconvex Low-Rank Semidefinite Relaxations (2211.17244v3)

Published 30 Nov 2022 in cs.LG, math.OC, and stat.ML

Abstract: Adversarial training is well-known to produce high-quality neural network models that are empirically robust against adversarial perturbations. Nevertheless, once a model has been adversarially trained, one often desires a certification that the model is truly robust against all future attacks. Unfortunately, when faced with adversarially trained models, all existing approaches have significant trouble making certifications that are strong enough to be practically useful. Linear programming (LP) techniques in particular face a "convex relaxation barrier" that prevent them from making high-quality certifications, even after refinement with mixed-integer linear programming (MILP) and branch-and-bound (BnB) techniques. In this paper, we propose a nonconvex certification technique, based on a low-rank restriction of a semidefinite programming (SDP) relaxation. The nonconvex relaxation makes strong certifications comparable to much more expensive SDP methods, while optimizing over dramatically fewer variables comparable to much weaker LP methods. Despite nonconvexity, we show how off-the-shelf local optimization algorithms can be used to achieve and to certify global optimality in polynomial time. Our experiments find that the nonconvex relaxation almost completely closes the gap towards exact certification of adversarially trained models.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (37)
  1. Efficient neural network verification via layer-based semidefinite relaxations and linear cuts. In IJCAI, pp.  2184–2190, 2021.
  2. The non-convex Burer-Monteiro approach works on smooth semidefinite programs. In Advances in Neural Information Processing Systems, pp. 2757–2765, 2016.
  3. Deterministic guarantees for Burer-Monteiro factorizations of smooth semidefinite programs. Communications on Pure and Applied Mathematics, 73(3):581–608, 2020.
  4. A nonlinear programming algorithm for solving semidefinite programs via low-rank factorization. Mathematical Programming, 95(2):329–357, 2003.
  5. Local minima and convergence in low-rank semidefinite programming. Mathematical Programming, 103(3):427–444, 2005.
  6. Rank-two relaxation heuristics for max-cut and other binary quadratic programs. SIAM Journal on Optimization, 12(2):503–521, 2002.
  7. Knitro: An integrated package for nonlinear optimization. In Large-scale nonlinear optimization, pp.  35–59. Springer, 2006.
  8. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp), pp. 39–57. IEEE, 2017.
  9. Lagrangian duality in 3d slam: Verification techniques and optimal solutions. In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp.  125–132. IEEE, 2015.
  10. Certified adversarial robustness via randomized smoothing. In International Conference on Machine Learning, pp. 1310–1320, 2019.
  11. Enabling certification of verification-agnostic networks via memory-efficient semidefinite programming. In Advances in Neural Information Processing Systems, 2020.
  12. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015. URL http://arxiv.org/abs/1412.6572.
  13. Reluplex: An efficient SMT solver for verifying deep neural networks. In International Conference on Computer Aided Verification, pp.  97–117. Springer, 2017.
  14. Adversarial machine learning at scale. arXiv preprint arXiv:1611.01236, 2016.
  15. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
  16. MOSEK, A. The MOSEK optimization toolbox for MATLAB manual, 2019. URL https://docs.mosek.com/9.0/toolbox.pdf.
  17. Numerical optimization. Springer Science & Business Media, 2006.
  18. The burer-monteiro sdp method can fail even above the barvinok-pataki bound. In Advances in Neural Information Processing Systems, 2022.
  19. Certified defenses against adversarial examples. In International Conference on Learning Representations, 2018a.
  20. Semidefinite relaxations for certifying robustness to adversarial examples. Advances in Neural Information Processing Systems, 31, 2018b.
  21. Rise: An incremental trust-region method for robust online sparse least-squares estimation. IEEE Transactions on Robotics, 30(5):1091–1108, 2014.
  22. Se-sync: A certifiably correct algorithm for synchronization over the special euclidean group. The International Journal of Robotics Research, 38(2-3):95–125, 2019.
  23. A convex relaxation barrier to tight robustness verification of neural networks. Advances in Neural Information Processing Systems, 32, 2019.
  24. Adversarial training for free! Advances in Neural Information Processing Systems, 32, 2019.
  25. Evaluating robustness of neural networks with mixed integer programming. In International Conference on Learning Representations, 2019.
  26. Rank optimality for the burer–monteiro factorization. SIAM journal on Optimization, 30(3):2577–2602, 2020.
  27. Beta-crown: Efficient bound propagation with per-neuron split constraints for neural network robustness verification. Advances in Neural Information Processing Systems, 34:29909–29921, 2021.
  28. Towards fast computation of certified robustness for relu networks. In International Conference on Machine Learning, pp. 5276–5285. PMLR, 2018a.
  29. Evaluating the robustness of neural networks: An extreme value theory approach. arXiv preprint arXiv:1801.10578, 2018b.
  30. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International Conference on Machine Learning, pp. 5286–5295, 2018.
  31. Fast is better than free: Revisiting adversarial training. In International Conference on Learning Representations, 2019.
  32. Fast and complete: Enabling complete neural network verification with rapid and massively parallel incomplete verifiers. In International Conference on Learning Representation (ICLR), 2021.
  33. Efficient neural network robustness certification with general activation functions. Advances in neural information processing systems, 31, 2018a.
  34. Efficient neural network robustness certification with general activation functions. Advances in neural information processing systems, 31, 2018b.
  35. General cutting planes for bound-propagation-based neural network verification. Advances in Neural Information Processing Systems, 2022.
  36. Zhang, R. Y. On the tightness of semidefinite relaxations for certifying robustness to adversarial examples. In Advances in Neural Information Processing Systems, 2020.
  37. Zhang, R. Y. Improved global guarantees for the nonconvex burer–monteiro factorization via rank overparameterization. arXiv preprint arXiv:2207.01789, 2022.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now