Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Uniformly Stable Algorithms for Adversarial Training and Beyond (2405.01817v1)

Published 3 May 2024 in cs.LG

Abstract: In adversarial machine learning, neural networks suffer from a significant issue known as robust overfitting, where the robust test accuracy decreases over epochs (Rice et al., 2020). Recent research conducted by Xing et al.,2021; Xiao et al., 2022 has focused on studying the uniform stability of adversarial training. Their investigations revealed that SGD-based adversarial training fails to exhibit uniform stability, and the derived stability bounds align with the observed phenomenon of robust overfitting in experiments. This motivates us to develop uniformly stable algorithms specifically tailored for adversarial training. To this aim, we introduce Moreau envelope-$\mathcal{A}$, a variant of the Moreau Envelope-type algorithm. We employ a Moreau envelope function to reframe the original problem as a min-min problem, separating the non-strong convexity and non-smoothness of the adversarial loss. Then, this approach alternates between solving the inner and outer minimization problems to achieve uniform stability without incurring additional computational overhead. In practical scenarios, we show the efficacy of ME-$\mathcal{A}$ in mitigating the issue of robust overfitting. Beyond its application in adversarial training, this represents a fundamental result in uniform stability analysis, as ME-$\mathcal{A}$ is the first algorithm to exhibit uniform stability for weakly-convex, non-smooth problems.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (56)
  1. A characterization of semi-supervised adversarially robust pac learnability. Advances in Neural Information Processing Systems, 35:23646–23659, 2022a.
  2. Improved generalization bounds for adversarially robust learning. Journal of Machine Learning Research, 23(175):1–31, 2022b.
  3. Adversarial learning guarantees for linear hypotheses and neural networks. In International Conference on Machine Learning, pp. 431–441. PMLR, 2020.
  4. Private stochastic convex optimization with optimal rates. Advances in neural information processing systems, 32, 2019.
  5. Stability of stochastic gradient descent on nonsmooth convex losses. Advances in Neural Information Processing Systems, 33:4381–4391, 2020.
  6. Stability and generalization. The Journal of Machine Learning Research, 2:499–526, 2002.
  7. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp), pp. 39–57. IEEE, 2017.
  8. Unlabeled data improves adversarial robustness. In Advances in Neural Information Processing Systems, pp. 11190–11201, 2019.
  9. Stability and convergence trade-off of iterative optimization algorithms. arXiv preprint arXiv:1804.01619, 2018.
  10. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning, pp. 2206–2216. PMLR, 2020.
  11. Pac-learning in the presence of evasion adversaries. arXiv preprint arXiv:1806.01471, 2018.
  12. Generalizable adversarial training via spectral normalization. In International Conference on Learning Representations, 2018.
  13. Generalization bounds for uniformly stable algorithms. Advances in Neural Information Processing Systems, 31, 2018.
  14. High probability generalization bounds for uniformly stable algorithms with nearly optimal rate. In Conference on Learning Theory, pp.  1270–1279. PMLR, 2019.
  15. Private stochastic convex optimization: optimal rates in linear time. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pp.  439–449, 2020.
  16. Theoretical investigation of generalization bounds for adversarial learning of deep neural networks. Journal of Statistical Theory and Practice, 15(2):1–28, 2021.
  17. Uncovering the limits of adversarial training against norm-bounded adversarial examples. arXiv preprint arXiv:2010.03593, 2020.
  18. Train faster, generalize better: Stability of stochastic gradient descent. In International Conference on Machine Learning, pp. 1225–1234. PMLR, 2016.
  19. Averaging weights leads to wider optima and better generalization. In 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018, pp.  876–885, 2018.
  20. Relationship between nonsmoothness in adversarial training, constraints of attacks, and flatness in the input space. IEEE Transactions on Neural Networks and Learning Systems, 2023.
  21. Adversarial risk bounds via function transformation. arXiv preprint arXiv:1810.09519, 2018.
  22. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pp. 1097–1105, 2012.
  23. Adversarial examples in the physical world. In Artificial intelligence safety and security, pp.  99–112. Chapman and Hall/CRC, 2018.
  24. Lei, Y. Stability and generalization of stochastic optimization with nonconvex and nonsmooth problems. In The Thirty Sixth Annual Conference on Learning Theory, pp. 191–227. PMLR, 2023.
  25. On the loss landscape of adversarial training: Identifying challenges and how to overcome them. Advances in Neural Information Processing Systems, 33:21476–21487, 2020.
  26. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018a.
  27. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018b.
  28. Vc classes are adversarially robustly learnable, but only improperly. In Conference on Learning Theory, pp.  2512–2530. PMLR, 2019.
  29. Deepfool: a simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  2574–2582, 2016.
  30. Moreau, J.-J. Proximité et dualité dans un espace hilbertien. Bulletin de la Société mathématique de France, 93:273–299, 1965.
  31. On the generalization analysis of adversarial learning. In International Conference on Machine Learning, pp. 16174–16196. PMLR, 2022.
  32. Robust stochastic approximation approach to stochastic programming. SIAM Journal on optimization, 19(4):1574–1609, 2009.
  33. Problem complexity and method efficiency in optimization. 1983.
  34. Nesterov, Y. Smooth minimization of non-smooth functions. Mathematical programming, 103:127–152, 2005.
  35. What is a good metric to study generalization of minimax learners? Advances in Neural Information Processing Systems, 35:38190–38203, 2022.
  36. The limitations of deep learning in adversarial settings. In 2016 IEEE European symposium on security and privacy (EuroS&P), pp.  372–387. IEEE, 2016.
  37. Fixing data augmentation to improve adversarial robustness. arXiv preprint arXiv:2103.01946, 2021.
  38. Overfitting in adversarially robust deep learning. In International Conference on Machine Learning, pp. 8093–8104. PMLR, 2020.
  39. Rockafellar, R. T. Monotone operators and the proximal point algorithm. SIAM j. control optim., 14(5):877–898, August 1976.
  40. A finite sample distribution-free performance bound for local discrimination rules. The Annals of Statistics, pp.  506–514, 1978.
  41. Adversarially robust generalization requires more data. In Advances in Neural Information Processing Systems, pp. 5014–5026, 2018.
  42. Intriguing properties of neural networks. In 2nd International Conference on Learning Representations, ICLR 2014, 2014.
  43. On adaptive attacks to adversarial example defenses. arXiv preprint arXiv:2002.08347, 2020.
  44. Stability and generalization for markov chain stochastic gradient methods. Advances in Neural Information Processing Systems, 35:37735–37748, 2022.
  45. Adversarial rademacher complexity of deep neural networks. arXiv preprint arXiv:2211.14966, 2022a.
  46. Stability analysis and generalization bounds of adversarial training. Advances in Neural Information Processing Systems, 35:15446–15459, 2022b.
  47. Adaptive smoothness-weighted adversarial training for multiple perturbations with its stability analysis. arXiv preprint arXiv:2210.00557, 2022c.
  48. Understanding adversarial robustness against on-manifold adversarial examples. arXiv preprint arXiv:2210.00430, 2022d.
  49. Smoothed-sgdmax: A stability-inspired algorithm to improve adversarial generalization. In NeurIPS ML Safety Workshop, 2022e.
  50. Pac-bayesian spectrally-normalized bounds for adversarially robust generalization. Advances in Neural Information Processing Systems, 36:36305–36323, 2023.
  51. On the algorithmic stability of adversarial training. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=xz80iPFIjvG.
  52. Stability and differential privacy of stochastic gradient descent for pairwise learning with non-smooth loss. In International Conference on Artificial Intelligence and Statistics, pp.  2026–2034. PMLR, 2021.
  53. Rademacher complexity for adversarially robust generalization. In International Conference on Machine Learning, pp. 7085–7094. PMLR, 2019.
  54. Sharper analysis for minibatch stochastic proximal point methods: Stability, smoothness, and deviation. Journal of Machine Learning Research, 20:1–54, 2023.
  55. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107–115, 2021.
  56. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning, pp. 7472–7482. PMLR, 2019.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Jiancong Xiao (15 papers)
  2. Jiawei Zhang (529 papers)
  3. Zhi-Quan Luo (115 papers)
  4. Asuman Ozdaglar (102 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets