Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Data-Dependent Stability Analysis of Adversarial Training (2401.03156v1)

Published 6 Jan 2024 in cs.LG

Abstract: Stability analysis is an essential aspect of studying the generalization ability of deep learning, as it involves deriving generalization bounds for stochastic gradient descent-based training algorithms. Adversarial training is the most widely used defense against adversarial example attacks. However, previous generalization bounds for adversarial training have not included information regarding the data distribution. In this paper, we fill this gap by providing generalization bounds for stochastic gradient descent-based adversarial training that incorporate data distribution information. We utilize the concepts of on-average stability and high-order approximate Lipschitz conditions to examine how changes in data distribution and adversarial budget can affect robust generalization gaps. Our derived generalization bounds for both convex and non-convex losses are at least as good as the uniform stability-based counterparts which do not include data distribution information. Furthermore, our findings demonstrate how distribution shifts from data poisoning attacks can impact robust generalization.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (48)
  1. A convergence theory for deep learning via over-parameterization. In International Conference on Machine Learning, 242–252.
  2. Stability of stochastic gradient descent on nonsmooth convex losses. Advances in Neural Information Processing Systems, 33: 4381–4391.
  3. Evasion attacks against machine learning at test time. In Joint European conference on machine learning and knowledge discovery in databases, 387–402. Springer.
  4. Stability and generalization. The Journal of Machine Learning Research, 2: 499–526.
  5. Robust overfitting may be mitigated by properly learned smoothening. In International Conference on Learning Representations.
  6. Sparsity Winning Twice: Better Robust Generaliztion from More Efficient Training. arXiv preprint arXiv:2202.09844.
  7. Gradient descent finds global minima of deep neural networks. In International conference on machine learning, 1675–1685.
  8. Train simultaneously, generalize better: Stability of gradient-based minimax learners. In International Conference on Machine Learning, 3174–3185.
  9. Learning to confuse: generating training time adversarial data with auto-encoder. Advances in Neural Information Processing Systems, 32.
  10. Adversarial examples make strong poisons. arXiv preprint arXiv:2106.10807.
  11. Robust Unlearnable Examples: Protecting Data Against Adversarial Learning. arXiv preprint arXiv:2203.14533.
  12. Achieving Optimal Adversarial Accuracy for Adversarial Deep Learning Using Stackelberg Games. Acta Mathematica Scientia, 42B(6): 2399–2418.
  13. Escaping from saddle points—online stochastic gradient for tensor decomposition. In Conference on learning theory, 797–842.
  14. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572.
  15. Improving robustness using generated data. Advances in Neural Information Processing Systems, 34: 4218–4233.
  16. Train faster, generalize better: Stability of stochastic gradient descent. In International conference on machine learning, 1225–1234.
  17. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770–778.
  18. Unlearnable examples: Making personal data unexploitable. arXiv preprint arXiv:2101.04898.
  19. Adversarial examples are not bugs, they are features. Advances in neural information processing systems, 32.
  20. Learning multiple layers of features from tiny images. Technical Report TR-2009.
  21. Data-dependent stability of stochastic gradient descent. In International Conference on Machine Learning, 2815–2824.
  22. Tiny imagenet visual recognition challenge. CS 231N, 7(7): 3.
  23. Why robust generalization in deep learning is difficult: Perspective of expressive power. Advances in Neural Information Processing Systems, 35: 4370–4384.
  24. On the loss landscape of adversarial training: Identifying challenges and how to overcome them. Advances in Neural Information Processing Systems, 33: 21476–21487.
  25. Going Grayscale: The Road to Understanding and Improving Unlearnable Examples. arXiv preprint arXiv:2111.13244.
  26. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083.
  27. Deepfool: a simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2574–2582.
  28. Robust stochastic approximation approach to stochastic programming. SIAM Journal on Optimization, 19(4): 1574–1609.
  29. Reading digits in natural images with unsupervised feature learning. In NIPS DLW.
  30. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and pattern recognition, 427–436.
  31. Transferable Unlearnable Examples. arXiv preprint arXiv:2210.10114.
  32. Overfitting in adversarially robust deep learning. In International Conference on Machine Learning, 8093–8104.
  33. Adversarially robust generalization requires more data. Advances in neural information processing systems, 31.
  34. Understanding adversarial training: Increasing local stability of neural nets through robust optimization. arXiv preprint arXiv:1511.05432.
  35. Learnability, stability and uniform convergence. The Journal of Machine Learning Research, 11: 2635–2670.
  36. Certifying some distributional robustness with principled adversarial training. arXiv preprint arXiv:1710.10571.
  37. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199.
  38. Can Adversarial Training Be Manipulated By Non-Robust Features? arXiv preprint arXiv:2201.13329.
  39. Better safe than sorry: Preventing delusive adversaries with adversarial training. Advances in Neural Information Processing Systems, 34: 16209–16225.
  40. Improving adversarial robustness requires revisiting misclassified examples. In International Conference on Learning Representations.
  41. Better diffusion models further improve adversarial training. arXiv preprint arXiv:2302.04638.
  42. Fooling Adversarial Training with Inducing Noise. arXiv preprint arXiv:2111.10130.
  43. Adversarial weight perturbation helps robust generalization. Advances in Neural Information Processing Systems, 33: 2958–2969.
  44. Stability analysis and generalization bounds of adversarial training. arXiv preprint arXiv:2210.00960.
  45. On the algorithmic stability of adversarial training. Advances in Neural Information Processing Systems, 34: 26523–26535.
  46. Understanding robust overfitting of adversarial training and beyond. In International Conference on Machine Learning, 25595–25610.
  47. Availability attacks create shortcuts. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2367–2376.
  48. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning, 7472–7482.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets