Papers
Topics
Authors
Recent
Search
2000 character limit reached

SpecFormer: Guarding Vision Transformer Robustness via Maximum Singular Value Penalization

Published 2 Jan 2024 in cs.CV and cs.LG | (2402.03317v2)

Abstract: Vision Transformers (ViTs) are increasingly used in computer vision due to their high performance, but their vulnerability to adversarial attacks is a concern. Existing methods lack a solid theoretical basis, focusing mainly on empirical training adjustments. This study introduces SpecFormer, tailored to fortify ViTs against adversarial attacks, with theoretical underpinnings. We establish local Lipschitz bounds for the self-attention layer and propose the Maximum Singular Value Penalization (MSVP) to precisely manage these bounds By incorporating MSVP into ViTs' attention layers, we enhance the model's robustness without compromising training efficiency. SpecFormer, the resulting model, outperforms other state-of-the-art models in defending against adversarial attacks, as proven by experiments on CIFAR and ImageNet datasets. Code is released at https://github.com/microsoft/robustlearn.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (64)
  1. Xcit: Cross-covariance image transformers. Advances in neural information processing systems, 34:20014–20027, 2021.
  2. Vivit: A video vision transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6836–6846, 2021.
  3. Recent advances in adversarial training for adversarial robustness. In IJCAI survey track, 2021a.
  4. Are transformers more robust than cnns? Advances in Neural Information Processing Systems, 34:26831–26843, 2021b.
  5. Analysis of representations for domain adaptation. Advances in neural information processing systems, 19, 2006.
  6. Understanding robustness of transformers for image classification. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10231–10241, 2021.
  7. Numerical analysis. Cengage learning, 2015.
  8. End-to-end object detection with transformers. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16, pages 213–229. Springer, 2020.
  9. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp), pages 39–57. Ieee, 2017.
  10. Ead: elastic-net attacks to deep neural networks via adversarial examples. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
  11. Parseval networks: Improving robustness to adversarial examples. In International conference on machine learning, pages 854–863. PMLR, 2017.
  12. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning, pages 2206–2216. PMLR, 2020.
  13. Lipschitz normalization for self-attention layers with application to graph neural networks. In International Conference on Machine Learning, pages 2456–2466. PMLR, 2021.
  14. A light recipe to train robust vision transformers. arXiv preprint arXiv:2209.07399, 2022.
  15. Scaling vision transformers to 22 billion parameters. arXiv preprint arXiv:2302.05442, 2023.
  16. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  17. An image is worth 16x16 words: Transformers for image recognition at scale. In International conference on learning representations (ICLR), 2020.
  18. Convit: Improving vision transformers with soft convolutional inductive biases. In International Conference on Machine Learning, pages 2286–2296. PMLR, 2021.
  19. H. Federer. Geometric Measure Theory. Classics in Mathematics. Springer Berlin Heidelberg, 1969. ISBN 9783642620102.
  20. Detecting adversarial samples from artifacts. arXiv preprint arXiv:1703.00410, 2017.
  21. Patch-fool: Are vision transformers always robust against adversarial perturbations? In International conference on learning representations (ICLR), 2022.
  22. Explaining and harnessing adversarial examples. In International conference on learning representations (ICLR), 2015.
  23. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  24. Formal guarantees on the robustness of a classifier against adversarial manipulation. Advances in neural information processing systems, 30, 2017.
  25. Jeremy Howard. Imagenette. https://github.com/fastai/imagenette, 2019.
  26. Limitations of the lipschitz constant as a defense against adversarial examples. In ECML PKDD 2018 Workshops: Nemesis 2018, UrbReas 2018, SoGood 2018, IWAISe 2018, and Green Data Mining 2018, Dublin, Ireland, September 10-14, 2018, Proceedings 18, pages 16–29. Springer, 2019.
  27. The lipschitz constant of self-attention. In International Conference on Machine Learning, pages 5562–5571. PMLR, 2021.
  28. Learning multiple layers of features from tiny images. 2009.
  29. Globally-robust neural networks. In International Conference on Machine Learning, pages 6212–6222. PMLR, 2021.
  30. Backdoor learning: A survey. IEEE Transactions on Neural Networks and Learning Systems, 2022.
  31. Give me your attention: Dot-product attention considered harmful for adversarial patch robustness. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15234–15243, 2022.
  32. Towards deep learning models resistant to adversarial attacks. In International conference on learning representations (ICLR), 2018.
  33. RV Mises and Hilda Pollaczek-Geiringer. Praktische verfahren der gleichungsauflösung. ZAMM-Journal of Applied Mathematics and Mechanics/Zeitschrift für Angewandte Mathematik und Mechanik, 9(1):58–77, 1929.
  34. When adversarial training meets vision transformers: Recipes from training to architecture. In Advances in neural information processing systems (NeurIPS), 2022.
  35. James A. Murdock. Perturbations: Theory and Methods. Society for Industrial and Applied Mathematics, 1999. doi: 10.1137/1.9781611971095. URL https://epubs.siam.org/doi/abs/10.1137/1.9781611971095.
  36. Intriguing properties of vision transformers. Advances in Neural Information Processing Systems, 34:23296–23308, 2021.
  37. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 427–436, 2015.
  38. Robustness and accuracy could be reconcilable by (proper) definition. In International Conference on Machine Learning, pages 17258–17277. PMLR, 2022.
  39. Towards the science of security and privacy in machine learning. arXiv preprint arXiv:1611.03814, 2016a.
  40. Distillation as a defense to adversarial perturbations against deep neural networks. In 2016 IEEE symposium on security and privacy (SP), pages 582–597. IEEE, 2016b.
  41. Vision transformers are robust learners. In Proceedings of the AAAI Conference on Artificial Intelligence, number 2, pages 2071–2081, 2022.
  42. Lipsformer: Introducing lipschitz continuity to vision transformers. In International conference on Learning Representations (ICLR), 2023.
  43. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  44. On the adversarial robustness of vision transformers. arXiv preprint arXiv:2103.15670, 2021.
  45. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
  46. Segmenter: Transformer for semantic segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7262–7272, 2021.
  47. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
  48. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1–9, 2015.
  49. On layer normalizations and residual connections in transformers. arXiv preprint arXiv:2206.00330, 2022.
  50. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pages 10347–10357. PMLR, 2021a.
  51. Going deeper with image transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 32–42, 2021b.
  52. Ensemble adversarial training: Attacks and defenses. In Ensemble adversarial training: Attacks and defenses, 2018.
  53. Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
  54. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  55. Exploring robust features for improving adversarial robustness. arXiv preprint arXiv:2309.04650, 2023.
  56. Deepnet: Scaling transformers to 1,000 layers. arXiv preprint arXiv:2203.00555, 2022a.
  57. Can cnns be more robust than transformers? arXiv preprint arXiv:2206.03452, 2022b.
  58. Mitigating adversarial effects through randomization. In International conference on learning representations (ICLR), 2018.
  59. Cdtrans: Cross-domain transformer for unsupervised domain adaptation. In International conference on learning representations (ICLR), 2022.
  60. Spectral norm regularization for improving the generalizability of deep learning. arXiv preprint arXiv:1705.10941, 2017.
  61. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning, pages 7472–7482. PMLR, 2019a.
  62. Recurjac: An efficient recursive algorithm for bounding jacobian matrix of neural networks and its applications. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 5757–5764, 2019b.
  63. Improving the robustness of deep neural networks via stability training. In Proceedings of the ieee conference on computer vision and pattern recognition, pages 4480–4488, 2016.
  64. Understanding the robustness in vision transformers. In International Conference on Machine Learning, pages 27378–27394. PMLR, 2022.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 2 tweets with 10 likes about this paper.

Sign Up for Free