Papers
Topics
Authors
Recent
Search
2000 character limit reached

Assessing Neural Network Robustness via Adversarial Pivotal Tuning

Published 17 Nov 2022 in cs.CV, cs.CR, and cs.LG | (2211.09782v2)

Abstract: The robustness of image classifiers is essential to their deployment in the real world. The ability to assess this resilience to manipulations or deviations from the training data is thus crucial. These modifications have traditionally consisted of minimal changes that still manage to fool classifiers, and modern approaches are increasingly robust to them. Semantic manipulations that modify elements of an image in meaningful ways have thus gained traction for this purpose. However, they have primarily been limited to style, color, or attribute changes. While expressive, these manipulations do not make use of the full capabilities of a pretrained generative model. In this work, we aim to bridge this gap. We show how a pretrained image generator can be used to semantically manipulate images in a detailed, diverse, and photorealistic way while still preserving the class of the original image. Inspired by recent GAN-based image inversion methods, we propose a method called Adversarial Pivotal Tuning (APT). Given an image, APT first finds a pivot latent space input that reconstructs the image using a pretrained generator. It then adjusts the generator's weights to create small yet semantic manipulations in order to fool a pretrained classifier. APT preserves the full expressive editing capabilities of the generative model. We demonstrate that APT is capable of a wide range of class-preserving semantic image manipulations that fool a variety of pretrained classifiers. Finally, we show that classifiers that are robust to other benchmarks are not robust to APT manipulations and suggest a method to improve them. Code available at: https://captaine.github.io/apt/

Definition Search Book Streamline Icon: https://streamlinehq.com
References (47)
  1. Image2stylegan: How to embed images into the stylegan latent space? In Proceedings of the IEEE international conference on computer vision, pages 4432–4441, 2019.
  2. Image2stylegan++: How to edit the embedded images? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8296–8305, 2020.
  3. Advances in adversarial attacks and defenses in computer vision: A survey. IEEE Access, 9:155161–155196, 2021.
  4. Adef: An iterative algorithm to construct adversarial deformations. arXiv preprint arXiv:1804.07729, 2018.
  5. Strike (with) a pose: Neural networks are easily fooled by strange poses of familiar objects. arXiv preprint arXiv:1811.11553, 2018.
  6. data2vec: A general framework for self-supervised learning in speech, vision and language. arXiv preprint arXiv:2202.03555, 2022.
  7. Unrestricted adversarial examples via semantic manipulation. arXiv preprint arXiv:1904.06347, 2019.
  8. Adversarial patch. arXiv preprint arXiv:1712.09665, 2017.
  9. Inverting the generator of a generative adversarial network. IEEE transactions on neural networks and learning systems, 30(7):1967–1974, 2018.
  10. Evaluating robustness to context-sensitive feature perturbations of different granularities. arXiv preprint arXiv:2001.11055, 2020.
  11. A rotation and a translation suffice: Fooling cnns with simple transformations. arXiv preprint arXiv:1712.02779, 2017.
  12. Roger Fletcher. Practical methods of optimization. John Wiley & Sons, 2013.
  13. Achieving robustness in the wild via adversarial mixing with disentangled representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1211–1220, 2020.
  14. Vision models are more robust and fair when pretrained on uncurated images without supervision, 2022.
  15. Collaborative learning for faster stylegan embedding. arXiv preprint arXiv:2007.01758, 2020.
  16. Masked autoencoders are scalable vision learners. arXiv:2111.06377, 2021.
  17. Deep residual learning for image recognition. CoRR, abs/1512.03385, 2015.
  18. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15262–15271, 2021.
  19. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 6629–6640, Red Hook, NY, USA, 2017. Curran Associates Inc.
  20. Semantic adversarial examples. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 1614–1619, 2018.
  21. Semantic adversarial attacks: Parametric transformations that fool deep classifiers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4773–4783, 2019.
  22. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8110–8119, 2020.
  23. Perceptual adversarial robustness: Defense against unseen threat models. arXiv preprint arXiv:2006.12655, 2020.
  24. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11523–11532, 2022.
  25. Dual manifold adversarial robustness: Defense against lp and non-lp adversarial attacks. Advances in Neural Information Processing Systems, 33:3487–3498, 2020.
  26. Precise recovery of latent vectors from generative adversarial networks. arXiv preprint arXiv:1702.04782, 2017.
  27. Frequency-driven imperceptible adversarial attack on semantic similarity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15315–15324, 2022.
  28. Learning inverse mapping by autoencoder based generative adversarial nets. In International Conference on Neural Information Processing, pages 207–216. Springer, 2017.
  29. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017.
  30. Prime: A few primitives can boost robustness to common corruptions. arXiv preprint arXiv:2112.13547, 2021.
  31. Null-text inversion for editing real images using guided diffusion models, 2022.
  32. Invertible conditional gans for image editing. arXiv preprint arXiv:1611.06355, 2016.
  33. Robustness and generalization via generative adversarial training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15711–15720, 2021.
  34. Semanticadv: Generating adversarial examples via attribute-conditioned image editing. In European Conference on Computer Vision, pages 19–37. Springer, 2020.
  35. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
  36. Pivotal tuning for latent-based editing of real images. ACM Trans. Graph., 2021.
  37. High-resolution image synthesis with latent diffusion models, 2021.
  38. Projected gans converge faster. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
  39. Stylegan-xl: Scaling stylegan to large diverse datasets, 2022.
  40. Colorfool: Semantic adversarial colorization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1151–1160, 2020.
  41. Constructing unrestricted adversarial examples with generative models. In Advances in Neural Information Processing Systems, pages 8312–8323, 2018.
  42. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
  43. Deepfakes and beyond: A survey of face manipulation and fake detection. Information Fusion, 64:131–148, 2020.
  44. Spatially transformed adversarial examples. In International Conference on Learning Representations, 2018.
  45. Towards feature space adversarial attack. arXiv preprint arXiv:2004.12385, 2020.
  46. The unreasonable effectiveness of deep features as a perceptual metric, 2018.
  47. Understanding the robustness in vision transformers. In International Conference on Machine Learning (ICML), 2022.
Citations (1)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up