Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

CLIP2Protect: Protecting Facial Privacy using Text-Guided Makeup via Adversarial Latent Search (2306.10008v2)

Published 16 Jun 2023 in cs.CV, cs.CR, and cs.LG

Abstract: The success of deep learning based face recognition systems has given rise to serious privacy concerns due to their ability to enable unauthorized tracking of users in the digital world. Existing methods for enhancing privacy fail to generate naturalistic images that can protect facial privacy without compromising user experience. We propose a novel two-step approach for facial privacy protection that relies on finding adversarial latent codes in the low-dimensional manifold of a pretrained generative model. The first step inverts the given face image into the latent space and finetunes the generative model to achieve an accurate reconstruction of the given image from its latent code. This step produces a good initialization, aiding the generation of high-quality faces that resemble the given identity. Subsequently, user-defined makeup text prompts and identity-preserving regularization are used to guide the search for adversarial codes in the latent space. Extensive experiments demonstrate that faces generated by our approach have stronger black-box transferability with an absolute gain of 12.06% over the state-of-the-art facial privacy protection approach under the face verification task. Finally, we demonstrate the effectiveness of the proposed approach for commercial face recognition systems. Our code is available at https://github.com/fahadshamshad/Clip2Protect.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (79)
  1. Over-exposed? privacy patterns and considerations in online and mobile photo sharing. In Proceedings of the SIGCHI conference on Human factors in computing systems, pages 357–366, 2007.
  2. State-of-the-art in the architecture, methods and applications of stylegan. In Computer Graphics Forum, volume 41, pages 591–611. Wiley Online Library, 2022.
  3. Unrestricted adversarial examples via semantic manipulation. arXiv preprint arXiv:1904.06347, 2019.
  4. Perceptual indistinguishability-net (pi-net): Facial image obfuscation with manipulable semantics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6478–6487, 2021.
  5. Mobilefacenets: Efficient cnns for accurate real-time face verification on mobile devices. In Chinese Conference on Biometric Recognition, pages 428–438. Springer, 2018.
  6. Lowkey: Leveraging adversarial attacks to protect social media users from facial recognition. In International Conference on Learning Representations, 2020.
  7. Differentially private facial obfuscation via generative adversarial networks. Future Generation Computer Systems, 129:358–379, 2022.
  8. Fast geometrically-perturbed adversarial faces. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1979–1988. IEEE, 2019.
  9. Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4690–4699, 2019.
  10. Boosting adversarial attacks with momentum. In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’18), pages 9185–9193, 2018.
  11. Evading defenses to transferable adversarial examples by translation-invariant attacks. In Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’19), pages 4312–4321, 2019.
  12. The elements of end-to-end deep face recognition: A survey of recent advances. ACM Computing Surveys (CSUR), 54(10s):1–42, 2022.
  13. A survey of vision-language pre-trained models. arXiv preprint arXiv:2202.10936, 2022.
  14. Clipdraw: Exploring text-to-drawing synthesis through language-image encoders. arXiv preprint arXiv:2106.14843, 2021.
  15. Generative adversarial networks. CoRR, abs/1406.2661, 2014.
  16. Ladn: Local adversarial disentangling network for facial makeup and de-makeup. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10481–10490, 2019.
  17. Dodging attack using carefully crafted natural makeup. arXiv preprint arXiv:2109.06467, 2021.
  18. Rebecca Heilweil. The world’s scariest facial recognition company explained. Vox, May, 8, 2020.
  19. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
  20. Kashmir Hill. The secretive company that might end privacy as we know it. The New York Times, 18:2020, 2020.
  21. Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132–7141, 2018.
  22. Protecting facial privacy: Generating adversarial identity masks via style-robust makeup transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15014–15023, 2022.
  23. Labeled faces in the wild: A database forstudying face recognition in unconstrained environments. In Workshop on faces in’Real-Life’Images: detection, alignment, and recognition, 2008.
  24. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125–1134, 2017.
  25. Adversarial image translation: Unrestricted adversarial examples in face recognition systems. arXiv preprint arXiv:1905.03421, 2019.
  26. Progressive growing of gans for improved quality, stability, and variation. In International Conference on Learning Representations, 2018.
  27. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8110–8119, 2020.
  28. Diffusionclip: Text-guided diffusion models for robust image manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2426–2435, 2022.
  29. Advhat: Real-world adversarial attack on arcface face id system. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 819–826. IEEE, 2021.
  30. Unnoticeable synthetic face replacement for image privacy protection. Neurocomputing, 457:322–333, 2021.
  31. Effective de-identification generative adversarial network for face anonymization. In Proceedings of the 29th ACM International Conference on Multimedia, pages 3182–3191, 2021.
  32. Clipstyler: Image style transfer with a single text condition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18062–18071, 2022.
  33. Deepblur: A simple and effective method for natural image obfuscation. arXiv preprint arXiv:2104.02655, 1, 2021.
  34. Anonymousnet: Natural face de-identification with measurable privacy. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 0–0, 2019.
  35. Towards transferable unrestricted adversarial examples with minimum changes. arXiv preprint arXiv:2201.01102, 2022.
  36. Face detection and encryption for privacy preserving in surveillance video. In Chinese Conference on Pattern Recognition and Computer Vision (PRCV), pages 162–172. Springer, 2018.
  37. Towards deep learning models resistant to adversarial attacks. In Proceedings of the 6th International Conference on Learning Representations (ICLR’18), 2018.
  38. Privacy–enhancing face biometrics: A comprehensive survey. IEEE Transactions on Information Forensics and Security, 2021.
  39. Unrestricted black-box adversarial attack using gan with limited queries. arXiv preprint arXiv:2208.11613, 2022.
  40. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741, 2021.
  41. Faceless person recognition: Privacy implications in social media. In European Conference on Computer Vision, pages 19–35. Springer, 2016.
  42. Adversarial image perturbation for privacy protection a game theory perspective. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 1491–1500. IEEE, 2017.
  43. Deep face recognition. 2015.
  44. Styleclip: Text-driven manipulation of stylegan imagery. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2085–2094, 2021.
  45. Face recognition accuracy of forensic examiners, superrecognizers, and face recognition algorithms. Proceedings of the National Academy of Sciences, 115(24):6171–6176, 2018.
  46. Robustness and generalization via generative adversarial training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15711–15720, 2021.
  47. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
  48. Deep learning for understanding faces: Machines may be just as good, or better, than humans. IEEE Signal Processing Magazine, 35(1):66–83, 2018.
  49. Pivotal tuning for latent-based editing of real images. ACM Transactions on Graphics (TOG), 42(1):1–13, 2022.
  50. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
  51. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815–823, 2015.
  52. Does a face mask protect my privacy?: Deep learning to predict protected attributes from masked face images. In Australasian Joint Conference on Artificial Intelligence, pages 91–102. Springer, 2022.
  53. Fawkes: Protecting privacy against unauthorized deep learning models. In 29th USENIX security symposium (USENIX Security 20), pages 1589–1604, 2020.
  54. A general framework for adversarial examples with objectives. ACM Transactions on Privacy and Security (TOPS), 22(3):1–30, 2019.
  55. Constructing unrestricted adversarial examples with generative models. Advances in Neural Information Processing Systems, 31, 2018.
  56. Natural and effective obfuscation by head inpainting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5050–5059, 2018.
  57. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014.
  58. Fairness and privacy preservation for facial images: Gan-based methods. Computers & Security, 122:102902, 2022.
  59. Designing an encoder for stylegan image manipulation. ACM Transactions on Graphics (TOG), 40(4):1–14, 2021.
  60. Clip-nerf: Text-and-image driven manipulation of neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3835–3844, 2022.
  61. Deep face recognition: A survey. Neurocomputing, 429:215–244, 2021.
  62. Gender obfuscation through face morphing. In 2021 IEEE International Workshop on Biometrics and Forensics (IWBF), pages 1–6. IEEE, 2021.
  63. Face recognition in real-world surveillance videos with deep learning method. In 2017 2nd international conference on image, vision and computing (icivc), pages 239–243. IEEE, 2017.
  64. Privacy-preserving face recognition in the frequency domain. 2022.
  65. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600–612, 2004.
  66. Hairclip: Design your hair by text and reference image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18072–18081, 2022.
  67. Sok: Anti-facial recognition technology. arXiv preprint arXiv:2112.04558, 2021.
  68. Spatially transformed adversarial examples. arXiv preprint arXiv:1801.02612, 2018.
  69. Improving transferability of adversarial patches on face recognition with generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11845–11854, 2021.
  70. Towards face encryption by generating adversarial identity masks. In Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV’21), pages 3897–3907, 2021.
  71. Adv-makeup: A new imperceptible and transferable attack on face recognition. In Proceedings of the 30th International Joint Conference on Artificial Intelligence (IJCAI’21), pages 1252–1258, 2021.
  72. Adv-makeup: A new imperceptible and transferable attack on face recognition. arXiv preprint arXiv:2105.03162, 2021.
  73. Natural color fool: Towards boosting black-box unrestricted attacks. arXiv preprint arXiv:2210.02041, 2022.
  74. Adversarial privacy-preserving filter. In Proceedings of the 28th ACM International Conference on Multimedia, pages 1423–1431, 2020.
  75. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE signal processing letters, 23(10):1499–1503, 2016.
  76. Towards large yet imperceptible adversarial image perturbations with perceptual color distance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1039–1048, 2020.
  77. Opom: Customized invisible cloak towards face privacy protection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
  78. Personal privacy protection via irrelevant faces tracking and pixelation in video live streaming. IEEE Transactions on Information Forensics and Security, 16:1088–1103, 2020.
  79. Generating adversarial examples by makeup attacks on face recognition. In 2019 IEEE International Conference on Image Processing (ICIP), pages 2516–2520. IEEE, 2019.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Fahad Shamshad (21 papers)
  2. Muzammal Naseer (67 papers)
  3. Karthik Nandakumar (57 papers)
Citations (20)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now