Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
144 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Multimodal Unlearnable Examples: Protecting Data against Multimodal Contrastive Learning (2407.16307v2)

Published 23 Jul 2024 in cs.MM and cs.CR

Abstract: Multimodal contrastive learning (MCL) has shown remarkable advances in zero-shot classification by learning from millions of image-caption pairs crawled from the Internet. However, this reliance poses privacy risks, as hackers may unauthorizedly exploit image-text data for model training, potentially including personal and privacy-sensitive information. Recent works propose generating unlearnable examples by adding imperceptible perturbations to training images to build shortcuts for protection. However, they are designed for unimodal classification, which remains largely unexplored in MCL. We first explore this context by evaluating the performance of existing methods on image-caption pairs, and they do not generalize effectively to multimodal data and exhibit limited impact to build shortcuts due to the lack of labels and the dispersion of pairs in MCL. In this paper, we propose Multi-step Error Minimization (MEM), a novel optimization process for generating multimodal unlearnable examples. It extends the Error-Minimization (EM) framework to optimize both image noise and an additional text trigger, thereby enlarging the optimized space and effectively misleading the model to learn the shortcut between the noise features and the text trigger. Specifically, we adopt projected gradient descent to solve the noise minimization problem and use HotFlip to approximate the gradient and replace words to find the optimal text trigger. Extensive experiments demonstrate the effectiveness of MEM, with post-protection retrieval results nearly half of random guessing, and its high transferability across different models. Our code is available on the https://github.com/thinwayliu/Multimodal-Unlearnable-Examples

Definition Search Book Streamline Icon: https://streamlinehq.com
References (86)
  1. Improving Adversarial Robustness via Channel-wise Activation Suppressing. In International Conference on Learning Representations.
  2. Cleanclip: Mitigating data poisoning attacks in multimodal contrastive learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 112–123.
  3. Representation learning: A review and new perspectives. PAMI 35, 8 (2013), 1798–1828.
  4. Nicholas Carlini and Andreas Terzis. 2021. Poisoning and backdooring contrastive learning. arXiv preprint arXiv:2106.09667 (2021).
  5. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 3558–3568.
  6. Universal Watermark Vaccine: Universal Adversarial Perturbations for Watermark Protection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops.
  7. A simple framework for contrastive learning of visual representations. In International conference on machine learning. PMLR, 1597–1607.
  8. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325 (2015).
  9. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05526 (2017).
  10. Learning a similarity metric discriminatively, with application to face verification. In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05), Vol. 1. IEEE, 539–546.
  11. Face Encryption via Frequency-Restricted Identity-Agnostic Attacks. In Proceedings of the 31st ACM International Conference on Multimedia.
  12. Hotflip: White-box adversarial examples for text classification. arXiv preprint arXiv:1712.06751 (2017).
  13. Privacy enhancing face obfuscation guided by semantic-aware attribution maps. 2023. Privacy-enhancing face obfuscation guided by semantic-aware attribution maps. IEEE Transactions on Information Forensics and Security (2023).
  14. Vse++: Improving visual-semantic embeddings with hard negatives. arXiv preprint arXiv:1707.05612 (2017).
  15. Re-thinking Data Availablity Attacks Against Deep Neural Networks. arXiv preprint arXiv:2305.10691 (2023).
  16. Learning to confuse: Generating training time adversarial data with auto-encoder. Advances in Neural Information Processing Systems 32 (2019).
  17. Adversarial examples make strong poisons. Advances in Neural Information Processing Systems 34 (2021), 30339–30351.
  18. Robust unlearnable examples: Protecting data against adversarial learning. arXiv preprint arXiv:2203.14533 (2022).
  19. Backdoor defense via adaptively splitting poisoned dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 4005–4014.
  20. Backdoor Attack with Sparse and Invisible Trigger. IEEE Transactions on Information Forensics and Security (2024).
  21. A survey on transferability of adversarial examples across deep neural networks. arXiv preprint arXiv:2310.17626 (2023).
  22. Are vision transformers robust to patch perturbations?. In European Conference on Computer Vision. Springer, 404–421.
  23. Effective and Efficient Vote Attack on Capsule Networks. In International Conference on Learning Representations (ICLR).
  24. Segpgd: An effective and efficient adversarial attack for evaluating and boosting segmentation robustness. In European Conference on Computer Vision. Springer, 308–325.
  25. Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access 7 (2019), 47230–47244.
  26. Isolation and Induction: Training Robust Deep Neural Networks against Model Stealing Attacks. In Proceedings of the 31st ACM International Conference on Multimedia.
  27. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR’06), Vol. 2. IEEE, 1735–1742.
  28. Indiscriminate poisoning attacks on unsupervised contrastive learning. arXiv preprint arXiv:2202.11202 (2022).
  29. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 9729–9738.
  30. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770–778.
  31. Unlearnable examples: Making personal data unexploitable. ICLR (2021).
  32. Texture Re-scalable Universal Adversarial Perturbation. IEEE Transactions on Information Forensics and Security (2024). https://doi.org/10.1109/TIFS.2024.3416030
  33. AdvFilter: predictive perturbation-aware filtering against adversarial attack via multi-domain learning. In Proceedings of the 29th ACM International Conference on Multimedia. 395–403.
  34. Personalization as a shortcut for few-shot backdoor attack against text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 21169–21178.
  35. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning. PMLR, 4904–4916.
  36. Revisiting and exploring efficient fast adversarial training via law: Lipschitz regularization and auto weight averaging. IEEE Transactions on Information Forensics and Security (2024).
  37. Improving fast adversarial training with prior-guided knowledge. IEEE Transactions on Pattern Analysis and Machine Intelligence (2024).
  38. LAS-AT: adversarial training with learnable attack strategy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13398–13408.
  39. Andrej Karpathy and Li Fei-Fei. 2015. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition. 3128–3137.
  40. Attribute and simile classifiers for face verification. In 2009 IEEE 12th international conference on computer vision. IEEE, 365–372.
  41. Towards unsupervised image captioning with shared multimodal embeddings. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 7414–7424.
  42. Influencer backdoor attack on semantic segmentation. ICLR (2024).
  43. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning. PMLR, 12888–12900.
  44. Learning to Optimize Permutation Flow Shop Scheduling via Graph-based Imitation Learning. Proceedings of the AAAI Conference on Artificial Intelligence (2024).
  45. Semi-supervised robust training with generalized perturbed neighborhood. Pattern Recognition 124 (2022), 108472.
  46. Poisoned forgery face: Towards backdoor attacks on face forgery detection. arXiv preprint arXiv:2402.11473 (2024).
  47. VL-Trojan: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language Models. arXiv preprint arXiv:2402.13851 (2024).
  48. A large-scale multiple-objective method for black-box attack against object detection. In European Conference on Computer Vision.
  49. Imitated detectors: Stealing knowledge of black-box object detectors. In Proceedings of the 30th ACM International Conference on Multimedia.
  50. Unlearning Backdoor Threats: Enhancing Backdoor Defense in Multimodal Contrastive Learning via Local Token Unlearning. arXiv preprint arXiv:2403.16257 (2024).
  51. Object Detectors in the Open Environment: Challenges, Solutions, and Outlook. arXiv preprint arXiv:2403.16271 (2024).
  52. Generate more imperceptible adversarial examples for object detection. In ICML 2021 Workshop on Adversarial Machine Learning.
  53. Efficient adversarial attacks for visual object tracking. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVI 16.
  54. Parallel rectangle flip attack: A query-based black-box attack against object detection. arXiv preprint arXiv:2201.08970 (2022).
  55. Badclip: Dual-embedding guided backdoor attack on multimodal contrastive learning. arXiv preprint arXiv:2311.12075 (2023).
  56. Pre-trained trojan attacks for visual recognition. arXiv preprint arXiv:2312.15172 (2023).
  57. Does few-shot learning suffer from backdoor attacks?. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 19893–19901.
  58. Watermark vaccine: Adversarial attacks to prevent watermark removal. In European Conference on Computer Vision. Springer, 1–17.
  59. Poisoning attack against estimating from pairwise comparisons. IEEE Transactions on Pattern Analysis and Machine Intelligence 44, 10 (2021), 6393–6408.
  60. A tale of hodgerank and spectral method: Target attack against rank aggregation is the fixed point of adversarial game. IEEE Transactions on Pattern Analysis and Machine Intelligence 45, 4 (2022), 4090–4108.
  61. Sequential manipulation against rank aggregation: theory and algorithm. IEEE transactions on pattern analysis and machine intelligence (2024).
  62. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 (2017).
  63. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:2111.09734 (2021).
  64. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018).
  65. Styleclip: Text-driven manipulation of stylegan imagery. In Proceedings of the IEEE/CVF international conference on computer vision. 2085–2094.
  66. Learning transferable visual models from natural language supervision. In International conference on machine learning. PMLR, 8748–8763.
  67. Cuda: Convolution-based unlearnable datasets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3862–3871.
  68. Autoregressive perturbations for data poisoning. Advances in Neural Information Processing Systems 35 (2022), 27374–27386.
  69. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems 35 (2022), 25278–25294.
  70. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision. 618–626.
  71. Poison frogs! targeted clean-label poisoning attacks on neural networks. Advances in neural information processing systems 31 (2018).
  72. TensorClog: An imperceptible poisoning attack on deep neural network applications. IEEE Access 7 (2019), 41498–41506.
  73. Axiomatic attribution for deep networks. In International conference on machine learning. PMLR, 3319–3328.
  74. Yfcc100m: The new data in multimedia research. Commun. ACM 59, 2 (2016), 64–73.
  75. Attention is all you need. Advances in neural information processing systems 30 (2017).
  76. Universal adversarial triggers for attacking and analyzing NLP. arXiv preprint arXiv:1908.07125 (2019).
  77. Transferable adversarial attacks for image and video object detection. arXiv preprint arXiv:1811.12641 (2018).
  78. Minimalism is King! High-Frequency Energy-based Screening for Data-Efficient Backdoor Attacks. IEEE Transactions on Information Forensics and Security (2024).
  79. Robust Contrastive Language-Image Pretraining against Data Poisoning and Backdoor Attacks. Advances in Neural Information Processing Systems 36 (2024).
  80. Data poisoning attacks against multimodal encoders. In International Conference on Machine Learning. PMLR, 39299–39313.
  81. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics 2 (2014), 67–78.
  82. Availability attacks create shortcuts. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2367–2376.
  83. Chia-Hung Yuan and Shan-Hung Wu. 2021. Neural tangent generalization attacks. In International Conference on Machine Learning. PMLR, 12230–12240.
  84. Context-aware attention network for image-text retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 3536–3545.
  85. Towards Robust Physical-world Backdoor Attacks on Lane Detection. arXiv preprint arXiv:2405.05553 (2024).
  86. Breaking the False Sense of Security in Backdoor Defense through Re-Activation Attack. arXiv preprint arXiv:2405.16134 (2024).
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now