Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

VillanDiffusion: A Unified Backdoor Attack Framework for Diffusion Models (2306.06874v5)

Published 12 Jun 2023 in cs.CR, cs.CV, and cs.LG

Abstract: Diffusion Models (DMs) are state-of-the-art generative models that learn a reversible corruption process from iterative noise addition and denoising. They are the backbone of many generative AI applications, such as text-to-image conditional generation. However, recent studies have shown that basic unconditional DMs (e.g., DDPM and DDIM) are vulnerable to backdoor injection, a type of output manipulation attack triggered by a maliciously embedded pattern at model input. This paper presents a unified backdoor attack framework (VillanDiffusion) to expand the current scope of backdoor analysis for DMs. Our framework covers mainstream unconditional and conditional DMs (denoising-based and score-based) and various training-free samplers for holistic evaluations. Experiments show that our unified framework facilitates the backdoor analysis of different DM configurations and provides new insights into caption-based backdoor attacks on DMs. Our code is available on GitHub: \url{https://github.com/IBM/villandiffusion}

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (61)
  1. Cold diffusion: Inverting arbitrary image transforms without noise. In ArXiv, 2022.
  2. Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models. In ICLR, 2022.
  3. Label-efficient semantic segmentation with diffusion models. In ICLR, 2022.
  4. Offline reinforcement learning via high-fidelity generative behavior modeling. In ArXiv, 2022.
  5. Diffusiondet: Diffusion model for object detection. In ArXiv, 2022.
  6. Trojdiff: Trojan attacks on diffusion models with diverse targets. In CVPR, 2023.
  7. Diffusion policy: Visuomotor policy learning via action diffusion. 2023.
  8. Fair generative modeling via weak supervision. In ICML, 2020.
  9. How to backdoor diffusion models? In CVPR, 2023.
  10. Soft diffusion: Score matching for general corruptions. In ArXiv, 2022.
  11. Diffusion models beat gans on image synthesis. In NIPS, 2021.
  12. Density estimation using real NVP. In ICLR, 2017.
  13. Advflow: Inconspicuous black-box adversarial attacks using normalizing flows. In NIPS, 2020.
  14. Bias correction of learned generative models using likelihood-free importance weighting. In NIPS, 2019.
  15. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In NIPS, 2017.
  16. Denoising diffusion probabilistic models. In NIPS, 2020.
  17. Cascaded diffusion models for high fidelity image generation. In JMLR, 2022.
  18. Classifier-free diffusion guidance. In NIPS Workshop on Deep Generative Models and Downstream Applications, 2021.
  19. Video diffusion models. In NeurIPS, 2022.
  20. Lora: Low-rank adaptation of large language models. 2021.
  21. Fastdiff: A fast conditional diffusion model for high-quality speech synthesis. In IJCAI, 2022.
  22. Planning with diffusion for flexible behavior synthesis. In ICML, 2022.
  23. Diff-tts: A denoising diffusion model for text-to-speech. In ISCA, 2021.
  24. Talk-to-edit: Fine-grained facial editing via dialog. In ICCV, 2021.
  25. Elucidating the design space of diffusion-based generative models. In NIPS, 2022.
  26. Guided-tts: A diffusion model for text-to-speech via classifier guidance. In ICML, 2022.
  27. Glow: Generative flow with invertible 1x1 convolutions. In NIPS, 2018.
  28. Variational diffusion models. 2021.
  29. Diffwave: A versatile diffusion model for audio synthesis. In ICLR, 2021.
  30. Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
  31. Diffusion-lm improves controllable text generation. In ArXiv, 2022.
  32. Pseudo numerical methods for diffusion models on manifolds. In ICLR, 2022.
  33. Deep learning face attributes in the wild. In ICCV, 2015.
  34. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. In NIPS, 2022.
  35. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. In NIPS, 2022.
  36. VIDM: video implicit diffusion models. CoRR, abs/2212.00235, 2022.
  37. GLIDE: towards photorealistic image generation and editing with text-guided diffusion models. In ICML, 2022.
  38. Imitating human behaviour with diffusion models. In CoRR, 2023.
  39. Justin N. M. Pinkney. Pokemon blip captions. https://huggingface.co/datasets/lambdalabs/pokemon-blip-captions/, 2022.
  40. Grad-tts: A diffusion probabilistic model for text-to-speech. In ICML, 2021.
  41. Hierarchical text-conditional image generation with clip latents. In ArXiv, 2022.
  42. Variational inference with normalizing flows. In ICML, 2015.
  43. High-resolution image synthesis with latent diffusion models. In CVPR, 2021.
  44. Compvis/stable diffusion v1-4. https://huggingface.co/CompVis/stable-diffusion-v1-4, 2022.
  45. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
  46. Photorealistic text-to-image diffusion models with deep language understanding. In ArXiv, 2022.
  47. Improved techniques for training gans. In NIPS, 2016.
  48. Progressive distillation for fast sampling of diffusion models. In ICLR, 2022.
  49. LAION-5B: an open large-scale dataset for training next generation image-text models. In NIPS, 2022.
  50. LAION-400M: open dataset of clip-filtered 400 million image-text pairs. NIPS Workshop, 2021.
  51. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2015.
  52. Denoising diffusion implicit models. In ICLR, 2021.
  53. Maximum likelihood training of score-based diffusion models. In NIPS, 2021.
  54. Generative modeling by estimating gradients of the data distribution. In NIPS, 2019.
  55. Improved techniques for training score-based generative models. In NIPS, 2020.
  56. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
  57. Rickrolling the artist: Injecting invisible backdoors into text-guided image generation models. In ArXiv, 2022.
  58. Diffusion policies as an expressive policy class for offline reinforcement learning. In CoRR, 2022.
  59. Fast sampling of diffusion models with exponential integrator. In ICLR, 2023.
  60. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.
  61. Unipc: A unified predictor-corrector framework for fast sampling of diffusion models. 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Sheng-Yen Chou (4 papers)
  2. Pin-Yu Chen (311 papers)
  3. Tsung-Yi Ho (57 papers)
Citations (34)
View on Semantic Scholar