Papers
Topics
Authors
Recent
Search
2000 character limit reached

Privacy-Preserving Diffusion Model Using Homomorphic Encryption

Published 9 Mar 2024 in cs.CR and cs.AI | (2403.05794v2)

Abstract: In this paper, we introduce a privacy-preserving stable diffusion framework leveraging homomorphic encryption, called HE-Diffusion, which primarily focuses on protecting the denoising phase of the diffusion process. HE-Diffusion is a tailored encryption framework specifically designed to align with the unique architecture of stable diffusion, ensuring both privacy and functionality. To address the inherent computational challenges, we propose a novel min-distortion method that enables efficient partial image encryption, significantly reducing the overhead without compromising the model's output quality. Furthermore, we adopt a sparse tensor representation to expedite computational operations, enhancing the overall efficiency of the privacy-preserving diffusion process. We successfully implement HE-based privacy-preserving stable diffusion inference. The experimental results show that HE-Diffusion achieves 500 times speedup compared with the baseline method, and reduces time cost of the homomorphically encrypted inference to the minute level. Both the performance and accuracy of the HE-Diffusion are on par with the plaintext counterpart. Our approach marks a significant step towards integrating advanced cryptographic techniques with state-of-the-art generative models, paving the way for privacy-preserving and efficient image generation in critical applications.

Authors (2)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (60)
  1. Minimum-distortion embedding. Foundations and Trends® in Machine Learning, 14(3):211–378, 2021.
  2. Autofhe: Automated adaption of cnns for efficient evaluation over fhe. arXiv preprint arXiv:2310.08012, 2023.
  3. Tenseal: A library for encrypted tensor operations using homomorphic encryption, 2021.
  4. Mp2ml: A mixed-protocol machine learning framework for private inference. In Proceedings of the 15th international conference on availability, reliability and security, pages 1–10, 2020.
  5. ngraph-he2: A high-throughput framework for neural network inference on encrypted data. In Proceedings of the 7th ACM Workshop on Encrypted Computing & Applied Homomorphic Cryptography, pages 45–56, 2019.
  6. ngraph-he: a graph compiler for deep learning on homomorphically encrypted data. In Proceedings of the 16th ACM international conference on computing frontiers, pages 3–13, 2019.
  7. Zvika Brakerski. Fully homomorphic encryption without modulus switching from classical gapsvp. In Annual Cryptology Conference, pages 868–886. Springer, 2012.
  8. Fully homomorphic encryption without bootstrapping. ACM Transactions on Computation Theory (TOCT), 6(3):1–36, 2014.
  9. High-speed fully homomorphic encryption over the integers. In Financial Cryptography and Data Security: FC 2014 Workshops, BITCOIN and WAHC 2014, Christ Church, Barbados, March 7, 2014, Revised Selected Papers 18, pages 169–180. Springer, 2014.
  10. Extracting training data from diffusion models. In 32nd USENIX Security Symposium (USENIX Security 23), pages 5253–5270, 2023.
  11. When homomorphic encryption marries secret sharing: Secure large-scale sparse logistic regression and applications in risk control. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 2652–2662, 2021.
  12. Logistic regression over encrypted data from fully homomorphic encryption. BMC medical genomics, 11:3–12, 2018.
  13. The-x: Privacy-preserving transformer inference with homomorphic encryption. arXiv preprint arXiv:2206.00216, 2022.
  14. Homomorphic encryption for arithmetic of approximate numbers. In ASIACRYPT, pages 409–437, 2017.
  15. Does fully homomorphic encryption need compute acceleration? arXiv preprint arXiv:2112.06396, 2021.
  16. Differentially private diffusion models. arXiv preprint arXiv:2210.09929, 2022.
  17. Gibbs construction in steganography. IEEE Transactions on Information Forensics and Security, 5(4):705–720, 2010.
  18. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the ACM SIGSAC Conference on Computer and Communications Security, pages 1322–1333, 2015.
  19. Practical methods for minimizing embedding impact in steganography. In Security, Steganography, and Watermarking of Multimedia Contents IX, volume 6505, pages 13–27. SPIE, 2007.
  20. Craig Gentry. A Fully Homomorphic Encryption Scheme. PhD thesis, Stanford University, 2009.
  21. Cryptonets: Applying neural networks to encrypted data with high throughput and accuracy. In ICML, 2016.
  22. Generative adversarial networks. Communications of the ACM, 63(11):139–144, 2020.
  23. Designing steganographic distortion using directional filters. In 2012 IEEE International workshop on information forensics and security (WIFS), pages 234–239. IEEE, 2012.
  24. Image quality metrics: Psnr vs. ssim. In 2010 20th international conference on pattern recognition, pages 2366–2369. IEEE, 2010.
  25. Privacy data diffusion modeling and preserving in online social network. IEEE Transactions on Knowledge and Data Engineering, 2022.
  26. Deep networks with stochastic depth. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pages 646–661. Springer, 2016.
  27. Privacy-preserving deep sequential model with matrix homomorphic encryption. In Proceedings of the 2022 ACM on Asia Conference on Computer and Communications Security, pages 377–391, 2022.
  28. Deepreduce: Relu reduction for fast private inference. In International Conference on Machine Learning, pages 4839–4849. PMLR, 2021.
  29. Model pruning enables efficient federated learning on edge devices. IEEE Transactions on Neural Networks and Learning Systems, 2022.
  30. Gpt2: Empirical slant delay model for radio space geodetic techniques. Geophysical research letters, 40(6):1069–1073, 2013.
  31. A survey on image steganography and steganalysis. J. Inf. Hiding Multim. Signal Process., 2(2):142–172, 2011.
  32. A new cost function for spatial image steganography. In 2014 IEEE International conference on image processing (ICIP), pages 4206–4210. IEEE, 2014.
  33. An efficient distortion cost function design for image steganography in spatial domain using quaternion representation. Signal Processing, 219:109370, 2024.
  34. Soft: Softmax-free transformer with linear complexity. Advances in Neural Information Processing Systems, 34:21297–21309, 2021.
  35. Membership inference attacks against diffusion models. arXiv preprint arXiv:2302.03262, 2023.
  36. Can homomorphic encryption be practical? In Proceedings of the 3rd ACM workshop on Cloud computing security workshop, pages 113–124, 2011.
  37. Gaugan: semantic image synthesis with spatially adaptive normalization. In ACM SIGGRAPH 2019 Real-Time Live!, pages 1–1. 2019.
  38. Autorep: Automatic relu replacement for fast private network inference. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5178–5188, 2023.
  39. Using high-dimensional image models to perform highly undetectable steganography. In Information Hiding: 12th International Conference, IH 2010, Calgary, AB, Canada, June 28-30, 2010, Revised Selected Papers 12, pages 161–177. Springer, 2010.
  40. Cheetah: Optimizing and accelerating homomorphic encryption for private inference. In 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pages 26–39. IEEE, 2021.
  41. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  42. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234–241. Springer, 2015.
  43. Partially encrypted machine learning using functional encryption. arXiv preprint arXiv:1905.10214, 2019.
  44. F1: A fast and programmable accelerator for fully homomorphic encryption. In MICRO-54: 54th Annual IEEE/ACM International Symposium on Microarchitecture, pages 238–252, 2021.
  45. The pii problem: Privacy and a new concept of personally identifiable information. NYUL rev., 86:1814, 2011.
  46. Microsoft SEAL (release 4.1). https://github.com/Microsoft/SEAL, January 2023. Microsoft Research, Redmond, WA.
  47. Carm: Cuda-accelerated rns multiplication in word-wise homomorphic encryption schemes for internet of things. IEEE Transactions on Computers, 2022.
  48. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020.
  49. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):1929–1958, 2014.
  50. Deep learning and the information bottleneck principle. IEEE Information Theory Workshop (ITW), pages 1–5, 2015.
  51. Marc Van Droogenbroeck. Partial encryption of images for real-time applications. In Fourth IEEE Signal Processing Symposium, 2004.
  52. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  53. Efficient exact gradient update for training deep networks with very large sparse targets. Advances in Neural Information Processing Systems, 28, 2015.
  54. Diffusers: State-of-the-art diffusion models. https://github.com/huggingface/diffusers, 2022.
  55. Accelerating fully homomorphic encryption using gpu. In 2012 IEEE conference on high performance extreme computing, pages 1–5. IEEE, 2012.
  56. Exploring the feasibility of fully homomorphic encryption. IEEE Transactions on Computers, 64(3):698–706, 2013.
  57. A new cost function for spatial image steganography based on 2d-ssa and wmf. IEEE Access, 9:30604–30614, 2021.
  58. Fpga accelerator for homomorphic encrypted sparse convolutional neural network inference. In 2022 IEEE 30th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM), pages 1–9. IEEE, 2022.
  59. Privacy-preserving visual learning using doubly permuted homomorphic encryption. In Proceedings of the IEEE international conference on computer vision, pages 2040–2050, 2017.
  60. Zama. Concrete ML: a privacy-preserving machine learning library using fully homomorphic encryption for data scientists, 2022. https://github.com/zama-ai/concrete-ml.
Citations (2)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 3 tweets with 0 likes about this paper.

Sign Up for Free