Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
194 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

SecFormer: Fast and Accurate Privacy-Preserving Inference for Transformer Models via SMPC (2401.00793v5)

Published 1 Jan 2024 in cs.LG, cs.CL, and cs.CR

Abstract: With the growing use of Transformer models hosted on cloud platforms to offer inference services, privacy concerns are escalating, especially concerning sensitive data like investment plans and bank account details. Secure Multi-Party Computing (SMPC) emerges as a promising solution to protect the privacy of inference data and model parameters. However, the application of SMPC in Privacy-Preserving Inference (PPI) for Transformer models often leads to considerable slowdowns or declines in performance. This is largely due to the multitude of nonlinear operations in the Transformer architecture, which are not well-suited to SMPC and difficult to circumvent or optimize effectively. To address this concern, we introduce a comprehensive PPI framework called SecFormer to achieve fast and accurate PPI for Transformer models. We successfully eliminate the high-cost exponential and maximum operations in PPI without sacrificing model performance and develop a suite of efficient SMPC protocols by employing suitable numerical computation methods to boost other complex nonlinear functions in PPI, including GeLU, LayerNorm, and a redesigned Softmax. Our extensive experiments reveal that SecFormer outperforms MPCFormer in performance, showing improvements of $3.4\%$ and $24.7\%$ for BERT${\text{BASE}}$ and BERT${\text{LARGE}}$, respectively. In terms of efficiency, SecFormer is 3.57 and 3.58 times faster than PUMA for BERT${\text{BASE}}$ and BERT${\text{LARGE}}$, demonstrating its effectiveness and speed.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (36)
  1. Chatgpt evaluation on sentence level relations: A focus on temporal, causal, and discourse relations. arXiv preprint arXiv:2304.14827.
  2. The-x: Privacy-preserving transformer inference with homomorphic encryption. In Findings of the Association for Computational Linguistics, pages 3510–3520.
  3. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  4. Puma: Secure inference of llama-7b in five minutes. arXiv preprint arXiv:2307.12533.
  5. Cryptonets: Applying neural networks to encrypted data with high throughput and accuracy. In Proceedings of the 33nd International Conference on Machine Learning, JMLR Workshop and Conference Proceedings, pages 201–210. JMLR.org.
  6. How to play any mental game or A completeness theorem for protocols with honest majority. In Proceedings of the 19th Annual ACM Symposium on Theory of Computing, 1987, pages 218–229. ACM.
  7. Robert E Goldschmidt. 1964. Applications of division by convergence. In M.Sc dissertation, Massachusetts Institute of Technology.
  8. Iron: Private inference on transformers. Advances in Neural Information Processing Systems, 35:15718–15731.
  9. Cheetah: Lean and fast secure two-party deep neural network inference. In Proceedings of 31st USENIX Security Symposium, pages 809–826. USENIX Association.
  10. Crypten: Secure multi-party computation meets machine learning. Advances in Neural Information Processing Systems, 34:4961–4973.
  11. Does BERT pretrained on clinical notes reveal sensitive data? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics, pages 946–959. Association for Computational Linguistics.
  12. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059. Association for Computational Linguistics.
  13. Mpcformer: fast, performant and private transformer inference with mpc. arXiv preprint arXiv:2211.01452.
  14. Oblivious neural network predictions via minionn transformations. In Proceedings of the 2017 ACM SIGSAC conference on computer and communications security, pages 619–631.
  15. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
  16. Peter Markstein. 2004a. Software division and square root using goldschmidt’s algorithms. In Proceedings of the 6th Conference on Real Numbers and Computers, pages 146–157. Citeseer.
  17. Peter W. Markstein. 2004b. Software division and square root using goldschmidt’s algorithms. In 6th Conference on Real Numbers and Computers, pages 146–157.
  18. Delphi: A cryptographic inference service for neural networks. In Proceedings of 29th USENIX Security Symposium, pages 2505–2522. USENIX Association.
  19. Payman Mohassel and Peter Rindal. 2018. Aby33{}^{\mbox{3}}start_FLOATSUPERSCRIPT 3 end_FLOATSUPERSCRIPT: A mixed protocol framework for machine learning. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, pages 35–52. ACM.
  20. Payman Mohassel and Yupeng Zhang. 2017. Secureml: A system for scalable privacy-preserving machine learning. In Proceedings of 2017 IEEE Symposium on Security and Privacy, pages 19–38. IEEE.
  21. OpenAI. 2023. Gpt-4 technical report. ArXiv, abs/2303.08774.
  22. Exploiting novel gpt-4 apis. arXiv preprint arXiv:2312.14302.
  23. Fábio Perez and Ian Ribeiro. 2022. Ignore previous prompt: Attack techniques for language models. CoRR, abs/2211.09527.
  24. Improving language understanding by generative pre-training.
  25. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
  26. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67.
  27. Sirnn: A math library for secure rnn inference. In Proceedings of 2021 IEEE Symposium on Security and Privacy, pages 1003–1020. IEEE.
  28. Adi Shamir. 1979. How to share a secret. Communications of the ACM, 22(11):612–613.
  29. Cryptgpu: Fast privacy-preserving machine learning on the gpu. In Proceedings of 2021 IEEE Symposium on Security and Privacy, pages 1021–1038. IEEE.
  30. Attention is all you need. In Advances in Neural Information Processing Systems. Curran Associates, Inc.
  31. SecureNN: 3-Party secure computation for neural network training. Proceedings on Privacy Enhancing Technologies, pages 26–49.
  32. Falcon: Honest-majority maliciously secure framework for private deep learning. Proceedings on Privacy Enhancing Technologies, pages 188–208.
  33. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations. OpenReview.net.
  34. Characterization of mpc-based private inference for transformer-based models. In Proceedings of 2022 IEEE International Symposium on Performance Analysis of Systems and Software, pages 187–197. IEEE.
  35. Andrew Chi-Chih Yao. 1986. How to generate and exchange secrets. In Annual Symposium on Foundations of Computer Science, pages 162–167.
  36. Mpcvit: Searching for mpc-friendly vision transformer with heterogeneous attention. arXiv preprint arXiv:2211.13955.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets