Papers
Topics
Authors
Recent
2000 character limit reached

PackVFL: Efficient HE Packing for Vertical Federated Learning (2405.00482v1)

Published 1 May 2024 in cs.CR and cs.LG

Abstract: As an essential tool of secure distributed machine learning, vertical federated learning (VFL) based on homomorphic encryption (HE) suffers from severe efficiency problems due to data inflation and time-consuming operations. To this core, we propose PackVFL, an efficient VFL framework based on packed HE (PackedHE), to accelerate the existing HE-based VFL algorithms. PackVFL packs multiple cleartexts into one ciphertext and supports single-instruction-multiple-data (SIMD)-style parallelism. We focus on designing a high-performant matrix multiplication (MatMult) method since it takes up most of the ciphertext computation time in HE-based VFL. Besides, devising the MatMult method is also challenging for PackedHE because a slight difference in the packing way could predominantly affect its computation and communication costs. Without domain-specific design, directly applying SOTA MatMult methods is hard to achieve optimal. Therefore, we make a three-fold design: 1) we systematically explore the current design space of MatMult and quantify the complexity of existing approaches to provide guidance; 2) we propose a hybrid MatMult method according to the unique characteristics of VFL; 3) we adaptively apply our hybrid method in representative VFL algorithms, leveraging distinctive algorithmic properties to further improve efficiency. As the batch size, feature dimension and model size of VFL scale up to large sizes, PackVFL consistently delivers enhanced performance. Empirically, PackVFL propels existing VFL algorithms to new heights, achieving up to a 51.52X end-to-end speedup. This represents a substantial 34.51X greater speedup compared to the direct application of SOTA MatMult methods.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (78)
  1. 2022. Lattigo v4. Online: https://github.com/tuneinsight/lattigo. EPFL-LDS, Tune Insight SA.
  2. State-of-the-art in artificial neural network applications: A survey. Heliyon 4, 11 (2018), e00938.
  3. A survey on homomorphic encryption schemes: Theory and implementation. ACM Computing Surveys (Csur) 51, 4 (2018), 1–35.
  4. Homomorphic encryption standard. In Protecting Privacy through Homomorphic Encryption. Springer, 31–62.
  5. Searching for exotic particles in high-energy physics with deep learning. Nature communications 5, 1 (2014), 1–9.
  6. Amos Beimel. 2011. Secret-sharing schemes: A survey. In International conference on coding and cryptology. Springer, 11–46.
  7. TenSEAL: A library for encrypted tensor operations using homomorphic encryption. arXiv preprint arXiv:2104.03152 (2021).
  8. Comparing local and central differential privacy using membership inference attacks. In IFIP Annual Conference on Data and Applications Security and Privacy. Springer, 22–42.
  9. Packed ciphertexts in LWE-based homomorphic encryption. In International Workshop on Public Key Cryptography. Springer, 1–13.
  10. (Leveled) fully homomorphic encryption without bootstrapping. ACM Transactions on Computation Theory (TOCT) (2014).
  11. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901.
  12. Accelerating Vertical Federated Learning. arXiv preprint arXiv:2207.11456 (2022).
  13. Secure Forward Aggregation for Vertical Federated Neural Networks. arXiv preprint arXiv:2207.00165 (2022).
  14. Leaf: A benchmark for federated settings. arXiv preprint arXiv:1812.01097 (2018).
  15. The bit security of Paillier’s encryption scheme and its applications. In International Conference on the Theory and Applications of Cryptographic Techniques. Springer, 229–243.
  16. When homomorphic encryption marries secret sharing: Secure large-scale sparse logistic regression and applications in risk control. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining.
  17. Efficient homomorphic conversion between (ring) LWE ciphertexts. In International Conference on Applied Cryptography and Network Security. Springer, 460–479.
  18. Secureboost: A lossless federated learning framework. IEEE Intelligent Systems (2021).
  19. Homomorphic encryption for arithmetic of approximate numbers. In International conference on the theory and application of cryptology and information security.
  20. NUS-WIDE: A Real-World Web Image Database from National University of Singapore. In Proc. of ACM Conf. on Image and Video Retrieval (CIVR’09). Santorini, Greece.
  21. EVA: An encrypted vector arithmetic language and compiler for efficient homomorphic computation. In Proceedings of the 41st ACM SIGPLAN Conference on Programming Language Design and Implementation. 546–561.
  22. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science 9, 3–4 (2014), 211–407.
  23. Fatima Zahra Errounda and Yan Liu. 2022. A Mobility Forecasting Framework with Vertical Federated Learning. In 2022 IEEE 46th Annual Computers, Software, and Applications Conference (COMPSAC). IEEE, 301–310.
  24. Junfeng Fan and Frederik Vercauteren. 2012. Somewhat practical fully homomorphic encryption. Cryptology ePrint Archive (2012).
  25. Artificial intelligence across company borders. Commun. ACM 65, 1 (2021), 34–36.
  26. Blindfl: Vertical federated machine learning without peeking into your data. In Proceedings of the 2022 International Conference on Management of Data. 1316–1330.
  27. Shai Halevi and Victor Shoup. 2014. Algorithms in helib. In Annual Cryptology Conference. Springer, 554–571.
  28. Iron: Private inference on transformers. Advances in Neural Information Processing Systems 35 (2022), 15718–15731.
  29. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. Advances in neural information processing systems 30 (2017).
  30. EFMVFL: an efficient and flexible multi-party vertical federated learning without a third party. arXiv preprint arXiv:2201.06244 (2022).
  31. Cheetah: Lean and fast secure {{\{{two-party}}\}} deep neural network inference. In 31st USENIX Security Symposium (USENIX Security 22). 809–826.
  32. Public mobility data enables COVID-19 forecasting and management at local and global scales. Scientific reports 11, 1 (2021), 13531.
  33. Flashe: Additively symmetric homomorphic encryption for cross-silo federated learning. arXiv preprint arXiv:2109.00675 (2021).
  34. Towards End-to-End Secure and Efficient Federated Learning for XGBoost. (2022).
  35. {{\{{GAZELLE}}\}}: A low latency framework for secure neural network inference. In 27th USENIX Security Symposium (USENIX Security 18).
  36. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836 (2016).
  37. Learning multiple layers of features from tiny images. (2009).
  38. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11 (1998), 2278–2324.
  39. Certified robustness to adversarial examples with differential privacy. In 2019 IEEE symposium on security and privacy (SP). IEEE, 656–672.
  40. Efficient asynchronous vertical federated learning via gradient prediction and double-end sparse compression. In 2020 16th international conference on control, automation, robotics and vision (ICARCV). IEEE, 291–296.
  41. Gang Liang and Sudarshan S Chawathe. 2004. Privacy-preserving inter-database operations. In International Conference on Intelligence and Security Informatics. Springer, 66–82.
  42. Extrapolation for large-batch training in deep learning. In International Conference on Machine Learning. PMLR, 6094–6104.
  43. Deep Neural Networks for High Dimension, Low Sample Size Data.. In IJCAI. 2287–2293.
  44. Fate: An industrial grade platform for collaborative learning with data protection. The Journal of Machine Learning Research 22, 1 (2021), 10320–10325.
  45. A communication efficient collaborative learning framework for distributed features. arXiv preprint arXiv:1912.11187 (2019).
  46. Squirrel: A Scalable Secure Two-Party Computation Framework for Training Gradient Boosting Decision Tree. Cryptology ePrint Archive (2023).
  47. On ideal lattices and learning with errors over rings. Journal of the ACM (JACM) 60, 6 (2013), 1–35.
  48. Towards Practical Secure Neural Network Inference: The Journey So Far and the Road Ahead. Cryptology ePrint Archive (2022).
  49. H Brendan McMahan et al. 2021. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning (2021).
  50. Delphi: A cryptographic inference service for neural networks. In 29th USENIX Security Symposium (USENIX Security 20). 2505–2522.
  51. Payman Mohassel and Peter Rindal. 2018. ABY3: A mixed protocol framework for machine learning. In Proceedings of the 2018 ACM SIGSAC conference on computer and communications security. 35–52.
  52. Payman Mohassel and Yupeng Zhang. 2017. Secureml: A system for scalable privacy-preserving machine learning. In 2017 IEEE symposium on security and privacy (SP). IEEE, 19–38.
  53. Multiparty Homomorphic Encryption: From Theory to Practice. IACR Cryptol. ePrint Arch. 2020 (2020), 304.
  54. Adding gradient noise improves learning for very deep networks. arXiv preprint arXiv:1511.06807 (2015).
  55. Post-quantum lattice-based cryptography implementations: A survey. ACM Computing Surveys (CSUR) 51, 6 (2019), 1–41.
  56. A homomorphic-encryption-based vertical federated learning scheme for rick management. Computer Science and Information Systems (2020).
  57. Pascal Paillier. 1999. Public-key cryptosystems based on composite degree residuosity classes. In International conference on the theory and applications of cryptographic techniques. Springer, 223–238.
  58. Federated Learning in a Medical Context: A Systematic Literature Review. ACM Transactions on Internet Technology (TOIT) (2021).
  59. F1: A fast and programmable accelerator for fully homomorphic encryption. In MICRO-54: 54th Annual IEEE/ACM International Symposium on Microarchitecture. 238–252.
  60. POSEIDON: Privacy-Preserving Federated Neural Network Learning. In 28th Annual Network and Distributed System Security Symposium, NDSS 2021, virtually, February 21-25, 2021.
  61. Peter W Shor. 1999. Polynomial-time algorithms for prime factorization and discrete logarithms on a quantum computer. SIAM review 41, 2 (1999), 303–332.
  62. FPGA-based hardware accelerator for leveled ring-lwe fully homomorphic encryption. IEEE Access 8 (2020), 168008–168025.
  63. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023).
  64. Split learning for health: Distributed deep learning without sharing raw patient data. arXiv preprint arXiv:1812.00564 (2018).
  65. XFL: A High Performace, Lightweighted Federated Learning Framework. arXiv preprint arXiv:2302.05076 (2023).
  66. Achieving Differential Privacy in Vertically Partitioned Multiparty Learning. arXiv preprint arXiv:1911.04587 (2019).
  67. Hercules: Boosting the Performance of Privacy-preserving Federated Learning. IEEE Transactions on Dependable and Secure Computing (2022).
  68. A quasi-newton method based vertical federated learning framework for logistic regression. arXiv preprint arXiv:1912.00513 (2019).
  69. A Survey on Vertical Federated Learning: From a Layered Perspective. arXiv preprint arXiv:2304.01829 (2023).
  70. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST) (2019).
  71. Parallel distributed logistic regression for vertical federated learning without third-party coordinator. arXiv preprint arXiv:1911.09824 (2019).
  72. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414 (2022).
  73. The emerging” big dimensionality”. IEEE Computational Intelligence Magazine 9, 3 (2014), 14–26.
  74. {{\{{BatchCrypt}}\}}: Efficient homomorphic encryption for {{\{{Cross-Silo}}\}} federated learning. In 2020 USENIX annual technical conference (USENIX ATC 20). 493–506.
  75. Sok: Fully homomorphic encryption accelerators. arXiv preprint arXiv:2212.01713 (2022).
  76. GALA: Greedy ComputAtion for Linear Algebra in Privacy-Preserved Neural Networks. In 28th Annual Network and Distributed System Security Symposium, NDSS 2021, virtually, February 21-25, 2021.
  77. Yifei Zhang and Hao Zhu. 2020. Additively homomorphical encryption based deep neural network for asymmetrically collaborative machine learning. arXiv preprint arXiv:2007.06849 (2020).
  78. Deep leakage from gradients. Advances in neural information processing systems (2019).

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now

Whiteboard

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up for Free