Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

LinGCN: Structural Linearized Graph Convolutional Network for Homomorphically Encrypted Inference (2309.14331v3)

Published 25 Sep 2023 in cs.LG, cs.AI, and cs.CR

Abstract: The growth of Graph Convolution Network (GCN) model sizes has revolutionized numerous applications, surpassing human performance in areas such as personal healthcare and financial systems. The deployment of GCNs in the cloud raises privacy concerns due to potential adversarial attacks on client data. To address security concerns, Privacy-Preserving Machine Learning (PPML) using Homomorphic Encryption (HE) secures sensitive client data. However, it introduces substantial computational overhead in practical applications. To tackle those challenges, we present LinGCN, a framework designed to reduce multiplication depth and optimize the performance of HE based GCN inference. LinGCN is structured around three key elements: (1) A differentiable structural linearization algorithm, complemented by a parameterized discrete indicator function, co-trained with model weights to meet the optimization goal. This strategy promotes fine-grained node-level non-linear location selection, resulting in a model with minimized multiplication depth. (2) A compact node-wise polynomial replacement policy with a second-order trainable activation function, steered towards superior convergence by a two-level distillation approach from an all-ReLU based teacher model. (3) an enhanced HE solution that enables finer-grained operator fusion for node-wise activation functions, further reducing multiplication level consumption in HE-based inference. Our experiments on the NTU-XVIEW skeleton joint dataset reveal that LinGCN excels in latency, accuracy, and scalability for homomorphically encrypted inference, outperforming solutions such as CryptoGCN. Remarkably, LinGCN achieves a 14.2x latency speedup relative to CryptoGCN, while preserving an inference accuracy of 75% and notably reducing multiplication depth.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (95)
  1. Point-gnn: Graph neural network for 3d object detection in a point cloud. In CVPR, pages 1711–1719, 2020.
  2. Graph neural network for traffic forecasting: A survey. Expert Systems with Applications, page 117921, 2022.
  3. Skeleton-based action recognition with spatial reasoning and temporal stack learning. In Proceedings of the European conference on computer vision (ECCV), pages 103–118, 2018.
  4. Spatial temporal graph convolutional networks for skeleton-based action recognition. In Thirty-second AAAI conference on artificial intelligence, 2018.
  5. Graph neural networks in recommender systems: a survey. ACM Computing Surveys (CSUR), 2020.
  6. Pietro Bongini et al. Molecular generative graph neural networks for drug discovery. Neurocomputing, 450:242–252, 2021.
  7. Anuroop Sriram et al. Towards training billion parameter graph neural networks for atomic simulations. ArXiv, abs/2203.09697, 2022.
  8. Daniel Manu et al. Co-exploration of graph neural network and network-on-chip design using automl. In Proceedings of the 2021 on Great Lakes Symposium on VLSI, pages 175–180, 2021.
  9. Rex Ying et al. Graph convolutional neural networks for web-scale recommender systems. KDD ’18, 2018.
  10. Rong Zhu et al. Aligraph: A comprehensive graph neural network platform. KDD ’19, 2019.
  11. Spg: Structure-private graph database via squeezepir. Proceedings of the VLDB Endowment, 16(7):1615–1628, 2023.
  12. Cryptogcn: Fast and scalable homomorphically encrypted graph convolutional network inference. Advances in Neural Information Processing Systems, 35:37676–37689, 2022.
  13. Privacy-preserving deep learning: Revisited and enhanced. In Applications and Techniques in Information Security: 8th International Conference, ATIS 2017, Auckland, New Zealand, July 6–7, 2017, Proceedings, pages 100–110. Springer, 2017.
  14. Deep leakage from gradients. Advances in neural information processing systems, 32, 2019.
  15. Homomorphic encryption for arithmetic of approximate numbers. In International Conference on the Theory and Application of Cryptology and Information Security, pages 409–437. Springer, 2017.
  16. (leveled) fully homomorphic encryption without bootstrapping. ACM Transactions on Computation Theory (TOCT), 6(3):1–36, 2014.
  17. Tapas: Tricks to accelerate (encrypted) prediction as a service. In International Conference on Machine Learning, pages 4490–4499. PMLR, 2018.
  18. Fast homomorphic evaluation of deep discretized neural networks. In Advances in Cryptology–CRYPTO 2018: 38th Annual International Cryptology Conference, Santa Barbara, CA, USA, August 19–23, 2018, Proceedings, Part III 38, pages 483–512. Springer, 2018.
  19. Cryptonets: Applying neural networks to encrypted data with high throughput and accuracy. In International conference on machine learning, pages 201–210. PMLR, 2016.
  20. Low latency privacy preserving inference. In International Conference on Machine Learning, pages 812–821. PMLR, 2019.
  21. Ntu rgb+d: A large scale dataset for 3d human activity analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1010–1019, 2016.
  22. Deep high-resolution representation learning for human pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5693–5703, 2019.
  23. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.
  24. Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7291–7299, 2017.
  25. Kim Laine. Simple encrypted arithmetic library (seal) manual, 2017.
  26. Security of homomorphic encryption. HomomorphicEncryption. org, Redmond WA, Tech. Rep, 2017.
  27. Craig Gentry. A fully homomorphic encryption scheme. Stanford university, 2009.
  28. Tfhe: fast fully homomorphic encryption over the torus. Journal of Cryptology, 33(1):34–91, 2020.
  29. Delphi: A cryptographic inference service for neural networks. In 29th USENIX Security Symposium (USENIX Security 20), pages 2505–2522, 2020.
  30. Cryptonas: Private inference on a relu budget. Advances in Neural Information Processing Systems, 33:16961–16971, 2020.
  31. Crypten: Secure multi-party computation meets machine learning. Advances in Neural Information Processing Systems, 34:4961–4973, 2021.
  32. {{\{{XONN}}\}}:{{\{{XNOR-based}}\}} oblivious deep neural network inference. In 28th USENIX Security Symposium (USENIX Security 19), pages 1501–1518, 2019.
  33. Sphynx: Relu-efficient network design for private inference. arXiv preprint arXiv:2106.11755, 2021.
  34. Safenet: A secure, accurate and fast neural network inference. In International Conference on Learning Representations, 2020.
  35. Selective network linearization for efficient private inference. In International Conference on Machine Learning, pages 3947–3961. PMLR, 2022.
  36. Deepreduce: Relu reduction for fast private inference. In International Conference on Machine Learning, pages 4839–4849. PMLR, 2021.
  37. Chet: an optimizing compiler for fully-homomorphic neural-network inferencing. In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 142–156, 2019.
  38. Secure human action recognition by encrypted neural network inference. Nature communications, 13(1):1–13, 2022.
  39. Hear: Human action recognition via neural networks on homomorphically encrypted data. arXiv preprint arXiv:2104.09164, 2021.
  40. Autorep: Automatic relu replacement for fast private network inference. arXiv preprint arXiv:2308.10134, 2023.
  41. Pasnet: Polynomial architecture search framework for two-party computation-based secure neural network deployment. In 2023 60th ACM/IEEE Design Automation Conference (DAC), pages 1–6. IEEE, 2023.
  42. Rrnet: Towards relu-reduced neural network for two-party computation based private inference. arXiv preprint arXiv:2302.02292, 2023.
  43. Safenet: A secure, accurate and fast neural network inference. In International Conference on Learning Representations, 2021.
  44. 1xn pattern for pruning convolutional neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
  45. Neuron-level structured pruning using polarization regularizer. Advances in neural information processing systems, 33:9865–9877, 2020.
  46. Binarized neural networks. Advances in neural information processing systems, 29, 2016.
  47. Training sparse neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 138–145, 2017.
  48. Understanding straight-through estimator in training activation quantized neural nets. arXiv preprint arXiv:1903.05662, 2019.
  49. Autoprune: Automatic network pruning by regularizing auxiliary parameters. Advances in neural information processing systems, 32, 2019.
  50. On polynomial approximations for privacy-preserving and verifiable relu networks. arXiv preprint arXiv:2011.05530, 2020.
  51. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.
  52. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In International Conference on Learning Representations, 2017.
  53. A full rns variant of approximate homomorphic encryption. In International Conference on Selected Areas in Cryptography, pages 347–368. Springer, 2018.
  54. Microsoft SEAL (release 3.7). https://github.com/Microsoft/SEAL, September 2021. Microsoft Research, Redmond, WA.
  55. Low-complexity deep convolutional neural networks on fully homomorphic encryption using multiplexed parallel convolutions. In International Conference on Machine Learning, pages 12403–12422. PMLR, 2022.
  56. Deeper insights into graph convolutional networks for semi-supervised learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
  57. Label informed attributed network embedding. In Proceedings of the tenth ACM international conference on web search and data mining, pages 731–739, 2017.
  58. Accel-gcn: High-performance gpu accelerator design for graph convolution networks. arXiv preprint arXiv:2308.11825, 2023.
  59. Brain network transformer. arXiv preprint arXiv:2210.06681, 2022.
  60. Hmc-tran: A tensor-core inspired hierarchical model compression for transformer-based dnns on gpu. In Proceedings of the 2021 on Great Lakes Symposium on VLSI, pages 169–174, 2021.
  61. The larger the fairer? small neural networks can achieve fairness for edge devices. In Proceedings of the 59th ACM/IEEE Design Automation Conference, pages 163–168, 2022.
  62. Lightweight run-time working memory compression for deployment of deep neural networks on resource-constrained mcus. In Proceedings of the 26th Asia and South Pacific Design Automation Conference, pages 607–614, 2021.
  63. An automatic and efficient bert pruning for edge ai systems. In 2022 23rd International Symposium on Quality Electronic Design (ISQED), pages 1–6. IEEE, 2022.
  64. Intermittent inference with nonuniformly compressed multi-exit neural network for energy harvesting powered devices. In 2020 57th ACM/IEEE Design Automation Conference (DAC), pages 1–6. IEEE, 2020.
  65. Codg-reram: An algorithm-hardware co-design to accelerate semi-structured gnns on reram. In 2022 IEEE 40th International Conference on Computer Design (ICCD), pages 280–289. IEEE, 2022.
  66. A length adaptive algorithm-hardware co-design of transformer on fpga through sparse attention and dynamic pipelining. In Proceedings of the 59th ACM/IEEE Design Automation Conference, pages 1135–1140, 2022.
  67. Accelerating dataset distillation via model augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11950–11959, 2023.
  68. Accommodating transformer onto fpga: Coupling the balanced model compression and fpga-implementation optimization. In Proceedings of the 2021 on Great Lakes Symposium on VLSI, pages 163–168, 2021.
  69. What makes convolutional models great on long sequence modeling? arXiv preprint arXiv:2210.09298, 2022.
  70. Generic neural architecture search via regression. Advances in Neural Information Processing Systems, 34:20476–20490, 2021.
  71. Zero-shot scene graph relation prediction through commonsense knowledge integration. In Machine Learning and Knowledge Discovery in Databases. Research Track: European Conference, ECML PKDD 2021, Bilbao, Spain, September 13–17, 2021, Proceedings, Part II 21, pages 466–482. Springer, 2021.
  72. Fbnetgen: Task-aware gnn-based fmri analysis via functional brain network generation. In International Conference on Medical Imaging with Deep Learning, pages 618–637. PMLR, 2022.
  73. Binary complex neural network acceleration on fpga. In 2021 IEEE 32nd International Conference on Application-specific Systems, Architectures and Processors (ASAP), pages 85–92. IEEE, 2021.
  74. Accelerating framework of transformer by hardware design and model compression co-optimization. In 2021 IEEE/ACM International Conference On Computer Aided Design (ICCAD), pages 1–9. IEEE, 2021.
  75. Digital-assisted analog in-memory computing with rram devices. In 2023 International VLSI Symposium on Technology, Systems and Applications (VLSI-TSA/VLSI-DAT), pages 1–4. IEEE, 2023.
  76. Accelerating transformer-based deep learning models on fpgas using column balanced block pruning. In 2021 22nd International Symposium on Quality Electronic Design (ISQED), pages 142–148. IEEE, 2021.
  77. Algorithm/accelerator co-design and co-search for edge ai. IEEE Transactions on Circuits and Systems II: Express Briefs, 69(7):3064–3070, 2022.
  78. Efficient approximate solution path algorithm for order weight l_1-norm with accuracy guarantee. In 2019 IEEE International Conference on Data Mining (ICDM), pages 958–963. IEEE, 2019.
  79. Towards sparsification of graph neural networks. In 2022 IEEE 40th International Conference on Computer Design (ICCD), pages 272–279. IEEE, 2022.
  80. Fast oscar and owl regression via safe screening rules. In International Conference on Machine Learning, pages 653–663. PMLR, 2020.
  81. Synthetic data can also teach: Synthesizing effective data for unsupervised visual representation learning. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2023.
  82. Enabling on-device self-supervised contrastive learning with selective data contrast. In 2021 58th ACM/IEEE Design Automation Conference (DAC), pages 655–660. IEEE, 2021.
  83. Distributed contrastive learning for medical image segmentation. Medical Image Analysis, 81:102564, 2022.
  84. Neurogenesis dynamics-inspired spiking neural network training acceleration. arXiv preprint arXiv:2304.12214, 2023.
  85. Decentralized unsupervised learning of visual representations. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI, pages 2326–2333, 2022.
  86. Dynamic sparse training via balancing the exploration-exploitation trade-off. arXiv preprint arXiv:2211.16667, 2022.
  87. Doubly sparse asynchronous learning for stochastic composite optimization. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI, pages 1916–1922, 2022.
  88. An accelerated doubly stochastic gradient method with faster explicit model identification. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pages 57–66, 2022.
  89. Enabling on-device cnn training by self-supervised instance filtering and error map pruning. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 39(11):3445–3457, 2020.
  90. Balance is essence: Accelerating sparse training via adaptive gradient correction. arXiv preprint arXiv:2301.03573, 2023.
  91. Neighborhood-regularized self-training for learning with few labels. arXiv preprint arXiv:2301.03726, 2023.
  92. Federated contrastive learning for dermatological disease diagnosis via on-device learning. In 2021 IEEE/ACM International Conference On Computer Aided Design (ICCAD), pages 1–7. IEEE, 2021.
  93. Federated contrastive learning for volumetric medical image segmentation. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part III 24, pages 367–377. Springer, 2021.
  94. Variance of the gradient also matters: Privacy leakage from gradients. In 2022 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, 2022.
  95. Against membership inference attack: Pruning is all you need. arXiv preprint arXiv:2008.13578, 2020.
Citations (31)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now