Papers
Topics
Authors
Recent
Search
2000 character limit reached

SABLE: Secure And Byzantine robust LEarning

Published 11 Sep 2023 in cs.LG, cs.CR, and cs.DC | (2309.05395v4)

Abstract: Due to the widespread availability of data, ML algorithms are increasingly being implemented in distributed topologies, wherein various nodes collaborate to train ML models via the coordination of a central server. However, distributed learning approaches face significant vulnerabilities, primarily stemming from two potential threats. Firstly, the presence of Byzantine nodes poses a risk of corrupting the learning process by transmitting inaccurate information to the server. Secondly, a curious server may compromise the privacy of individual nodes, sometimes reconstructing the entirety of the nodes' data. Homomorphic encryption (HE) has emerged as a leading security measure to preserve privacy in distributed learning under non-Byzantine scenarios. However, the extensive computational demands of HE, particularly for high-dimensional ML models, have deterred attempts to design purely homomorphic operators for non-linear robust aggregators. This paper introduces SABLE, the first homomorphic and Byzantine robust distributed learning algorithm. SABLE leverages HTS, a novel and efficient homomorphic operator implementing the prominent coordinate-wise trimmed mean robust aggregator. Designing HTS enables us to implement HMED, a novel homomorphic median aggregator. Extensive experiments on standard ML tasks demonstrate that SABLE achieves practical execution times while maintaining an ML accuracy comparable to its non-private counterpart.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (80)
  1. Generating One-Hot Maps Under Encryption. In Cyber Security, Cryptology, and Machine Learning - 7th International Symposium, CSCML 2023, Be’er Sheva, Israel, June 29-30, 2023, Proceedings (Lecture Notes in Computer Science, Vol. 13914), Shlomi Dolev, Ehud Gudes, and Pascal Paillier (Eds.). Springer, 96–116. https://doi.org/10.1007/978-3-031-34671-2_8
  2. On the concrete hardness of Learning with Errors. Journal of Mathematical Cryptology 9, 3 (2015), 169–203. https://doi.org/10.1515/jmc-2015-0016
  3. Byzantine-Resilient Non-Convex Stochastic Gradient Descent. In International Conference on Learning Representations.
  4. Fixing by Mixing: A Recipe for Optimal Byzantine ML under Heterogeneity. In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics (Proceedings of Machine Learning Research, Vol. 206), Francisco Ruiz, Jennifer Dy, and Jan-Willem van de Meent (Eds.). PMLR, 1232–1300. https://proceedings.mlr.press/v206/allouah23a.html
  5. On the Privacy-Robustness-Utility Trilemma in Distributed Learning. In International Conference on Machine Learning.
  6. OpenFHE: Open-Source Fully Homomorphic Encryption Library. Cryptology ePrint Archive, Paper 2022/915. https://eprint.iacr.org/2022/915 https://eprint.iacr.org/2022/915.
  7. A Little Is Enough: Circumventing Defenses For Distributed Learning. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, 8-14 December 2019, Long Beach, CA, USA.
  8. Dimitri Bertsekas and John Tsitsiklis. 2015. Parallel and distributed computation: numerical methods. Athena Scientific.
  9. Machine Learning with Adversaries: Byzantine Tolerant Gradient Descent. In Advances in Neural Information Processing Systems 30, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.). Curran Associates, Inc., 119–129.
  10. Practical Secure Aggregation for Privacy-Preserving Machine Learning. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (Dallas, Texas, USA) (CCS ’17). Association for Computing Machinery, New York, NY, USA, 1175–1191. https://doi.org/10.1145/3133956.3133982
  11. Zvika Brakerski. 2012. Fully Homomorphic Encryption without Modulus Switching from Classical GapSVP. Lecture Notes in Computer Science, Vol. 7417. Springer Berlin Heidelberg, Berlin, Heidelberg, 868–886. https://doi.org/10.1007/978-3-642-32009-5_50
  12. Fully Homomorphic Encryption without Bootstrapping. Electron. Colloquium Comput. Complex. 18 (2011), 111. http://dblp.uni-trier.de/db/journals/eccc/eccc18.html#BrakerskiGV11
  13. FLTrust: Byzantine-robust Federated Learning via Trust Bootstrapping. In 28th Annual Network and Distributed System Security Symposium, NDSS 2021, virtually, February 21-25, 2021. The Internet Society. https://www.ndss-symposium.org/ndss-paper/fltrust-byzantine-robust-federated-learning-via-trust-bootstrapping/
  14. Depth Optimized Efficient Homomorphic Sorting. In Proceedings of the 4th International Conference on Progress in Cryptology – LATINCRYPT 2015 - Volume 9230. Springer-Verlag, Berlin, Heidelberg, 61–80. https://doi.org/10.1007/978-3-319-22174-8_4
  15. Olive Chakraborty and Martin Zuber. 2022. Efficient and Accurate Homomorphic Comparisons. In Proceedings of the 10th Workshop on Encrypted Computing & Applied Homomorphic Cryptography (Los Angeles, CA, USA) (WAHC’22). Association for Computing Machinery, New York, NY, USA, 35–46. https://doi.org/10.1145/3560827.3563375
  16. Distributed statistical machine learning in adversarial settings: Byzantine gradient descent. Proceedings of the ACM on Measurement and Analysis of Computing Systems 1, 2 (2017), 1–25.
  17. Homomorphic Encryption for Arithmetic of Approximate Numbers. In Advances in Cryptology – ASIACRYPT 2017, Tsuyoshi Takagi and Thomas Peyrin (Eds.). Springer International Publishing, Cham, 409–437.
  18. TFHE: Fast Fully Homomorphic Encryption Over the Torus. J. Cryptol. 33, 1 (jan 2020), 34–91. https://doi.org/10.1007/s00145-019-09319-x
  19. Henry Corrigan-Gibbs and Dan Boneh. 2017. Prio: Private, Robust, and Scalable Computation of Aggregate Statistics. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17). USENIX Association, Boston, MA, 259–282. https://www.usenix.org/conference/nsdi17/technical-sessions/presentation/corrigan-gibbs
  20. Li Deng. 2012. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine 29, 6 (2012), 141–142.
  21. The Hidden Vulnerability of Distributed Learning in Byzantium. In Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 80), Jennifer Dy and Andreas Krause (Eds.). PMLR, 3521–3530. https://proceedings.mlr.press/v80/mhamdi18a.html
  22. Distributed Momentum for Byzantine-resilient Stochastic Gradient Descent. In 9th International Conference on Learning Representations, ICLR 2021, Vienna, Austria, May 4–8, 2021. OpenReview.net.
  23. Junfeng Fan and Frederik Vercauteren. 2012. Somewhat Practical Fully Homomorphic Encryption. (2012). https://eprint.iacr.org/2012/144 Report Number: 144.
  24. Robust Collaborative Learning with Linear Gradient Overhead. In Proceedings of the 40th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 202), Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (Eds.). PMLR, 9761–9813. https://proceedings.mlr.press/v202/farhadkhani23a.html
  25. Byzantine Machine Learning Made Easy By Resilient Averaging of Momentums. In Proceedings of the 39th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 162), Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (Eds.). PMLR, 6246–6283.
  26. Inverting Gradients - How easy is it to break privacy in federated learning?. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 16937–16947. https://proceedings.neurips.cc/paper_files/paper/2020/file/c4ede56bbd98819ae6112b20ac6bf145-Paper.pdf
  27. CryptoNets: Applying Neural Networks to Encrypted Data with High Throughput and Accuracy. In Proceedings of The 33rd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 48), Maria Florina Balcan and Kilian Q. Weinberger (Eds.). PMLR, New York, New York, USA, 201–210. https://proceedings.mlr.press/v48/gilad-bachrach16.html
  28. Differential Privacy and Byzantine Resilience in SGD: Do They Add Up?. In Proceedings of the 2021 ACM Symposium on Principles of Distributed Computing (Virtual Event, Italy) (PODC’21). Association for Computing Machinery, New York, NY, USA, 391–401. https://doi.org/10.1145/3465084.3467919
  29. Nirupam Gupta and Nitin H Vaidya. 2020. Fault-tolerance in distributed optimization: The case of redundancy. In Proceedings of the 39th Symposium on Principles of Distributed Computing. 365–374.
  30. Shai Halevi and Victor Shoup. 2020. Design and implementation of HElib: a homomorphic encryption library. Cryptology ePrint Archive, Paper 2020/1481. https://eprint.iacr.org/2020/1481 https://eprint.iacr.org/2020/1481.
  31. Efficient, Private and Robust Federated Learning. In Annual Computer Security Applications Conference (Virtual Event, USA) (ACSAC ’21). Association for Computing Machinery, New York, NY, USA, 45–60. https://doi.org/10.1145/3485832.3488014
  32. Secure Byzantine-Robust Machine Learning. arXiv:2006.04747 [cs.LG]
  33. Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification. https://doi.org/10.48550/ARXIV.1909.06335
  34. Ilia Iliashenko and Vincent Zucca. 2021. Faster homomorphic comparison operations for BGV and BFV. Proceedings on Privacy Enhancing Technologies 2021 (07 2021), 246–264. https://doi.org/10.2478/popets-2021-0046
  35. GAZELLE: A Low Latency Framework for Secure Neural Network Inference. In Proceedings of the 27th USENIX Conference on Security Symposium (Baltimore, MD, USA) (SEC’18). USENIX Association, USA, 1651–1668.
  36. Learning from History for Byzantine Robust Optimization. International Conference On Machine Learning, Vol 139 139 (2021).
  37. Byzantine-Robust Learning on Heterogeneous Datasets via Bucketing. In International Conference on Learning Representations. https://openreview.net/forum?id=jXKKDEi5vJt
  38. MLGuard: Mitigating Poisoning Attacks in Privacy Preserving Distributed Collaborative Learning. 2020 29th International Conference on Computer Communications and Networks (ICCCN) (2020), 1–9. https://api.semanticscholar.org/CorpusID:218594342
  39. The CIFAR-10 dataset. online: http://www. cs. toronto. edu/kriz/cifar. html 55, 5 (2014).
  40. The Byzantine Generals Problem. ACM Trans. Program. Lang. Syst. 4, 3 (jul 1982), 382–401. https://doi.org/10.1145/357172.357176
  41. RSA: Byzantine-Robust Stochastic Aggregation Methods for Distributed Learning from Heterogeneous Datasets. Proceedings of the AAAI Conference on Artificial Intelligence 33, 01 (Jul. 2019), 1544–1551. https://doi.org/10.1609/aaai.v33i01.33011544
  42. Approximate Byzantine Fault-Tolerance in Distributed Optimization. In Proceedings of the 2021 ACM Symposium on Principles of Distributed Computing (Virtual Event, Italy) (PODC’21). Association for Computing Machinery, New York, NY, USA, 379–389. https://doi.org/10.1145/3465084.3467902
  43. Glyph: Fast and Accurately Training Deep Neural Networks on Encrypted Data. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 9193–9202. https://proceedings.neurips.cc/paper_files/paper/2020/file/685ac8cadc1be5ac98da9556bc1c8d9e-Paper.pdf
  44. Privacy-preserving federated learning based on multi-key homomorphic encryption. International Journal of Intelligent Systems 37, 9 (2022), 5880–5901. https://doi.org/10.1002/int.22818 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/int.22818
  45. Privacy-preserving Byzantine-robust federated learning. Computer Standards & Interfaces 80 (2022), 103561. https://doi.org/10.1016/j.csi.2021.103561
  46. A Secure Federated Learning framework using Homomorphic Encryption and Verifiable Computing. In 2021 Reconciling Data Analytics, Automation, Privacy, and Security: A Big Data Challenge (RDAAPS). 1–8. https://doi.org/10.1109/RDAAPS48126.2021.9452005
  47. Privacy-Preserving Byzantine-Robust Federated Learning via Blockchain Systems. IEEE Transactions on Information Forensics and Security 17 (2022), 2848–2861. https://doi.org/10.1109/TIFS.2022.3196274
  48. Poster: FLATEE: Federated Learning Across Trusted Execution Environments. In 2021 IEEE European Symposium on Security and Privacy (EuroS&P). 707–709. https://doi.org/10.1109/EuroSP51992.2021.00054
  49. Performance Analysis of Sorting of FHE Data: Integer-Wise Comparison vs Bit-Wise Comparison. In 31st IEEE International Conference on Advanced Information Networking and Applications, AINA 2017, Taipei, Taiwan, March 27-29, 2017, Leonard Barolli, Makoto Takizawa, Tomoya Enokido, Hui-Huang Hsu, and Chi-Yi Lin (Eds.). IEEE Computer Society, 902–908. https://doi.org/10.1109/AINA.2017.85
  50. Lucien K. L. Ng and Sherman S. M Chow. 2023. SoK: Cryptographic Neural-Network Computation. In 44th IEEE Symposium on Security and Privacy, SP 2023, San Francisco, CA, USA, 22-25 May 2023. IEEE Computer Society, 497–514. https://doi.org/10.1109/SP46215.2023.00198
  51. FLGUARD: Secure and Private Federated Learning. (01 2021).
  52. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32. Curran Associates, Inc., 8024–8035. http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf
  53. Privacy-Preserving Deep Learning via Additively Homomorphic Encryption. IEEE Transactions on Information Forensics and Security 13, 5 (2018), 1333–1345. https://doi.org/10.1109/TIFS.2017.2787987
  54. Robust Aggregation for Federated Learning. IEEE Transactions on Signal Processing 70 (2022), 1142–1154. https://doi.org/10.1109/TSP.2022.3153135
  55. John M. Pollard. 1971. The fast Fourier transform in a finite field. Math. Comp. 25 (1971), 365–374.
  56. Boris T Polyak. 1964. Some methods of speeding up the convergence of iteration methods. USSR computational mathematics and mathematical physics 4, 5 (1964), 1–17.
  57. FheFL: Fully Homomorphic Encryption Friendly Privacy-Preserving Federated Learning with Byzantine Users. arXiv preprint arXiv:2306.05112 (2023).
  58. EIFFeL: Ensuring Integrity for Federated Learning. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security (Los Angeles, CA, USA) (CCS ’22). Association for Computing Machinery, New York, NY, USA, 2535–2549. https://doi.org/10.1145/3548606.3560611
  59. SEAL 2023. Microsoft SEAL (release 4.1). https://github.com/Microsoft/SEAL. Microsoft Research, Redmond, WA..
  60. SPEED: secure, PrivatE, and efficient deep learning. Machine Learning 110, 4 (mar 2021), 675–694. https://doi.org/10.1007/s10994-021-05970-3
  61. Secure k-ish Nearest Neighbors Classifier. Proc. Priv. Enhancing Technol. 2020, 3 (2020), 42–61. https://doi.org/10.2478/popets-2020-0045
  62. Byzantine-Resilient Secure Federated Learning. https://doi.org/10.48550/ARXIV.2007.11115
  63. A Secure Federated Learning: Analysis of Different Cryptographic Tools.
  64. Combining homomorphic encryption and differential privacy in federated learning. In Proceedings of the 20th Annual International Conference on Privacy, Security & Trust.
  65. Efficient Private Comparison Queries Over Encrypted Databases Using Fully Homomorphic Encryption With Finite Fields. IEEE Transactions on Dependable and Secure Computing 18, 6 (2021), 2861–2874. https://doi.org/10.1109/TDSC.2020.2967740
  66. A Hybrid Approach to Privacy-Preserving Federated Learning. In Proceedings of the 12th ACM Workshop on Artificial Intelligence and Security (London, United Kingdom) (AISec’19). Association for Computing Machinery, New York, NY, USA, 1–11. https://doi.org/10.1145/3338501.3357370
  67. Secure Byzantine-Robust Distributed Learning via Clustering. arXiv:2110.02940 [cs.CR]
  68. BPFL: A Blockchain Based Privacy-Preserving Federated Learning Scheme. In 2021 IEEE Global Communications Conference (GLOBECOM). 1–6. https://doi.org/10.1109/GLOBECOM46510.2021.9685821
  69. Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms. arXiv preprint arXiv:1708.07747 (2017).
  70. Generalized Byzantine-tolerant SGD. arXiv:1802.10116 [cs.DC]
  71. Fall of Empires: Breaking Byzantine-tolerant SGD by Inner Product Manipulation. In Proceedings of the Thirty-Fifth Conference on Uncertainty in Artificial Intelligence, UAI 2019, Tel Aviv, Israel, July 22-25, 2019. 83.
  72. Byzantine-robust distributed learning: Towards optimal statistical rates. In International Conference on Machine Learning. PMLR, 5650–5659.
  73. Opacus: User-Friendly Differential Privacy Library in PyTorch. https://doi.org/10.48550/ARXIV.2109.12298
  74. BatchCrypt: Efficient Homomorphic Encryption for Cross-Silo Federated Learning. In 2020 USENIX Annual Technical Conference (USENIX ATC 20). USENIX Association, 493–506. https://www.usenix.org/conference/atc20/presentation/zhang-chengliang
  75. LSFL: A Lightweight and Secure Federated Learning Scheme for Edge Computing. IEEE Transactions on Information Forensics and Security 18 (2023), 365–379. https://doi.org/10.1109/TIFS.2022.3221899
  76. idlg: Improved deep leakage from gradients. arXiv preprint arXiv:2001.02610 (2020).
  77. SEAR: Secure and Efficient Aggregation for Byzantine-Robust Federated Learning. IEEE Transactions on Dependable and Secure Computing 19, 05 (sep 2022), 3329–3342. https://doi.org/10.1109/TDSC.2021.3093711
  78. Heng Zhu and Qing Ling. 2022. Bridging Differential Privacy and Byzantine-Robustness via Model Aggregation. arXiv:2205.00107 [cs.LG]
  79. Deep Leakage from Gradients. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32. Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2019/file/60a6c4002cc7b29142def8871531281a-Paper.pdf
  80. Martin Zuber and Renaud Sirdey. 2021. Efficient homomorphic evaluation of k-NN classifiers. Proc. Priv. Enhancing Technol. 2021, 2 (2021), 111–129.
Citations (1)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up