Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
144 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Privacy-aware Berrut Approximated Coded Computing for Federated Learning (2405.01704v3)

Published 2 May 2024 in cs.LG, cs.CC, cs.DC, cs.IT, and math.IT

Abstract: Federated Learning (FL) is an interesting strategy that enables the collaborative training of an AI model among different data owners without revealing their private datasets. Even so, FL has some privacy vulnerabilities that have been tried to be overcome by applying some techniques like Differential Privacy (DP), Homomorphic Encryption (HE), or Secure Multi-Party Computation (SMPC). However, these techniques have some important drawbacks that might narrow their range of application: problems to work with non-linear functions and to operate large matrix multiplications and high communication and computational costs to manage semi-honest nodes. In this context, we propose a solution to guarantee privacy in FL schemes that simultaneously solves the previously mentioned problems. Our proposal is based on the Berrut Approximated Coded Computing, a technique from the Coded Distributed Computing paradigm, adapted to a Secret Sharing configuration, to provide input privacy to FL in a scalable way. It can be applied for computing non-linear functions and treats the special case of distributed matrix multiplication, a key primitive at the core of many automated learning tasks. Because of these characteristics, it could be applied in a wide range of FL scenarios, since it is independent of the machine learning models or aggregation algorithms used in the FL scheme. We provide analysis of the achieved privacy and complexity of our solution and, due to the extensive numerical results performed, a good trade-off between privacy and precision can be observed.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (44)
  1. M. Khan, F. G. Glavin, and M. Nickles, “Federated learning as a privacy solution - an overview,” Procedia Computer Science, vol. 217, pp. 316–325, 2023, 4th Int. Conf. on Industry 4.0 and Smart Manufacturing. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1877050922023055
  2. L. Lyu, H. Yu, X. Ma, C. Chen, L. Sun, J. Zhao, Q. Yang, and S. Y. Philip, “Privacy and robustness in federated learning: Attacks and defenses,” IEEE Trans. on neural networks and learning systems, 2022.
  3. L. Lyu, H. Yu, and Q. Yang, “Threats to federated learning: A survey,” arXiv preprint arXiv:2003.02133, 2020.
  4. N. Carlini, S. Chien, M. Nasr, S. Song, A. Terzis, and F. Tramèr, “Membership inference attacks from first principles,” in 43rd IEEE Symp. on Security and Privacy, SP 2022, San Francisco, CA, USA, May 22-26, 2022.   IEEE, 2022, pp. 1897–1914.
  5. H. Hu, Z. Salcic, L. Sun, G. Dobbie, P. S. Yu, and X. Zhang, “Membership inference attacks on machine learning: A survey,” ACM Computing Surveys, vol. 54, no. 11s, pp. 1–37, 2022.
  6. Z. Wang, M. Song, Z. Zhang, Y. Song, Q. Wang, and H. Qi, “Beyond inferring class representatives: User-level privacy leakage from federated learning,” in IEEE INFOCOM 2019-IEEE conference on computer communications.   IEEE, 2019, pp. 2512–2520.
  7. L. Melis, C. Song, E. De Cristofaro, and V. Shmatikov, “Exploiting unintended feature leakage in collaborative learning,” in 2019 IEEE Symp. on security and privacy (SP).   IEEE, 2019, pp. 691–706.
  8. J. Geiping, H. Bauermeister, H. Dröge, and M. Moeller, “Inverting gradients-how easy is it to break privacy in federated learning?” Adv. in neural information processing systems, vol. 33, pp. 16 937–16 947, 2020.
  9. X. Yin, Y. Zhu, and J. Hu, “A comprehensive survey of privacy-preserving federated learning: A taxonomy, review, and future directions,” ACM Computing Surveys, vol. 54, no. 6, pp. 1–36, 2021.
  10. C. Dwork, “Differential privacy,” in Int. Colloquium on Automata, Languages, and Programming.   Berlin, Heidelberg: Springer, 2006, pp. 1–12.
  11. M. Chase, H. Chen, J. Ding, S. Goldwasser, S. Gorbunov, J. Hoffstein, K. Lauter, S. Lokam, D. Moody, T. Morrison, A. Sahai, and V. Vaikuntanathan, “Security of homomorphic encryption,” HomomorphicEncryption.org, Redmond, WA, Tech. Rep., July 2017.
  12. D. Evans, V. Kolesnikov, and M. Rosulek, “A pragmatic introduction to secure multi-party computation,” Foundations and Trends® in Privacy and Security, vol. 2, pp. 70–246, 2018.
  13. P. K. et al., “Advances and open problems in federated learning,” Foundations and Trends in Machine Learning, vol. 14, no. 1–2, pp. 1–210, 2021. [Online]. Available: http://dx.doi.org/10.1561/2200000083
  14. S. Sharma, S. Sharma, and A. Athaiya, “Activation functions in neural networks,” Towards Data Sci, vol. 6, no. 12, pp. 310–316, 2017.
  15. B. George, S. Seals, and I. Aban, “Survival analysis and regression models,” Journal of Nuclear Cardiology, vol. 21, no. 4, pp. 686–694, 2014. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1071358123073154
  16. S. Huda, J. Yearwood, H. F. Jelinek, M. M. Hassan, G. Fortino, and M. Buckland, “A hybrid feature selection with ensemble classification for imbalanced healthcare data: A case study for brain tumor diagnosis,” IEEE Access, vol. 4, pp. 9145–9154, 2016.
  17. K. Pillutla, S. M. Kakade, and Z. Harchaoui, “Robust aggregation for federated learning,” IEEE Trans. on Signal Processing, vol. 70, pp. 1142–1154, 2022.
  18. S. Li, E. Ngai, and T. Voigt, “Byzantine-robust aggregation in federated learning empowered industrial iot,” IEEE Trans. on Industrial Informatics, vol. 19, no. 2, pp. 1165–1175, 2023.
  19. R. Yuster and U. Zwick, “Fast sparse matrix multiplication,” ACM Trans. Algorithms, vol. 1, no. 1, p. 2–13, jul 2005. [Online]. Available: https://doi.org/10.1145/1077464.1077466
  20. S. Smith, N. Ravindran, N. D. Sidiropoulos, and G. Karypis, “Splatt: Efficient and parallel sparse tensor-matrix multiplication,” in 2015 IEEE Int. Parallel and Distributed Processing Symposium, 2015, pp. 61–70.
  21. Z. Zhang, H. Wang, S. Han, and W. J. Dally, “Sparch: Efficient architecture for sparse matrix multiplication,” in 2020 IEEE Int. Symp. on High Performance Computer Architecture (HPCA), 2020, pp. 261–274.
  22. S. Li and S. Avestimehr, “Coded computing: Mitigating fundamental bottlenecks in large-scale distributed computing and machine learning,” Foundations and Trends in Communications and Information Theory, vol. 17, no. 1, pp. 1–148, 2020. [Online]. Available: http://dx.doi.org/10.1561/0100000103
  23. J. S. Ng, W. Y. B. Lim, N. C. Luong, Z. Xiong, A. Asheralieva, D. Niyato, C. Leung, and C. Miao, “A survey of coded distributed computing,” 2020.
  24. A. Beimel, “Secret-sharing schemes: A survey,” in Int. Conf. on coding and cryptology.   Springer, 2011, pp. 11–46.
  25. A. K. Chattopadhyay, S. Saha, A. Nag, and S. Nandi, “Secret sharing: A comprehensive survey, taxonomy and applications,” Computer Science Review, vol. 51, p. 100608, 2024.
  26. A. Shamir, “How to share a secret,” Commun. ACM, vol. 22, no. 11, pp. 612–613, nov 1979. [Online]. Available: https://doi.org/10.1145/359168.359176
  27. T. Jahani-Nezhad and M. A. Maddah-Ali, “Berrut approximated coded computing: Straggler resistance beyond polynomial computing,” IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 45, no. 1, pp. 111–122, 2023.
  28. S. Li, M. A. Maddah-Ali, Q. Yu, and A. S. Avestimehr, “A fundamental tradeoff between computation and communication in distributed computing,” IEEE Trans. on Inf. Theory, vol. 64, no. 1, pp. 109–128, 2017.
  29. K. Lee, M. Lam, R. Pedarsani, D. Papailiopoulos, and K. Ramchandran, “Speeding up distributed machine learning using codes,” IEEE Trans. on Inf. Theory, vol. 64, no. 3, pp. 1514–1529, 2017.
  30. S. Li, M. A. Maddah-Ali, and A. S. Avestimehr, “Coded mapreduce,” in 2015 53rd Annual Allerton Conf. on Communication, Control, and Computing (Allerton).   IEEE, 2015, pp. 964–971.
  31. S. Ulukus, S. Avestimehr, M. Gastpar, S. A. Jafar, R. Tandon, and C. Tian, “Private retrieval, computing, and learning: Recent progress and future challenges,” IEEE J. on Selected Areas in Communications, vol. 40, no. 3, pp. 729–748, 2022.
  32. Q. Yu, S. Li, N. Raviv, S. M. M. Kalan, M. Soltanolkotabi, and S. A. Avestimehr, “Lagrange coded computing: Optimal design for resiliency, security, and privacy,” in Proc. of the 22nd Int. Conf. on Artificial Intelligence and Statistics, vol. 89.   PMLR, 16–18 Apr 2019, pp. 1215–1225. [Online]. Available: https://proceedings.mlr.press/v89/yu19b.html
  33. M. Soleymani, H. Mahdavifar, and A. S. Avestimehr, “Analog lagrange coded computing,” IEEE J. on Selected Areas in Information Theory, vol. 2, no. 1, pp. 283–295, 2021.
  34. H. A. Nodehi and M. A. Maddah-Ali, “Secure coded multi-party computation for massive matrix operations,” in arXiv preprint arXiv:1908.04255, 2019.
  35. M. Kim, H. Yang, and J. Lee, “Fully private and secure coded matrix multiplication with colluding workers,” ICT Express, vol. 9, no. 4, pp. 722–727, 2023. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S2405959523000115
  36. Q. Yu and A. S. Avestimehr, “Entangled polynomial codes for secure, private, and batch distributed matrix multiplication: Breaking the "cubic" barrier,” in 2020 IEEE Int. Symp. on Information Theory (ISIT), 2020, pp. 245–250.
  37. T. Jahani-Nezhad and M. A. Maddah-Ali, “Codedsketch: A coding scheme for distributed computation of approximated matrix multiplication,” IEEE Trans. on Inf. Theory, vol. 67, no. 6, pp. 4185–4196, 2021.
  38. J. Zhu, Q. Yan, X. Tang, and S. Li, “Symmetric private polynomial computation from lagrange encoding,” IEEE Trans. on Information Theory, vol. 68, no. 4, pp. 2704–2718, Apr. 2022.
  39. N. Raviv and D. A. Karpuk, “Private polynomial computation from lagrange encoding,” in 2019 IEEE Int. Symp. on Information Theory (ISIT).   IEEE, Jul. 2019.
  40. I. Issa, A. B. Wagner, and S. Kamath, “An operational approach to information leakage,” IEEE Trans. on Information Theory, vol. 66, no. 3, pp. 1625–1657, 2020.
  41. L. Schumacher, K. Pedersen, and P. Mogensen, “From antenna spacings to theoretical capacities - guidelines for simulating mimo systems,” in The 13th IEEE Int. Symp. on Personal, Indoor and Mobile Radio Communications, vol. 2, 2002, pp. 587–592 vol.2.
  42. H. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Int. Conf. on Artificial Intelligence and Statistics, 2016. [Online]. Available: https://api.semanticscholar.org/CorpusID:14955348
  43. S. P. Karimireddy, S. Kale, M. Mohri, S. Reddi, S. Stich, and A. T. Suresh, “SCAFFOLD: Stochastic controlled averaging for federated learning,” in Proc. of the 37th Int. Conf. on Machine Learning, vol. 119.   PMLR, 13–18 Jul 2020, pp. 5132–5143. [Online]. Available: https://proceedings.mlr.press/v119/karimireddy20a.html
  44. Z. Zhu, J. Hong, and J. Zhou, “Data-free knowledge distillation for heterogeneous federated learning,” in Proc. of the 38th Int. Conf. on Machine Learning, ser. Proc. of Machine Learning Research, M. Meila and T. Zhang, Eds., vol. 139.   PMLR, 18–24 Jul 2021, pp. 12 878–12 889. [Online]. Available: https://proceedings.mlr.press/v139/zhu21b.html
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now