Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

MP-SL: Multihop Parallel Split Learning (2402.00208v1)

Published 31 Jan 2024 in cs.LG and cs.DC

Abstract: Federated Learning (FL) stands out as a widely adopted protocol facilitating the training of Machine Learning (ML) models while maintaining decentralized data. However, challenges arise when dealing with a heterogeneous set of participating devices, causing delays in the training process, particularly among devices with limited resources. Moreover, the task of training ML models with a vast number of parameters demands computing and memory resources beyond the capabilities of small devices, such as mobile and Internet of Things (IoT) devices. To address these issues, techniques like Parallel Split Learning (SL) have been introduced, allowing multiple resource-constrained devices to actively participate in collaborative training processes with assistance from resourceful compute nodes. Nonetheless, a drawback of Parallel SL is the substantial memory allocation required at the compute nodes, for instance training VGG-19 with 100 participants needs 80 GB. In this paper, we introduce Multihop Parallel SL (MP-SL), a modular and extensible ML as a Service (MLaaS) framework designed to facilitate the involvement of resource-constrained devices in collaborative and distributed ML model training. Notably, to alleviate memory demands per compute node, MP-SL supports multihop Parallel SL-based training. This involves splitting the model into multiple parts and utilizing multiple compute nodes in a pipelined manner. Extensive experimentation validates MP-SL's capability to handle system heterogeneity, demonstrating that the multihop configuration proves more efficient than horizontally scaled one-hop Parallel SL setups, especially in scenarios involving more cost-effective compute nodes.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (36)
  1. Fedsl: Federated split learning on distributed sequential data in recurrent neural networks. arXiv preprint arXiv:2011.03180, 2020.
  2. A survey on device-to-device communication in cellular networks. IEEE Communications Surveys & Tutorials, 16(4):1801–1819, 2014.
  3. Practical secure aggregation for privacy-preserving machine learning. In proc. of ACM SIGSAC CCS, pages 1175–1191, 2017.
  4. Cheese: distributed clustering-based hybrid federated split learning over edge networks. IEEE Trans. on Parallel and Distributed Systems, 2023.
  5. Geeps: Scalable deep learning on distributed gpus with a gpu-specialized parameter server. In Proc. of the eleventh european conference on computer systems, pages 1–16, 2016.
  6. Flower: a friendly federated learning framewor. https://flower.dev/.
  7. {{\{{PCAT}}\}}: Functionality and data stealing from split learning by {{\{{Pseudo-Client}}\}} attack. In 32nd USENIX Security Symposium (USENIX Security 23), pages 5271–5288, 2023.
  8. Gharib Gharibi et al. An automated framework for distributed deep learning–a tool demo. In 2022 IEEE 42nd International Conference on Distributed Computing Systems (ICDCS), pages 1302–1305. IEEE, 2022.
  9. In Gim and JeongGil Ko. Memory-efficient dnn training on mobile devices. In Proc. of the 20th Annual International Conference on Mobile Systems, Applications and Services, pages 464–476, 2022.
  10. Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual, 2023.
  11. Deep residual learning for image recognition. In Proc. of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  12. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32, 2019.
  13. Privacy-sensitive parallel split learning. In Int. Conference on Information Networking (ICOIN), pages 7–9. IEEE, 2020.
  14. Beyond data and model parallelism for deep neural networks. Proceed. of Machine Learning and Systems, 1:1–13, 2019.
  15. Neurosurgeon: Collaborative intelligence between the cloud and mobile edge. ACM SIGARCH Computer Architecture News, 45(1):615–629, 2017.
  16. A bargaining game for personalized, energy efficient split learning over wireless networks. In Wireless Communications and Networking Conf.(WCNC), pages 1–6. IEEE, 2023.
  17. Learning multiple layers of features from tiny images. In Toronto, ON, Canada, 2009.
  18. Label leakage and protection in two-party split learning. arXiv preprint arXiv:2102.08504, 2021.
  19. Distance-based online label inference attacks against split learning. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023.
  20. Communication-efficient learning of deep networks from decentralized data. In AISTATS, pages 1273–1282. PMLR, 2017.
  21. Pipedream: Generalized pipeline parallelism for dnn training. In Proc. of the 27th ACM Symposium on Operating Systems Principles, pages 1–15, 2019.
  22. Spliteasy: A practical approach for training ml models on mobile devices. In Proc. of the 22nd HotMobile, page 37–43, 2021.
  23. Ares: Adaptive resource-aware split learning for internet of things. Computer Networks, 218:109380, 2022.
  24. Mohamed Samir et al. Pygrid: A software development and assessment framework for grid-aware software defined networking. International Journal of Network Management, 28(5):e2033, 2018.
  25. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
  26. Splitfed: When federated learning meets split learning. In Proc. of the AAAI Conference on Artificial Intelligence, volume 36, pages 8485–8493, 2022.
  27. The role of compute nodes in privacy-aware decentralized ai. In Proc. of EMDL, pages 19–24, 2022.
  28. Workflow optimization for parallel split learning. In proc. of IEEE INFOCOM, 2024.
  29. Split learning for health: Distributed deep learning without sharing raw patient data. arXiv preprint arXiv:1812.00564, 2018.
  30. Nopeek: Information leakage reduction to share activations in distributed deep learning. In 2020 International Conference on Data Mining Workshops (ICDMW), pages 933–942. IEEE, 2020.
  31. Melon: Breaking the memory wall for resource-efficient on-device machine learning. In Proc. of International Conference on Mobile Systems, Applications and Services, pages 450–463, 2022.
  32. Coopfl: Accelerating federated learning with dnn partitioning and offloading in heterogeneous edge computing. Computer Networks, 220:109490, 2023.
  33. Fedadapt: Adaptive offloading for iot devices in federated learning. IEEE Internet of Things Journal, 2022.
  34. Split learning over wireless networks: Parallel design and resource management. IEEE Journal on Selected Areas in Communications, 41(4):1051–1066, 2023.
  35. Alpa: Automating inter-and {{\{{Intra-Operator}}\}} parallelism for distributed deep learning. In 16th USENIX OSDI, pages 559–578, 2022.
  36. Alexander Ziller et al. Pysyft: A library for easy federated learning. In Federated Learning Systems, pages 111–139. Springer, 2021.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Joana Tirana (2 papers)
  2. Spyros Lalis (9 papers)
  3. Dimitris Chatzopoulos (14 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now