Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Communication-Efficient Multimodal Federated Learning: Joint Modality and Client Selection (2401.16685v1)

Published 30 Jan 2024 in cs.LG and cs.DC

Abstract: Multimodal federated learning (FL) aims to enrich model training in FL settings where clients are collecting measurements across multiple modalities. However, key challenges to multimodal FL remain unaddressed, particularly in heterogeneous network settings where: (i) the set of modalities collected by each client will be diverse, and (ii) communication limitations prevent clients from uploading all their locally trained modality models to the server. In this paper, we propose multimodal Federated learning with joint Modality and Client selection (mmFedMC), a new FL methodology that can tackle the above-mentioned challenges in multimodal settings. The joint selection algorithm incorporates two main components: (a) A modality selection methodology for each client, which weighs (i) the impact of the modality, gauged by Shapley value analysis, (ii) the modality model size as a gauge of communication overhead, against (iii) the frequency of modality model updates, denoted recency, to enhance generalizability. (b) A client selection strategy for the server based on the local loss of modality model at each client. Experiments on five real-world datasets demonstrate the ability of mmFedMC to achieve comparable accuracy to several baselines while reducing the communication overhead by over 20x. A demo video of our methodology is available at https://liangqiy.com/mmfedmc/.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (59)
  1. L. Yuan, D.-J. Han, V. P. Chellapandi, S. H. Żak, and C. G. Brinton, “Fedmfs: Federated multimodal fusion learning with selective modality communication,” arXiv preprint arXiv:2310.07048, 2023.
  2. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Artificial intelligence and statistics.   PMLR, 2017, pp. 1273–1282.
  3. L. Yuan, L. Sun, P. S. Yu, and Z. Wang, “Decentralized federated learning: A survey and perspective,” arXiv preprint arXiv:2306.01603, 2023.
  4. W. Fang, Z. Yu, Y. Jiang, Y. Shi, C. N. Jones, and Y. Zhou, “Communication-efficient stochastic zeroth-order optimization for federated learning,” IEEE Transactions on Signal Processing, vol. 70, pp. 5058–5073, 2022.
  5. V. P. Chellapandi, A. Upadhyay, A. Hashemi, and S. H. Żak, “On the Convergence of Decentralized Federated Learning Under Imperfect Information Sharing,” IEEE Control Systems Letters, 2023.
  6. L. Yuan, Z. Wang, and C. G. Brinton, “Digital ethics in federated learning,” arXiv preprint arXiv:2310.03178, 2023.
  7. H. Yu, Z. Chen, X. Zhang, X. Chen, F. Zhuang, H. Xiong, and X. Cheng, “Fedhar: Semi-supervised online learning for personalized federated human activity recognition,” IEEE Transactions on Mobile Computing, 2021.
  8. Q. Wu, X. Chen, Z. Zhou, and J. Zhang, “Fedhome: Cloud-edge based personalized federated learning for in-home health monitoring,” IEEE Transactions on Mobile Computing, vol. 21, no. 8, pp. 2818–2832, 2020.
  9. S. Baghersalimi, T. Teijeiro, A. Aminifar, and D. Atienza, “Decentralized federated learning for epileptic seizures detection in low-power wearable systems,” IEEE Transactions on Mobile Computing, 2023.
  10. L. Yuan, Y. Ma, L. Su, and Z. Wang, “Peer-to-peer federated continual learning for naturalistic driving action recognition,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 5249–5258.
  11. V. P. Chellapandi, L. Yuan, S. H. Zak, and Z. Wang, “A Survey of Federated Learning for Connected and Automated Vehicles,” 2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC), 2023.
  12. V. P. Chellapandi, L. Yuan, C. G. Brinton, S. H. Zak, and Z. Wang, “Federated Learning for Connected and Automated Vehicles: A Survey of Existing Approaches and Challenges,” IEEE Transactions on Intelligent Vehicles, 2023.
  13. D. Upadhyay, H. Shui, and D. Filev, “Methods and systems for estimating a remaining useful life of an asset,” Mar. 2 2023, uS Patent App. 17/446,708.
  14. D. Lahat, T. Adali, and C. Jutten, “Multimodal data fusion: an overview of methods, challenges, and prospects,” Proceedings of the IEEE, vol. 103, no. 9, pp. 1449–1477, 2015.
  15. T. Baltrušaitis, C. Ahuja, and L.-P. Morency, “Multimodal machine learning: A survey and taxonomy,” IEEE transactions on pattern analysis and machine intelligence, vol. 41, no. 2, pp. 423–443, 2018.
  16. G. Lee, S.-H. Jung, and D. Han, “An adaptive sensor fusion framework for pedestrian indoor navigation in dynamic environments,” IEEE Transactions on Mobile Computing, vol. 20, no. 2, pp. 320–336, 2019.
  17. Y. Li, L. Liu, H. Qin, S. Deng, M. A. El-Yacoubi, and G. Zhou, “Adaptive deep feature fusion for continuous authentication with data augmentation,” IEEE Transactions on Mobile Computing, 2022.
  18. J. Zhang, Q. Wang, Q. Wang, and Z. Zheng, “Multimodal fusion framework based on statistical attention and contrastive attention for sign language recognition,” IEEE Transactions on Mobile Computing, 2023.
  19. P. Bharti, D. De, S. Chellappan, and S. K. Das, “Human: Complex activity recognition with multi-modal multi-positional body sensing,” IEEE Transactions on Mobile Computing, vol. 18, no. 4, pp. 857–870, 2018.
  20. P. K. Atrey, M. A. Hossain, A. El Saddik, and M. S. Kankanhalli, “Multimodal fusion for multimedia analysis: a survey,” Multimedia systems, vol. 16, pp. 345–379, 2010.
  21. S. K. D’mello and J. Kory, “A review and meta-analysis of multimodal affect detection systems,” ACM computing surveys (CSUR), vol. 47, no. 3, pp. 1–36, 2015.
  22. D. Ramachandram and G. W. Taylor, “Deep multimodal learning: A survey on recent advances and trends,” IEEE signal processing magazine, vol. 34, no. 6, pp. 96–108, 2017.
  23. A. Kline, H. Wang, Y. Li, S. Dennis, M. Hutch, Z. Xu, F. Wang, F. Cheng, and Y. Luo, “Multimodal machine learning in precision health: A scoping review,” npj Digital Medicine, vol. 5, no. 1, p. 171, 2022.
  24. P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings et al., “Advances and open problems in federated learning,” Foundations and Trends® in Machine Learning, vol. 14, no. 1–2, pp. 1–210, 2021.
  25. Y. Zhao, P. Barnaghi, and H. Haddadi, “Multimodal federated learning on iot data,” in 2022 IEEE/ACM Seventh International Conference on Internet-of-Things Design and Implementation (IoTDI).   IEEE, 2022, pp. 43–54.
  26. F. Sattler, S. Wiedemann, K.-R. Müller, and W. Samek, “Robust and communication-efficient federated learning from non-iid data,” IEEE transactions on neural networks and learning systems, vol. 31, no. 9, pp. 3400–3413, 2019.
  27. W. Y. B. Lim, N. C. Luong, D. T. Hoang, Y. Jiao, Y.-C. Liang, Q. Yang, D. Niyato, and C. Miao, “Federated learning in mobile edge networks: A comprehensive survey,” IEEE Communications Surveys & Tutorials, vol. 22, no. 3, pp. 2031–2063, 2020.
  28. A. Imteaj, U. Thakker, S. Wang, J. Li, and M. H. Amini, “A survey on federated learning for resource-constrained iot devices,” IEEE Internet of Things Journal, vol. 9, no. 1, pp. 1–24, 2021.
  29. W. Fang, D.-J. Han, and C. G. Brinton, “Submodel partitioning in hierarchical federated learning: Algorithm design and convergence analysis,” arXiv preprint arXiv:2310.17890, 2023.
  30. J. Zhang, X. Cheng, C. Wang, Y. Wang, Z. Shi, J. Jin, A. Song, W. Zhao, L. Wen, and T. Zhang, “Fedada: Fast-convergent adaptive federated learning in heterogeneous mobile edge computing environment,” World Wide Web, vol. 25, no. 5, pp. 1971–1998, 2022.
  31. Y.-W. Chu, S. Hosseinalipour, E. Tenorio, L. Cruz, K. Douglas, A. Lan, and C. Brinton, “Mitigating biases in student performance prediction via attention-based personalized federated learning,” in Proceedings of the 31st ACM International Conference on Information & Knowledge Management, 2022, pp. 3033–3042.
  32. A. M. Abdelmoniem, C.-Y. Ho, P. Papageorgiou, and M. Canini, “Empirical analysis of federated learning in heterogeneous environments,” in Proceedings of the 2nd European Workshop on Machine Learning and Systems, 2022, pp. 1–9.
  33. M. Sundararajan and A. Najmi, “The many shapley values for model explanation,” in International conference on machine learning.   PMLR, 2020, pp. 9269–9278.
  34. S. M. Lundberg and S.-I. Lee, “A unified approach to interpreting model predictions,” Advances in neural information processing systems, vol. 30, 2017.
  35. S. M. Lundberg, G. Erion, H. Chen, A. DeGrave, J. M. Prutkin, B. Nair, R. Katz, J. Himmelfarb, N. Bansal, and S.-I. Lee, “From local explanations to global understanding with explainable ai for trees,” Nature Machine Intelligence, vol. 2, no. 1, pp. 2522–5839, 2020.
  36. Y. J. Cho, J. Wang, and G. Joshi, “Client selection in federated learning: Convergence analysis and power-of-choice selection strategies,” arXiv preprint arXiv:2010.01243, 2020.
  37. P. Qi, D. Chiaro, and F. Piccialli, “FL-FD: Federated learning-based fall detection with multimodal data fusion,” Information Fusion, p. 101890, 2023.
  38. B. Xiong, X. Yang, F. Qi, and C. Xu, “A unified framework for multi-modal federated learning,” Neurocomputing, vol. 480, pp. 110–118, 2022.
  39. T. Feng, D. Bose, T. Zhang, R. Hebbar, A. Ramakrishna, R. Gupta, M. Zhang, S. Avestimehr, and S. Narayanan, “Fedmultimodal: A benchmark for multimodal federated learning,” arXiv preprint arXiv:2306.09486, 2023.
  40. B. Salehi, J. Gu, D. Roy, and K. Chowdhury, “Flash: Federated learning for automated selection of high-band mmwave sectors,” in IEEE INFOCOM 2022-IEEE Conference on Computer Communications.   IEEE, 2022, pp. 1719–1728.
  41. J. Chen and A. Zhang, “Fedmsplit: Correlation-adaptive federated multi-task learning across multimodal split networks,” in Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022, pp. 87–96.
  42. L. Yuan, H. Chen, R. Ewing, and J. Li, “Passive radio frequency-based 3d indoor positioning system via ensemble learning,” arXiv preprint arXiv:2304.06513, 2023.
  43. S. Yang, L. Yuan, and J. Li, “Extraction and denoising of human signature on radio frequency spectrums,” in 2023 IEEE International Conference on Consumer Electronics (ICCE).   IEEE, 2023, pp. 1–6.
  44. L. Yuan, J. Andrews, H. Mu, A. Vakil, R. Ewing, E. Blasch, and J. Li, “Interpretable passive multi-modal sensor fusion for human identification and activity recognition,” Sensors, vol. 22, no. 15, p. 5787, 2022.
  45. L. Yuan, H. Chen, R. Ewing, E. Blasch, and J. Li, “Three dimensional indoor positioning based on passive radio frequency signal strength distribution,” IEEE Internet of Things Journal, vol. 10, no. 15, pp. 13 933–13 944, 2023.
  46. Z. Wang, Z. Wan, and X. Wan, “Transmodality: An end2end fusion method with transformer for multimodal sentiment analysis,” in Proceedings of The Web Conference 2020, 2020, pp. 2514–2520.
  47. Q. Pan, H. Cao, Y. Zhu, J. Liu, and B. Li, “Contextual client selection for efficient federated learning over edge devices,” IEEE Transactions on Mobile Computing, 2023.
  48. Y. Xu, Z. Jiang, H. Xu, Z. Wang, C. Qian, and C. Qiao, “Federated learning with client selection and gradient compression in heterogeneous edge systems,” IEEE Transactions on Mobile Computing, 2023.
  49. B. Luo, W. Xiao, S. Wang, J. Huang, and L. Tassiulas, “Tackling system and statistical heterogeneity for federated learning with adaptive client sampling,” in IEEE INFOCOM 2022-IEEE conference on computer communications.   IEEE, 2022, pp. 1739–1748.
  50. L. Fu, H. Zhang, G. Gao, M. Zhang, and X. Liu, “Client selection in federated learning: Principles, challenges, and opportunities,” IEEE Internet of Things Journal, 2023.
  51. S. Wang, R. Morabito, S. Hosseinalipour, M. Chiang, and C. G. Brinton, “Device sampling and resource optimization for federated learning in cooperative edge networks,” arXiv preprint arXiv:2311.04350, 2023.
  52. L. Yuan, L. Su, and Z. Wang, “Federated transfer-ordered-personalized learning for driver monitoring application,” IEEE Internet of Things Journal, vol. 10, no. 20, pp. 18 292–18 301, May 2023.
  53. J. DelPreto, C. Liu, Y. Luo, M. Foshey, Y. Li, A. Torralba, W. Matusik, and D. Rus, “Actionsense: A multimodal dataset and recording framework for human activities using wearable sensors in a kitchen environment,” Advances in Neural Information Processing Systems, vol. 35, pp. 13 800–13 813, 2022.
  54. D. Anguita, A. Ghio, L. Oneto, X. Parra, J. L. Reyes-Ortiz et al., “A public domain dataset for human activity recognition using smartphones.” in Esann, vol. 3, 2013, p. 3.
  55. P. Wagner, N. Strodthoff, R.-D. Bousseljot, D. Kreiseler, F. I. Lunze, W. Samek, and T. Schaeffter, “Ptb-xl, a large publicly available electrocardiography dataset,” Scientific data, vol. 7, no. 1, p. 154, 2020.
  56. N. Strodthoff, P. Wagner, T. Schaeffter, and W. Samek, “Deep learning for ecg analysis: Benchmarks and insights from ptb-xl,” IEEE Journal of Biomedical and Health Informatics, vol. 25, no. 5, pp. 1519–1528, 2020.
  57. S. Poria, D. Hazarika, N. Majumder, G. Naik, E. Cambria, and R. Mihalcea, “MELD: A multimodal multi-party dataset for emotion recognition in conversations,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.   Florence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 527–536. [Online]. Available: https://aclanthology.org/P19-1050
  58. C. Persello, R. Hänsch, G. Vivone, K. Chen, Z. Yan, D. Tang, H. Huang, M. Schmitt, and X. Sun, “2023 ieee grss data fusion contest: Large-scale fine-grained building classification for semantic urban reconstruction,” 2022. [Online]. Available: https://dx.doi.org/10.21227/mrnt-8w27
  59. R. Chen, L. Li, K. Xue, C. Zhang, M. Pan, and Y. Fang, “Energy efficient federated learning over heterogeneous mobile devices via joint design of weight quantization and wireless transmission,” IEEE Transactions on Mobile Computing, 2022.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Liangqi Yuan (17 papers)
  2. Dong-Jun Han (32 papers)
  3. Su Wang (66 papers)
  4. Devesh Upadhyay (23 papers)
  5. Christopher G. Brinton (109 papers)
Citations (7)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube