Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Federated Learning for Time-Series Healthcare Sensing with Incomplete Modalities (2405.11828v2)

Published 20 May 2024 in cs.LG

Abstract: Many healthcare sensing applications utilize multimodal time-series data from sensors embedded in mobile and wearable devices. Federated Learning (FL), with its privacy-preserving advantages, is particularly well-suited for health applications. However, most multimodal FL methods assume the availability of complete modality data for local training, which is often unrealistic. Moreover, recent approaches tackling incomplete modalities scale poorly and become inefficient as the number of modalities increases. To address these limitations, we propose FLISM, an efficient FL training algorithm with incomplete sensing modalities while maintaining high accuracy. FLISM employs three key techniques: (1) modality-invariant representation learning to extract effective features from clients with a diverse set of modalities, (2) modality quality-aware aggregation to prioritize contributions from clients with higher-quality modality data, and (3) global-aligned knowledge distillation to reduce local update shifts caused by modality differences. Extensive experiments on real-world datasets show that FLISM not only achieves high accuracy but is also faster and more efficient compared with state-of-the-art methods handling incomplete modality problems in FL. We release the code as open-source at https://github.com/AdibaOrz/FLISM.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (65)
  1. 1951. On information and sufficiency. The annals of mathematical statistics 22, 1 (1951), 79–86.
  2. 2021. pytorch-OpCounter: Tool to count the FLOPs of your PyTorch model. https://github.com/Lyken17/pytorch-OpCounter/.
  3. applevisionpro 2024. Apple Vision Pro. https://www.apple.com/apple-vision-pro/.
  4. Gated multimodal units for information fusion. arXiv preprint arXiv:1702.01992 (2017).
  5. Deep learning–based multimodal data fusion: Case study in food intake episodes detection using wearable sensors. JMIR mHealth and uHealth 9, 1 (2021), e21926.
  6. Pierre Baldi. 2012. Autoencoders, unsupervised learning, and deep architectures. In Proceedings of ICML workshop on unsupervised and transfer learning. JMLR Workshop and Conference Proceedings, 37–49.
  7. Seeing through fog without seeing fog: Deep multimodal sensor fusion in unseen adverse weather. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 11682–11692.
  8. Flame: Federated learning across multi-device environments. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6, 3 (2022), 1–29.
  9. Cocoa: Cross modality contrastive learning for sensor data. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6, 3 (2022), 1–28.
  10. Multimodal multitask deep learning model for Alzheimer’s disease progression detection based on time series data. Neurocomputing 412 (2020), 197–215.
  11. FedMultimodal: A Benchmark For Multimodal Federated Learning. arXiv preprint arXiv:2306.09486 (2023).
  12. Tiantian Feng and Shrikanth Narayanan. 2019. Imputing missing data in large-scale multivariate biomedical wearable recordings using bidirectional recurrent neural networks with temporal activation regularization. In 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE, 2529–2534.
  13. Knocker: Vibroacoustic-based object recognition with smartphones. Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies 3, 3 (2019), 1–21.
  14. Assessing the state of self-supervised human activity recognition using wearables. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6, 3 (2022), 1–47.
  15. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015).
  16. Collossl: Collaborative self-supervised learning for human activity recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6, 1 (2022), 1–28.
  17. Supervised contrastive learning. Advances in neural information processing systems 33 (2020), 18661–18673.
  18. FedFN: Feature Normalization for Alleviating Data Heterogeneity Problem in Federated Learning. arXiv preprint arXiv:2311.13267 (2023).
  19. Sylvia D Kreibig. 2010. Autonomic nervous system activity in emotion: A review. Biological psychology 84, 3 (2010), 394–421.
  20. FedMEKT: Distillation-based Embedding Knowledge Transfer for Multimodal Federated Learning. arXiv preprint arXiv:2307.13214 (2023).
  21. Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks 3361, 10 (1995), 1995.
  22. Model-contrastive federated learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 10713–10722.
  23. Federated learning: Challenges, methods, and future directions. IEEE signal processing magazine 37, 3 (2020), 50–60.
  24. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems 2 (2020), 429–450.
  25. Emotion recognition from multi-channel EEG data through convolutional recurrent neural network. In 2016 IEEE international conference on bioinformatics and biomedicine (BIBM). IEEE, 352–359.
  26. Deepfusion: Lidar-camera deep fusion for multi-modal 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 17182–17191.
  27. Real time object detection using LiDAR and camera fusion for autonomous driving. Scientific Reports 13, 1 (2023), 8056.
  28. Oura Health Ltd. 2015. Oura Ring. https://ouraring.com/.
  29. Are multimodal transformers robust to missing modality?. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 18177–18186.
  30. Smil: Multimodal learning with severely missing modality. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 2302–2310.
  31. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics. PMLR, 1273–1282.
  32. Multimodality sensing for eating recognition.. In PervasiveHealth. 130–137.
  33. Francisco Javier Ordóñez and Daniel Roggen. 2016. Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition. Sensors 16, 1 (2016), 115.
  34. Time2Stop: Adaptive and Explainable Human-AI Loop for Smartphone Overuse Intervention. arXiv preprint arXiv:2403.05584 (2024).
  35. ADMarker: A Multi-Modal Federated Learning System for Monitoring Digital Biomarkers of Alzheimer’s Disease. arXiv preprint arXiv:2310.15301 (2023).
  36. Harmony: Heterogeneous Multi-Modal Federated Learning through Disentangled Model Training. In Proceedings of the 21st Annual International Conference on Mobile Systems, Applications and Services. 530–543.
  37. K-EmoCon, a multimodal sensor dataset for continuous emotion recognition in naturalistic conversations. Scientific Data 7, 1 (2020), 293.
  38. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32 (2019).
  39. Thomas PlÖtz. 2021. Applying machine learning for sensor data analysis in interactive systems: Common pitfalls of pragmatic use and ways to avoid them. ACM Computing Surveys (CSUR) 54, 6 (2021), 1–25.
  40. Learning transferable visual models from natural language supervision. In International conference on machine learning. PMLR, 8748–8763.
  41. Dhanesh Ramachandram and Graham W Taylor. 2017. Deep multimodal learning: A survey on recent advances and trends. IEEE signal processing magazine 34, 6 (2017), 96–108.
  42. Attila Reiss and Didier Stricker. 2012. Introducing a new benchmarked dataset for activity monitoring. In 2012 16th international symposium on wearable computers. IEEE, 108–109.
  43. Pic2word: Mapping pictures to words for zero-shot composed image retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 19305–19314.
  44. Flash: Federated learning for automated selection of high-band mmwave sectors. In IEEE INFOCOM 2022-IEEE Conference on Computer Communications. IEEE, 1719–1728.
  45. Introducing wesad, a multimodal dataset for wearable stress and affect detection. In Proceedings of the 20th ACM international conference on multimodal interaction. 400–408.
  46. VIGAN: Missing view imputation with generative adversarial networks. In 2017 IEEE International conference on big data (Big Data). IEEE, 766–775.
  47. MyDJ: Sensing food intakes with an attachable on your eyeglass frame. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 1–17.
  48. Timo Sztyler and Heiner Stuckenschmidt. 2016. On-body localization of wearable devices: An investigation of position-aware activity recognition. In 2016 IEEE International Conference on Pervasive Computing and Communications (PerCom). IEEE, 1–9.
  49. Context recognition in-the-wild: Unified model for multi-modal sensors and multi-label classification. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 4 (2018), 1–22.
  50. Stef Van Buuren. 2018. Flexible imputation of missing data. CRC press.
  51. Attention is all you need. Advances in neural information processing systems 30 (2017).
  52. Lele Wang and Yingping Huang. 2022. LiDAR–camera fusion for road detection using a recurrent conditional random field model. Scientific Reports 12, 1 (2022), 11320.
  53. StudentLife: assessing mental health, academic performance and behavioral trends of college students using smartphones. In Proceedings of the 2014 ACM international joint conference on pervasive and ubiquitous computing. 3–14.
  54. A unified framework for multi-modal federated learning. Neurocomputing 480 (2022), 110–118.
  55. Characterizing impacts of heterogeneity in federated learning upon large-scale smartphone data. In Proceedings of the Web Conference 2021. 935–946.
  56. Deep convolutional neural networks on multichannel time series for human activity recognition.. In Ijcai, Vol. 15. Buenos Aires, Argentina, 3995–4001.
  57. Optimal sparse linear prediction for block-missing multi-modality data without imputation. J. Amer. Statist. Assoc. 115, 531 (2020), 1406–1419.
  58. Han Yu and Akane Sano. 2023. Semi-supervised learning for wearable-based momentary stress detection in the wild. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 7, 2 (2023), 1–23.
  59. Multimodal federated learning via contrastive representation ensemble. arXiv preprint arXiv:2302.08888 (2023).
  60. Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv preprint arXiv:1606.06259 (2016).
  61. Convolutional neural networks for human activity recognition using mobile sensors. In 6th international conference on mobile computing, applications and services. IEEE, 197–205.
  62. Unified multi-modal image synthesis for missing modality imputation. arXiv preprint arXiv:2304.05340 (2023).
  63. Zhongheng Zhang. 2016. Missing data imputation: focusing on single imputation. Annals of translational medicine 4, 1 (2016).
  64. Multimodal federated learning on iot data. In 2022 IEEE/ACM Seventh International Conference on Internet-of-Things Design and Implementation (IoTDI). IEEE, 43–54.
  65. AutoFed: Heterogeneity-Aware Federated Multimodal Learning for Robust Autonomous Driving. In Proceedings of the 29th Annual International Conference on Mobile Computing and Networking.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Adiba Orzikulova (4 papers)
  2. Jaehyun Kwak (2 papers)
  3. Jaemin Shin (10 papers)
  4. Sung-Ju Lee (22 papers)