Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
166 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Non-contact Multimodal Indoor Human Monitoring Systems: A Survey (2312.07601v1)

Published 11 Dec 2023 in eess.SP and cs.LG

Abstract: Indoor human monitoring systems leverage a wide range of sensors, including cameras, radio devices, and inertial measurement units, to collect extensive data from users and the environment. These sensors contribute diverse data modalities, such as video feeds from cameras, received signal strength indicators and channel state information from WiFi devices, and three-axis acceleration data from inertial measurement units. In this context, we present a comprehensive survey of multimodal approaches for indoor human monitoring systems, with a specific focus on their relevance in elderly care. Our survey primarily highlights non-contact technologies, particularly cameras and radio devices, as key components in the development of indoor human monitoring systems. Throughout this article, we explore well-established techniques for extracting features from multimodal data sources. Our exploration extends to methodologies for fusing these features and harnessing multiple modalities to improve the accuracy and robustness of machine learning models. Furthermore, we conduct comparative analysis across different data modalities in diverse human monitoring tasks and undertake a comprehensive examination of existing multimodal datasets. This extensive survey not only highlights the significance of indoor human monitoring systems but also affirms their versatile applications. In particular, we emphasize their critical role in enhancing the quality of elderly care, offering valuable insights into the development of non-contact monitoring solutions applicable to the needs of aging populations.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (136)
  1. World Health Organization, “Progress report on the United Nations decade of healthy ageing, 2021-2023,” in UN Decade of Healthy Ageing, 2023.
  2. H. Gokalp and M. Clarke, “Monitoring activities of daily living of the elderly and the potential for its use in telecare and telehealth: a review,” TELEMEDICINE and e-HEALTH, 2013.
  3. J. K. Aggarwal and L. Xia, “Human activity recognition from 3d data: A review,” Pattern Recognition Letters, 2014.
  4. J. M. Fernandes, J. S. Silva, A. Rodrigues, and F. Boavida, “A survey of approaches to unobtrusive sensing of humans,” ACM Computing Surveys, 2022.
  5. J. Wang, Y. Chen, S. Hao, X. Peng, and L. Hu, “Deep learning for sensor-based activity recognition: A survey,” Pattern recognition letters, vol. 119, pp. 3–11, 2019.
  6. A. Haque, A. Milstein, and L. Fei-Fei, “Illuminating the dark spaces of healthcare with ambient intelligence,” Nature, 2020.
  7. X. Li, Q. Yu, B. Alzahrani, A. Barnawi, A. Alhindi, D. Alghazzawi, and Y. Miao, “Data fusion for intelligent crowd monitoring and management systems: A survey,” IEEE Access, 2021.
  8. A. Shokouhmand, S. Eckstrom, B. Gholami, and N. Tavassolian, “Camera-augmented non-contact vital sign monitoring in real time,” IEEE Sensors Journal, 2022.
  9. C. Qiu, D. Zhang, Y. Hu, H. Li, Q. Sun, and Y. Chen, “Radio-assisted human detection,” IEEE Transactions on Multimedia, 2022.
  10. M. Plöthner, K. Schmidt, L. de Jong, J. Zeidler, and K. Damm, “Needs and preferences of informal caregivers regarding outpatient care for the elderly: a systematic literature review.” BMC Geriatrics, 2019.
  11. V. Stanford, “Using pervasive computing to deliver elder care,” IEEE Pervasive computing, 2002.
  12. M. J. Page, J. E. McKenzie, P. M. Bossuyt, I. Boutron, T. C. Hoffmann, C. D. Mulrow, L. Shamseer, J. M. Tetzlaff, E. A. Akl, S. E. Brennan et al., “The prisma 2020 statement: an updated guideline for reporting systematic reviews,” International Journal of Surgery, 2021.
  13. N. Krahnstoever, J. Rittscher, P. Tu, K. Chean, and T. Tomlinson, “Activity recognition using visual tracking and rfid,” in 2005 Seventh IEEE Workshops on Applications of Computer Vision (WACV/MOTION’05)-Volume 1, vol. 1.   IEEE, 2005, pp. 494–500.
  14. Z. Sun, Q. Ke, H. Rahmani, M. Bennamoun, G. Wang, and J. Liu, “Human action recognition from various data modalities: A review,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
  15. J. Li, W. Gao, Y. Wu, Y. Liu, and Y. Shen, “High-quality indoor scene 3d reconstruction with rgb-d cameras: A brief review,” Computational Visual Media, vol. 8, pp. 1–25, 03 2022.
  16. S. Yousefi, H. Narui, S. Dayal, S. Ermon, and S. Valaee, “A survey on behavior recognition using wifi channel state information,” IEEE Communications Magazine, vol. 55, no. 10, pp. 98–104, 2017.
  17. M. Ciliberto, V. Fortes Rey, A. Calatroni, P. Lukowicz, and D. Roggen, “Opportunity++: A multimodal dataset for video-and wearable, object and ambient sensors-based human activity recognition,” Frontiers in Computer Science, 2021.
  18. P. P. Morita, K. S. Sahu, and A. Oetomo, “Health monitoring using smart home technologies: Scoping review,” JMIR mHealth and uHealth, 2023.
  19. M. Yuan, S. Wei, J. Zhao, and M. Sun, “A systematic survey on human behavior recognition methods,” SN Computer Science, vol. 3, no. 1, pp. 1–25, 2022.
  20. C. Wang and J. Yan, “A comprehensive survey of rgb-based and skeleton-based human action recognition,” IEEE Access, 2023.
  21. S. Wang and G. Zhou, “A review on radio based activity recognition,” Digital Communications and Networks, 2015.
  22. J. C. Soto, I. Galdino, E. Caballero, V. Ferreira, D. Muchaluat-Saade, and C. Albuquerque, “A survey on vital signs monitoring based on wi-fi csi data,” Computer Communications, 2022.
  23. X. Tang, Z. Zhang, and Y. Qin, “On-road object detection and tracking based on radar and vision fusion: A review,” IEEE Intelligent Transportation Systems Magazine, 2022.
  24. T. Baltrušaitis, C. Ahuja, and L.-P. Morency, “Multimodal machine learning: A survey and taxonomy,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018.
  25. W. C. Sleeman, R. Kapoor, and P. Ghosh, “Multimodal classification: Current landscape, taxonomy and future directions,” ACM Computing Surveys, 2022.
  26. L. Fan, T. Li, Y. Yuan, and D. Katabi, “In-home daily-life captioning using radio signals,” in European Conference on Computer Vision, 2020.
  27. M. Zhao, T. Li, M. Abu Alsheikh, Y. Tian, H. Zhao, A. Torralba, and D. Katabi, “Through-wall human pose estimation using radio signals,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 7356–7365.
  28. A. Zhao, J. Li, J. Dong, L. Qi, Q. Zhang, N. Li, X. Wang, and H. Zhou, “Multimodal gait recognition for neurodegenerative diseases,” IEEE transactions on cybernetics, 2021.
  29. W. Shao, Z. You, L. Liang, X. Hu, C. Li, W. Wang, and B. Hu, “A multi-modal gait analysis-based detection system of the risk of depression,” IEEE Journal of Biomedical and Health Informatics, 2022.
  30. H. Zou, J. Yang, H. Prasanna Das, H. Liu, Y. Zhou, and C. J. Spanos, “Wifi and vision multimodal learning for accurate and robust device-free human activity recognition,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, 2019, pp. 0–0.
  31. S. Ardianto and H.-M. Hang, “Multi-view and multi-modal action recognition with learned fusion,” in 2018 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC).   IEEE, 2018, pp. 1601–1604.
  32. A. M. De Boissiere and R. Noumeir, “Infrared and 3d skeleton feature fusion for rgb-d action recognition,” IEEE Access, vol. 8, pp. 168 297–168 308, 2020.
  33. R. Memmesheimer, N. Theisen, and D. Paulus, “Gimme signals: Discriminative signal encoding for multimodal activity recognition,” in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2020, pp. 10 394–10 401.
  34. H. Li, A. Shrestha, F. Fioranelli, J. Le Kernec, and H. Heidari, “Hierarchical classification on multimodal sensing for human activity recogintion and fall detection,” in 2018 IEEE SENSORS, 2018.
  35. N. Robertson and I. Reid, “A general method for human activity recognition in video,” Computer Vision and Image Understanding, vol. 104, no. 2-3, pp. 232–248, 2006.
  36. Y. Nie, A. Dai, X. Han, and M. Nießner, “Pose2room: Understanding 3d scenes from human activities,” arXiv preprint arXiv:2112.03030, 2021.
  37. J. Han and B. Bhanu, “Human activity recognition in thermal infrared imagery,” in 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05)-Workshops.   IEEE, 2005, pp. 17–17.
  38. T. Li, L. Fan, M. Zhao, Y. Liu, and D. Katabi, “Making the invisible visible: Action recognition through walls and occlusions,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 872–881.
  39. M. J. Bocus and R. Piechocki, “A comprehensive ultra-wideband dataset for non-cooperative contextual sensing,” Scientific Data, vol. 9, no. 1, pp. 1–13, 2022.
  40. M. J. Bocus, W. Li, S. Vishwakarma, R. Kou, C. Tang, K. Woodbridge, I. Craddock, R. McConville, R. Santos-Rodriguez, K. Chetty et al., “Operanet, a multimodal activity recognition dataset acquired from radio frequency and vision-based sensors,” Scientific data, 2022.
  41. L. Guo, L. Wang, J. Liu, W. Zhou, and B. Lu, “Huac: Human activity recognition using crowdsourced wifi signals and skeleton data,” Wireless Communications and Mobile Computing, vol. 2018, 2018.
  42. A. K. Koupai, M. J. Bocus, R. Santos-Rodriguez, R. J. Piechocki, and R. McConville, “Self-supervised multimodal fusion transformer for passive activity recognition,” arXiv preprint arXiv:2209.03765, 2022.
  43. Q. Zhang and Y. Li, “Indoor positioning method based on infrared vision and uwb fusion,” in Journal of Physics: Conference Series, 2021.
  44. N. L. Bragazzi, R. Khamisy-Farah, and M. Converti, “Ensuring equitable, inclusive and meaningful gender identity-and sexual orientation-related data collection in the healthcare sector: insights from a critical, pragmatic systematic review of the literature,” International Review of Psychiatry, 2022.
  45. S. He, Z. Han, C. Iglesias, V. Mehta, and M. Bolic, “A real-time respiration monitoring and classification system using a depth camera and radars,” Frontiers in Physiology, 2022.
  46. L. Ren, L. Kong, F. Foroughian, H. Wang, P. Theilmann, and A. E. Fathy, “Comparison study of noncontact vital signs detection using a doppler stepped-frequency continuous-wave radar and camera-based imaging photoplethysmography,” IEEE Transactions on Microwave Theory and Techniques, vol. 65, no. 9, pp. 3519–3529, 2017.
  47. X. Yang, Z. Zhang, X. Li, Y. Zheng, and Y. Shen, “Remote radar-camera vital sign monitoring system using a graph-based extraction algorithm,” in 2021 46th International Conference on Infrared, Millimeter and Terahertz Waves (IRMMW-THz).   IEEE, 2021, pp. 1–2.
  48. Z. Xie, B. Zhou, X. Cheng, E. Schoenfeld, and F. Ye, “Vitalhub: Robust, non-touch multi-user vital signs monitoring using depth camera-aided uwb,” in IEEE International Conference on Healthcare Informatics, 2021.
  49. C. Yang, B. Bruce, X. Liu, B. Gholami, and N. Tavassolian, “A hybrid radar-camera respiratory monitoring system based on an impulse-radio ultrawideband radar,” in Annual International Conference of the IEEE Engineering in Medicine & Biology Society, 2020.
  50. D.-M. Chian, C.-K. Wen, C.-J. Wang, M.-H. Hsu, and F.-K. Wang, “Vital signs identification system with doppler radars and thermal camera,” IEEE Transactions on Biomedical Circuits and Systems, vol. 16, no. 1, pp. 153–167, 2022.
  51. A. Vilesov, P. Chari, A. Armouti, A. B. Harish, K. Kulkarni, A. Deoghare, L. Jalilian, and A. Kadambi, “Blending camera and 77 ghz radar sensing for equitable, robust plethysmography,” ACM Transactions on Graphics, 2022.
  52. J. C. Soto, I. Galdino, E. Caballero, V. Ferreira, D. Muchaluat-Saade, and C. Albuquerque, “A survey on vital signs monitoring based on wi-fi csi data,” Computer Communications, vol. 195, pp. 99–110, 2022.
  53. V. Selvaraju, N. Spicher, J. Wang, N. Ganapathy, J. M. Warnecke, S. Leonhardt, R. Swaminathan, and T. M. Deserno, “Continuous monitoring of vital signs using cameras: a systematic review,” Sensors, vol. 22, no. 11, p. 4097, 2022.
  54. L. Zhang, C. Fu, C. Li, and H. Hong, “Rf and camera-based vital signs monitoring applications,” in Contactless Vital Signs Monitoring.   Elsevier, 2022, pp. 303–326.
  55. Y. Rong, P. C. Theofanopoulos, G. C. Trichopoulos, and D. W. Bliss, “A new principle of pulse detection based on terahertz wave plethysmography,” Scientific reports, vol. 12, no. 1, pp. 1–15, 2022.
  56. E. Cardillo, C. Li, and A. Caddemi, “Vital sign detection and radar self-motion cancellation through clutter identification,” IEEE Transactions on Microwave Theory and Techniques, vol. 69, no. 3, pp. 1932–1942, 2021.
  57. T. K. V. Dai, K. Oleksak, T. Kvelashvili, F. Foroughian, C. Bauder, P. Theilmann, A. E. Fathy, and O. Kilic, “Enhancement of remote vital sign monitoring detection accuracy using multiple-input multiple-output 77 ghz fmcw radar,” IEEE Journal of Electromagnetics, RF and Microwaves in Medicine and Biology, vol. 6, no. 1, pp. 111–122, 2022.
  58. K.-C. Peng, M.-C. Sung, F.-K. Wang, and T.-S. Horng, “Noncontact vital sign sensing under nonperiodic body movement using a novel frequency-locked-loop radar,” IEEE Transactions on Microwave Theory and Techniques, vol. 69, no. 11, pp. 4762–4773, 2021.
  59. Y. Wang, Y. Shui, X. Yang, Z. Li, and W. Wang, “Multi-target vital signs detection using frequency-modulated continuous wave radar,” EURASIP Journal on Advances in Signal Processing, vol. 2021, no. 1, pp. 1–19, 2021.
  60. C. Feng, X. Jiang, M.-G. Jeong, H. Hong, C.-H. Fu, X. Yang, E. Wang, X. Zhu, and X. Liu, “Multitarget vital signs measurement with chest motion imaging based on mimo radar,” IEEE Transactions on Microwave Theory and Techniques, vol. 69, no. 11, pp. 4735–4747, 2021.
  61. Y. Shi, L. Du, X. Chen, X. Liao, Z. Yu, Z. Li, C. Wang, and S. Xue, “Robust gait recognition based on deep cnns with camera and radar sensor fusion,” IEEE Internet of Things Journal, 2023.
  62. H. Li, P. Zhang, S. Al Moubayed, S. N. Patel, and A. P. Sample, “Id-match: A hybrid computer vision and rfid system for recognizing individuals in groups,” in CHI Conference on Human Factors in Computing Systems, 2016.
  63. H. Chen, S. Munir, and S. Lin, “Rfcam: Uncertainty-aware fusion of camera and wi-fi for real-time human identification with mobile devices,” Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2022.
  64. D. Cao, R. Liu, H. Li, S. Wang, W. Jiang, and C. X. Lu, “Cross vision-rf gait re-identification with low-cost rgb-d cameras and mmwave radars,” Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2022.
  65. H. Liu, A. Alali, M. Ibrahim, B. B. Cao, N. Meegan, H. Li, M. Gruteser, S. Jain, K. Dana, A. Ashok, B. Cheng, and H. Lu, “Vi-fi: Associating moving subjects across vision and wireless sensors,” in ACM/IEEE International Conference on Information Processing in Sensor Networks, 2022.
  66. L. Deng, J. Yang, S. Yuan, H. Zou, C. X. Lu, and L. Xie, “Gaitfi: Robust device-free human identification via wifi and vision multimodal learning,” IEEE Internet of Things Journal, 2022.
  67. S. Fang, T. Islam, S. Munir, and S. Nirjon, “Eyefi: Fast human identification through vision and wifi-based trajectory matching,” in International Conference on Distributed Computing in Sensor Systems, 2020.
  68. A. Luchetti, A. Carollo, L. Santoro, M. Nardello, D. Brunelli, and P. Bosetti, “Human identification and tracking using ultra-wideband-vision data fusion in unstructured environments,” Acta IMEKO e-Journal of the International Measurement Confederation, 2021.
  69. C. Wan, L. Wang, and V. V. Phoha, “A survey on gait recognition,” ACM Computing Surveys, 2018.
  70. A. Nambiar, A. Bernardino, and J. C. Nascimento, “Gait-based person re-identification: A survey,” ACM Computing Surveys, 2019.
  71. A. Sepas-Moghaddam and A. Etemad, “Deep gait recognition: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
  72. T. Suwannaphong, R. McConville, and I. Craddock, “Radio signal strength indication augmentation for one-shot learning in indoor localisation,” in Proceedings of the 1st ACM Workshop on Smart Wearable Systems and Applications, 2022, pp. 7–12.
  73. W. Jiang, F. Li, L. Mei, R. Liu, and S. Wang, “Visble: Vision-enhanced ble device tracking,” in IEEE International Conference on Sensing, Communication, and Networking, 2022.
  74. T. Ishihara, K. M. Kitani, C. Asakawa, and M. Hirose, “Deep radio-visual localization,” in IEEE Winter Conference on Applications of Computer Vision (WACV), 2018.
  75. K. Deng, D. Zhao, Q. Han, S. Wang, Z. Zhang, A. Zhou, and H. Ma, “Geryon: Edge assisted real-time and robust object detection on drones via mmwave radar and camera fusion,” Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2022.
  76. Y. Li, J. Deng, Y. Zhang, J. Ji, H. Li, and Y. Zhang, “EZFusion: A close look at the integration of lidar, millimeter-wave radar, and camera for accurate 3d object detection and tracking,” IEEE Robotics and Automation Letters, 2022.
  77. H. Li, R. Liu, S. Wang, W. Jiang, and C. X. Lu, “Pedestrian liveness detection based on mmwave radar and camera fusion,” in IEEE International Conference on Sensing, Communication, and Networking, 2022.
  78. S. Papaioannou, H. Wen, A. Markham, and N. Trigoni, “Fusion of radio and camera sensor data for accurate indoor positioning,” in IEEE International Conference on Mobile Ad Hoc and Sensor Systems, 2014.
  79. Y. Zhao, J. Xu, J. Wu, J. Hao, and H. Qian, “Enhancing camera-based multimodal indoor localization with device-free movement measurement using wifi,” IEEE Internet of Things Journal, 2020.
  80. J. Cai and H. Cai, “Robust hybrid approach of vision-based tracking and radio-based identification and localization for 3d tracking of multiple construction workers,” Journal of Computing in Civil Engineering, 2020.
  81. J. Xu, H. Chen, K. Qian, E. Dong, M. Sun, C. Wu, L. Zhang, and Z. Yang, “Ivr: Integrated vision and radio localization with zero human effort,” Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2019.
  82. T. H. Nguyen, T.-M. Nguyen, and L. Xie, “Range-focused fusion of camera-imu-uwb for accurate and drift-reduced localization,” IEEE Robotics and Automation Letters, 2021.
  83. F. Liu, J. Zhang, J. Wang, H. Han, and D. Yang, “An uwb/vision fusion scheme for determining pedestrians’ indoor location,” Sensors, 2020.
  84. L. Varotto, A. Cenedese, and A. Cavallaro, “Probabilistic radio-visual active sensing for search and tracking,” in European Control Conference, 2021.
  85. R. Streubel and B. Yang, “Fusion of stereo camera and mimo-fmcw radar for pedestrian tracking in indoor environments,” in 2016 19th International Conference on Information Fusion (Fusion).   IEEE, 2016, pp. 565–572.
  86. A. Pearce, J. A. Zhang, and R. Xu, “A combined mmwave tracking and classification framework using a camera for labeling and supervised learning,” Sensors, vol. 22, no. 22, p. 8859, 2022.
  87. T.-Y. Lim, S. A. Markowitz, and M. N. Do, “Radical: A synchronized fmcw radar, depth, imu and rgb camera data dataset with low-level fmcw radar signals,” IEEE Journal of Selected Topics in Signal Processing, vol. 15, no. 4, pp. 941–953, 2021.
  88. K. Cai, Q. Xia, P. Li, J. Stankovic, and C. X. Lu, “Robust human detection under visual degradation via thermal and mmwave radar fusion,” in International Conference on Embedded Wireless Systems and Networks, 2023.
  89. M. Tarkowski, K. Bizewski, M. Rzymowski, K. Nyka, and L. Kulas, “Wireless multimodal localization sensor for industrial applications,” in 2016 21st International Conference on Microwave, Radar and Wireless Communications (MIKON).   IEEE, 2016, pp. 1–4.
  90. P. Woznica, M. Tarkowski, M. Plotka, and L. Kulas, “Rf indoor positioning system supported by wireless computer vision sensors,” in 2014 20th International Conference on Microwaves, Radar and Wireless Communications (MIKON).   IEEE, 2014, pp. 1–3.
  91. C.-S. Wang and L.-C. Cheng, “Rfid & vision based indoor positioning and identification system,” in 2011 IEEE 3rd international conference on communication software and networks.   IEEE, 2011, pp. 506–510.
  92. M. Sturari, D. Liciotti, R. Pierdicca, E. Frontoni, A. Mancini, M. Contigiani, and P. Zingaretti, “Robust and affordable retail customer profiling by vision and radio beacon sensor fusion,” Pattern Recognition Letters, 2016.
  93. R. Xu, W. Dong, A. Sharma, and M. Kaess, “Learned depth estimation of 3d imaging radar for indoor mapping,” in 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2022, pp. 13 260–13 267.
  94. C. X. Lu, S. Rosa, P. Zhao, B. Wang, C. Chen, J. A. Stankovic, N. Trigoni, and A. Markham, “See through smoke: robust indoor mapping with low-cost mmwave radar,” in Proceedings of the 18th International Conference on Mobile Systems, Applications, and Services, 2020, pp. 14–27.
  95. N. Long, K. Wang, R. Cheng, K. Yang, and J. Bai, “Fusion of millimeter wave radar and rgb-depth sensors for assisted navigation of the visually impaired,” in Millimetre Wave and Terahertz Sensors and Technology XI, vol. 10800.   SPIE, 2018, pp. 21–28.
  96. H. Ding, Z. Chen, C. Zhao, F. Wang, G. Wang, W. Xi, and J. Zhao, “MI-Mesh: 3D human mesh construction by fusing image and millimeter wave,” Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2023.
  97. W. Guo, J. Wang, and S. Wang, “Deep multimodal representation learning: A survey,” IEEE Access, vol. 7, pp. 63 373–63 394, 2019.
  98. M. Sturari, D. Liciotti, R. Pierdicca, E. Frontoni, A. Mancini, M. Contigiani, and P. Zingaretti, “Robust and affordable retail customer profiling by vision and radio beacon sensor fusion,” Pattern Recognition Letters, vol. 81, pp. 30–40, 2016.
  99. P. Stotko, M. Weinmann, and R. Klein, “Albedo estimation for real-time 3d reconstruction using rgb-d and ir data,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 150, pp. 213–225, 2019.
  100. M. Muaaz, A. Chelli, A. A. Abdelgawwad, A. C. Mallofré, and M. Pätzold, “Wiwehar: Multimodal human activity recognition using wi-fi and wearable sensing modalities,” IEEE Access, 2020.
  101. Z. Qin, Y. Zhang, S. Meng, Z. Qin, and K.-K. R. Choo, “Imaging and fusing time series for wearable sensor-based human activity recognition,” Information Fusion, vol. 53, pp. 80–87, 2020.
  102. M. Dimitrievski, L. Jacobs, P. Veelaert, and W. Philips, “People tracking by cooperative fusion of radar and camera sensors,” in IEEE Intelligent Transportation Systems Conference, 2019.
  103. F. Cui, Y. Song, J. Wu, Z. Xie, C. Song, Z. Xu, and K. Ding, “Online multi-target tracking for pedestrian by fusion of millimeter wave radar and vision,” in IEEE Radar Conference, 2021.
  104. B. B. Cao, A. Alali, H. Liu, N. Meegan, M. Gruteser, K. Dana, A. Ashok, and S. Jain, “Vitag: Online wifi fine time measurements aided vision-motion identity association in multi-person environments,” in IEEE International Conference on Sensing, Communication, and Networking, 2022.
  105. R. J. Piechocki, X. Wang, and M. J. Bocus, “Multimodal sensor fusion in the latent representation space,” Scientific Reports, 2023.
  106. C. Xie, D. Zhang, Z. Wu, C. Yu, Y. Hu, Q. Sun, and Y. Chen, “Accurate human pose estimation using rf signals,” in 2022 IEEE 24th International Workshop on Multimedia Signal Processing (MMSP).   IEEE, 2022, pp. 1–6.
  107. A. Sengupta, F. Jin, R. Zhang, and S. Cao, “mm-pose: Real-time human skeletal posture estimation using mmwave radars and cnns,” IEEE Sensors Journal, vol. 20, no. 17, pp. 10 032–10 044, 2020.
  108. W. Jiang, H. Xue, C. Miao, S. Wang, S. Lin, C. Tian, S. Murali, H. Hu, Z. Sun, and L. Su, “Towards 3d human pose construction using wifi,” in Proceedings of the 26th Annual International Conference on Mobile Computing and Networking, 2020, pp. 1–14.
  109. F. Wang, S. Zhou, S. Panev, J. Han, and D. Huang, “Person-in-wifi: Fine-grained person perception using wifi,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 5452–5461.
  110. S. He, V. Mehta, and M. Bolic, “A joint localization assisted respiratory rate estimation using ir-uwb radars,” in Annual International Conference of the IEEE Engineering in Medicine & Biology Society, 2020.
  111. Y. Song, T. Jin, Y. Dai, Y. Song, and X. Zhou, “Through-wall human pose reconstruction via uwb mimo radar and 3d cnn,” Remote Sensing, vol. 13, no. 2, p. 241, 2021.
  112. C. Gu, G. Wang, Y. Li, T. Inoue, and C. Li, “A hybrid radar-camera sensing system with phase compensation for random body movement cancellation in doppler vital sign detection,” IEEE transactions on microwave theory and techniques, vol. 61, no. 12, pp. 4678–4688, 2013.
  113. G. Charan, T. Osman, A. Hredzak, N. Thawdar, and A. Alkhateeb, “Vision-position multi-modal beam prediction using real millimeter wave datasets,” in 2022 IEEE Wireless Communications and Networking Conference, 2022.
  114. D. Li, J. Xu, Z. Yang, Q. Zhang, Q. Ma, L. Zhang, and P. Chen, “Motion inspires notion: Self-supervised visual-lidar fusion for environment depth estimation,” in Annual International Conference on Mobile Systems, Applications and Services, 2022.
  115. F. Ofli, R. Chaudhry, G. Kurillo, R. Vidal, and R. Bajcsy, “Berkeley mhad: A comprehensive multimodal human action database,” in IEEE Workshop on Applications of Computer Vision, 2013.
  116. C. Chen, R. Jafari, and N. Kehtarnavaz, “Utd-mhad: A multimodal dataset for human action recognition utilizing a depth camera and a wearable inertial sensor,” in IEEE International Conference on Image Processing, 2015.
  117. B. Kwolek and M. Kepski, “Human fall detection on embedded platform using depth maps and wireless accelerometer,” Computer Methods and Programs in Biomedicine, 2014.
  118. X. Chao, Z. Hou, and Y. Mo, “Czu-mhad: A multimodal dataset for human action recognition utilizing a depth camera and 10 wearable inertial sensors,” IEEE Sensors Journal, vol. 22, no. 7, pp. 7034–7042, 2022.
  119. A. Alkhateeb, G. Charan, T. Osman, A. Hredzak, J. Morais, U. Demirhan, and N. Srinivas, “Deepsense 6g: A large-scale real-world multi-modal sensing and communication dataset,” IEEE Communications Magazine, 2023.
  120. A. Chen, X. Wang, S. Zhu, Y. Li, J. Chen, and Q. Ye, “Mmbody benchmark: 3d body reconstruction dataset and analysis for millimeter wave radar,” in ACM International Conference on Multimedia, 2022.
  121. L. K. Topham, W. Khan, D. Al-Jumeily, A. Waraich, and A. J. Hussain, “A diverse and multi-modal gait dataset of indoor and outdoor walks acquired using multiple cameras and sensors,” Scientific Data, 2023.
  122. A. Sengupta, A. Yoshizawa, and S. Cao, “Automatic radar-camera dataset generation for sensor-fusion applications,” IEEE Robotics and Automation Letters, 2022.
  123. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, 2017.
  124. P. Xu, X. Zhu, and D. A. Clifton, “Multimodal learning with transformers: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
  125. Y. Kim, S. Kim, J. W. Choi, and D. Kum, “CRAFT: Camera-radar 3D object detection with spatio-contextual fusion transformer,” in Proceedings of the AAAI Conference on Artificial Intelligence, 2023.
  126. Y. Lei, Z. Wang, F. Chen, G. Wang, P. Wang, and Y. Yang, “Recent advances in multi-modal 3d scene understanding: A comprehensive survey and evaluation,” arXiv preprint arXiv:2310.15676, 2023.
  127. H. Yun, J. Na, and G. Kim, “Dense 2d-3d indoor prediction with sound via aligned cross-modal distillation,” in IEEE/CVF International Conference on Computer Vision, 2023.
  128. M. Nourani, C. Roy, T. Rahman, E. D. Ragan, N. Ruozzi, and V. Gogate, “Don’t explain without verifying veracity: An evaluation of explainable ai with video activity recognition,” arXiv preprint arXiv:2005.02335, 2020.
  129. M. Z. Uddin and A. Soylu, “Human activity recognition using wearable sensors, discriminant analysis, and long short-term memory-based neural structured learning,” Scientific Reports, 2021.
  130. S. Schmidt, J. Stankowicz, J. Carmack, and S. Kuzdeba, “Riftnext: Explainable deep neural rf scene classification,” in Proceedings of the 3rd ACM Workshop on Wireless Security and Machine Learning, 2021, pp. 79–84.
  131. M. A. Lee, M. Tan, Y. Zhu, and J. Bohg, “Detect, reject, correct: Crossmodal compensation of corrupted sensors,” in IEEE International Conference on Robotics and Automation, 2021.
  132. H. Viswanathan and P. E. Mogensen, “Communications in the 6g era,” IEEE Access, 2020.
  133. X. Li, Y. Cui, J. A. Zhang, F. Liu, D. Zhang, and L. Hanzo, “Integrated human activity sensing and communications,” IEEE Communications Magazine, 2022.
  134. A. Adhikary, M. S. Munir, A. D. Raha, Y. Qiao, Z. Han, and C. S. Hong, “Integrated sensing, localization, and communication in holographic mimo-enabled wireless network: A deep learning approach,” IEEE Transactions on Network and Service Management, 2023.
  135. Z. Zhao, R. Liu, and J. Li, “Integrated sensing and communication based breath monitoring using 5g network,” in International Wireless Communications and Mobile Computing (IWCMC), 2023.
  136. Z. Zhou, X. Li, J. He, X. Bi, Y. Chen, G. Wang, and P. Zhu, “6g integrated sensing and communication - sensing assisted environmental reconstruction and communication,” in IEEE International Conference on Acoustics, Speech and Signal Processing, 2023.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now