Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
129 tokens/sec
GPT-4o
28 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Continuous Multi-user Activity Tracking via Room-Scale mmWave Sensing (2309.11957v1)

Published 21 Sep 2023 in cs.HC

Abstract: Continuous detection of human activities and presence is essential for developing a pervasive interactive smart space. Existing literature lacks robust wireless sensing mechanisms capable of continuously monitoring multiple users' activities without prior knowledge of the environment. Developing such a mechanism requires simultaneous localization and tracking of multiple subjects. In addition, it requires identifying their activities at various scales, some being macro-scale activities like walking, squats, etc., while others are micro-scale activities like typing or sitting, etc. In this paper, we develop a holistic system called MARS using a single Commercial off the-shelf (COTS) Millimeter Wave (mmWave) radar, which employs an intelligent model to sense both macro and micro activities. In addition, it uses a dynamic spatial time sharing approach to sense different subjects simultaneously. A thorough evaluation of MARS shows that it can infer activities continuously with a weighted F1-Score of > 94% and an average response time of approx 2 sec, with 5 subjects and 19 different activities.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (95)
  1. 2011. Robodo SEN40 GY-273 HMC5883L Module Triple Axis Compass Magnetometer Sensor Module for Arduino. https://www.amazon.in/Robodo-Electronics-SEN40-HMC5883L-Magnetometer/dp/B0787LH8XR. [Accessed September 21, 2023].
  2. 2013. Tower Pro MG995 Servo Motor. https://www.towerpro.com.tw/product/mg995/. [Accessed September 21, 2023].
  3. 2018. DCA1000EVM. https://www.ti.com/tool/DCA1000EVM. [Accessed September 21, 2023].
  4. 2018. IWR1642BOOST. https://www.ti.com/tool/IWR1642BOOST. Accessed September 21, 2023.
  5. 2018. mmWave Demo Visualizer — dev.ti.com. https://dev.ti.com/gallery/view/mmwave/mmWave_Demo_Visualizer/. [Accessed September 21, 2023].
  6. 2019. Raspberry Pi 4 Model-B. https://www.raspberrypi.com/products/raspberry-pi-4-model-b/. [Accessed September 21, 2023].
  7. 2020. High Voltage Power Monitor — Monsoon Solutions . https://www.msoon.com/high-voltage-power-monitor.
  8. 2020. IWR1443BOOST. https://www.ti.com/tool/IWR1443BOOST. Accessed September 21, 2023.
  9. 2022. ADLs-IADLs. https://betterhealthwhileaging.net/what-are-adls-and-iadls/. [Accessed September 21, 2023].
  10. Fadel Adib and Dina Katabi. 2013. See through walls with WiFi!. In Proceedings of the ACM SIGCOMM 2013 conference on SIGCOMM. 75–86.
  11. Smart homes that monitor breathing and heart rate. In Proceedings of the 33rd annual ACM conference on human factors in computing systems. 837–846.
  12. Vid2Doppler: Synthesizing Doppler radar data from videos for training privacy-preserving activity recognition. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–10.
  13. Keystroke recognition using wifi signals. In Proceedings of the 21st annual international conference on mobile computing and networking. 90–102.
  14. Acoustic-based sensing and applications: A survey. Computer Networks 181 (2020), 107447.
  15. Using mmWave sensors to enhance drone safety and productivity. Texas Instruments white paper SPYY001 (2017).
  16. IMU2Doppler: Cross-Modal Domain Adaptation for Doppler-based Activity Recognition Using IMU Data. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 4 (2021), 1–20.
  17. Noncontact wideband sonar for human activity detection and classification. IEEE Sensors Journal 14, 11 (2014), 4043–4054.
  18. RFGo: a seamless self-checkout system for apparel stores using RFID. In Proceedings of the 26th Annual International Conference on Mobile Computing and Networking. 1–14.
  19. CounterIntelligence: Augmented reality kitchen. In Proc. CHI, Vol. 2239. Citeseer, 45.
  20. Activity recognition in smart homes using UWB radars. Procedia Computer Science 170 (2020), 10–17.
  21. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition. 961–970.
  22. Pingping Cai and Sanjib Sur. 2023. MilliPCD: Beyond Traditional Vision Indoor Point Cloud Generation via Handheld Millimeter-Wave Devices. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6, 4 (2023), 1–24.
  23. Human activity recognition from accelerometer data using a wearable device. In Iberian conference on pattern recognition and image analysis. Springer, 289–296.
  24. Rf-based human activity recognition using signal adapted convolutional neural network. IEEE Transactions on Mobile Computing 22, 1 (2021), 487–499.
  25. WiFi CSI based passive human activity recognition using attention based BLSTM. IEEE Transactions on Mobile Computing 18, 11 (2018), 2714–2724.
  26. MoVi-Fi: Motion-robust vital signs waveform recovery via deep interpreted RF sensing. In Proceedings of the 27th Annual International Conference on Mobile Computing and Networking. ACM, 392–405.
  27. Multiperspective thermal IR and video arrays for 3D body tracking and driver activity analysis. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05)-Workshops. IEEE, 3–3.
  28. Seamless interaction in space. In Proceedings of the 23rd Australian Computer-Human Interaction Conference. 88–97.
  29. Exposing the CSI: A Systematic Investigation of CSI-based Wi-Fi Sensing Capabilities and Limitations. In Proceedings of the 21st International Conference on Pervasive Computing and Communications (PerCom ’23). 81–90.
  30. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. Ieee, 248–255.
  31. RF-net: A unified meta-learning framework for RF-enabled one-shot human activity recognition. In Proceedings of the 18th Conference on Embedded Networked Sensor Systems. 517–530.
  32. A density-based algorithm for discovering clusters in large spatial databases with noise.. In kdd, Vol. 96. 226–231.
  33. In-home daily-life captioning using radio signals. In European Conference on Computer Vision. Springer, 105–123.
  34. Tagfree activity identification with rfids. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 1 (2018), 1–23.
  35. Wearable human activity recognition by electrocardiograph and accelerometer. In 2013 IEEE 43rd international symposium on multiple-valued logic. IEEE, 12–17.
  36. Mmpoint-GNN: graph neural network with dynamic edges for human activity recognition through a millimeter-wave radar. In 2021 International Joint Conference on Neural Networks (IJCNN). IEEE, 1–7.
  37. MmSense: Multi-person detection and identification via mmWave sensing. In Proceedings of the 3rd ACM Workshop on Millimeter-wave Networks and Sensing Systems. 45–50.
  38. WiFi-enabled smart human dynamics monitoring. In Proceedings of the 15th ACM Conference on Embedded Network Sensor Systems. 1–13.
  39. Comma. ai Marketing Plan. (2018).
  40. Ju Han and Bir Bhanu. 2005. Human activity recognition in thermal infrared imagery. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05)-Workshops. IEEE, 17–17.
  41. Michael Harville and Dalong Li. 2004. Fast, integrated person tracking and activity recognition with plan-view templates from a single stereo camera. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., Vol. 2. IEEE, II–II.
  42. HexiWear. [n. d.]. HexiWear. https://www.mikroe.com/hexiwear
  43. Au-id: Automatic user identification and authentication through the motions captured from sequential human activities using rfid. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 2 (2019), 1–26.
  44. Cesar Iovescu and Sandeep Rao. 2020. The fundamentals of millimeter wave radar sensors. Texas Instruments (2020).
  45. Employee acceptance of wearable technology in the workplace. Applied ergonomics 78 (2019), 148–156.
  46. Towards environment independent device free human activity recognition. In Proceedings of the 24th annual international conference on mobile computing and networking. ACM, 289–304.
  47. Bringing gesture recognition to all devices. In 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI 14). 303–316.
  48. Emily W Lam and Thomas DC Little. 2018. Refining light-based positioning for indoor smart spaces. In Proceedings of the 4th ACM MobiHoc Workshop on Experiences with the Design and Implementation of Smart Objects. 1–8.
  49. Isah A Lawal and Sophia Bano. 2019. Deep human activity recognition using wearable sensors. In Proceedings of the 12th ACM International Conference on PErvasive Technologies Related to Assistive Environments. 45–48.
  50. Experience: practical problems for acoustic sensing. In Proceedings of the 28th Annual International Conference on Mobile Computing And Networking. 381–390.
  51. LASense: Pushing the Limits of Fine-grained Activity Sensing Using Acoustic Signals. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6, 1 (2022), 1–27.
  52. VocalPrint: exploring a resilient and secure voice authentication via mmWave biometric interrogation. In Proceedings of the 18th Conference on Embedded Networked Sensor Systems. ACM, 312–325.
  53. Making the invisible visible: Action recognition through walls and occlusions. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 872–881.
  54. Beyond respiration: Contactless sleep sound-activity recognition using RF signals. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 3 (2019), 1–22.
  55. Wavoice: A noise-resistant multi-modal speech recognition system fusing mmwave and audio signals. In Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems. ACM, 97–110.
  56. See through smoke: robust indoor mapping with low-cost mmwave radar. In Proceedings of the 18th International Conference on Mobile Systems, Applications, and Services. 14–27.
  57. Human activity recognition in RGB-D videos by dynamic images. Multimedia Tools and Applications 79, 27 (2020), 19787–19801.
  58. Ramon Nitzberg. 1972. Constant-false-alarm-rate signal processors for several types of interference. IEEE Trans. Aerospace Electron. Systems 1 (1972), 27–34.
  59. Ultra-wideband radar-based activity recognition using deep learning. IEEE Access 9 (2021), 138132–138143.
  60. Pantomime: Mid-air gesture recognition with sparse millimeter-wave radar point clouds. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 1 (2021), 1–27.
  61. Jacopo Pegoraro and Michele Rossi. 2021. Real-time people tracking and identification from sparse mm-wave radar point-clouds. IEEE Access 9 (2021), 78504–78520.
  62. RIO: A Pervasive RFID-based Touch Gesture Interface. In Proceedings of the 23rd Annual International Conference on Mobile Computing and Networking (Snowbird, Utah, USA) (MobiCom ’17). 261–274.
  63. Sandeep Rao. 2017. Introduction to mmWave sensing: FMCW radars. Texas Instruments (TI) mmWave Training Series (2017), 1–11.
  64. S Rao. 2020. Introduction to mmwave radar sensing: Fmcw radars. Texas Instruments (2020), 1–70.
  65. Reuters. 2022. Tesla will remove more vehicle sensors amid Autopilot scrutiny. https://auto.economictimes.indiatimes.com/news/passenger-vehicle/cars/tesla-will-remove-more-vehicle-sensors-amid-autopilot-scrutiny/94654643 [Accessed September 21, 2023].
  66. SolAR: Energy positive human activity recognition using solar cells. In 2021 IEEE International Conference on Pervasive Computing and Communications (PerCom). IEEE, 1–10.
  67. Barriers to the adoption of wearable sensors in the workplace: A survey of occupational safety and health professionals. Human factors 60, 3 (2018), 351–362.
  68. Semantic segmentation on radar point clouds. In 2018 21st International Conference on Information Fusion (FUSION). IEEE, 2179–2186.
  69. mmAssist: Passive Monitoring of Driver’s Attentiveness Using mmWave Sensors. In 2023 15th International Conference on COMmunication Systems & NETworkS (COMSNETS). IEEE, 545–553.
  70. mmDrive: mmWave Sensing for Live Monitoring and On-Device Inference of Dangerous Driving. In 2023 IEEE International Conference on Pervasive Computing and Communications (PerCom). IEEE, 2–11.
  71. millieye: A lightweight mmwave radar and camera fusion system for robust object detection. In Proceedings of the International Conference on Internet-of-Things Design and Implementation. 145–157.
  72. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).
  73. Radhar: Human activity recognition from point clouds generated through a millimeter-wave radar. In Proceedings of the 3rd ACM Workshop on Millimeter-wave Networks and Sensing Systems. ACM, 51–56.
  74. Robust and practical WiFi human sensing using on-device learning with a domain adaptive model. In Proceedings of the 7th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation. 150–159.
  75. MultiTrack: Multi-user tracking and activity recognition using commodity WiFi. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–12.
  76. Millimeter Wave FMCW RADARs for Perception, Recognition and Localization in Automotive Applications: A Survey. IEEE Transactions on Intelligent Vehicles 7, 3 (2022), 533–555.
  77. mmEve: eavesdropping on smartphone’s earpiece via COTS mmWave device. In Proceedings of the 28th Annual International Conference on Mobile Computing And Networking. ACM, 338–351.
  78. We can hear you with Wi-Fi!. In Proceedings of the 20th annual international conference on Mobile computing and networking. 593–604.
  79. DF-Sense: Multi-user Acoustic Sensing for Heartbeat Monitoring with Dualforming. In Proceedings of the 21st Annual International Conference on Mobile Systems, Applications and Services. 1–13.
  80. C-FMCW based contactless respiration detection using acoustic signal. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 4 (2018), 1–20.
  81. Understanding and modeling of wifi signal based human activity recognition. In Proceedings of the 21st annual international conference on mobile computing and networking. 65–76.
  82. m-activity: Accurate and real-time human activity recognition via millimeter wave radar. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 8298–8302.
  83. Yanwen Wang and Yuanqing Zheng. 2018. Modeling RFID signal reflection for contact-free activity recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 4 (2018), 1–22.
  84. Teng Wei and Xinyu Zhang. 2015. mtrack: High-precision passive tracking using millimeter wave radios. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking. ACM, 117–129.
  85. Wikipedia. [n. d.]. Amazon Echo. https://en.wikipedia.org/wiki/Amazon_Echo
  86. mm3DFace: Nonintrusive 3D Facial Reconstruction Leveraging mmWave Signals. In Proceedings of the 21st Annual International Conference on Mobile Systems, Applications and Services. 462–474.
  87. Monitoring vital signs using millimeter wave. In Proceedings of the 17th ACM international symposium on mobile ad hoc networking and computing. ACM, 211–220.
  88. Noninvasive human activity recognition using millimeter-wave radar. IEEE Systems Journal (2022).
  89. mmWave radar-based hand gesture recognition using range-angle image. In 2020 IEEE 91st Vehicular Technology Conference (VTC2020-Spring). IEEE, 1–5.
  90. Pi-ViMo: Physiology-inspired Robust Vital Sign Monitoring using mmWave Radars. ACM Transactions on Internet of Things 4, 2 (2023), 1–27.
  91. Mobi2Sense: empowering wireless sensing with mobility. In Proceedings of the 28th Annual International Conference on Mobile Computing And Networking. 268–281.
  92. Vibrosight: Long-range vibrometry for smart environment sensing. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology. 225–236.
  93. Through-wall human pose estimation using radio signals. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7356–7365.
  94. mid: Tracking and identifying people with millimeter wave radar. In 2019 15th International Conference on Distributed Computing in Sensor Systems (DCOSS). IEEE, 33–40.
  95. SiWa: see into walls via deep UWB radar. In Proceedings of the 27th Annual International Conference on Mobile Computing and Networking. 323–336.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now