Unimodal and Multimodal Sensor Fusion for Wearable Activity Recognition
Abstract: Combining different sensing modalities with multiple positions helps form a unified perception and understanding of complex situations such as human behavior. Hence, human activity recognition (HAR) benefits from combining redundant and complementary information (Unimodal/Multimodal). Even so, it is not an easy task. It requires a multidisciplinary approach, including expertise in sensor technologies, signal processing, data fusion algorithms, and domain-specific knowledge. This Ph.D. work employs sensing modalities such as inertial, pressure (audio and atmospheric pressure), and textile capacitive sensing for HAR. The scenarios explored are gesture and hand position tracking, facial and head pattern recognition, and body posture and gesture recognition. The selected wearable devices and sensing modalities are fully integrated with machine learning-based algorithms, some of which are implemented in the embedded device, on the edge, and tested in real-time.
- H. T. Butt, M. Pancholi, M. Musahl, P. Murthy, M. A. Sanchez, and D. Stricker, “Inertial motion capture using adaptive sensor fusion and joint angle drift correction,” in 2019 22th International Conference on Information Fusion (FUSION). IEEE, 2019, pp. 1–8.
- K. Li, R. Zhang, B. Liang, F. Guimbretière, and C. Zhang, “Eario: A low-power acoustic sensing earable for continuously tracking detailed facial movements,” Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, vol. 6, no. 2, pp. 1–24, 2022.
- J. Shin, S. Lee, T. Gong, H. Yoon, H. Roh, A. Bianchi, and S.-J. Lee, “Mydj: Sensing food intakes with an attachable on your eyeglass frame,” in Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, 2022, pp. 1–17.
- H. Bello, B. Zhou, and P. Lukowicz, “Facial muscle activity recognition with reconfigurable differential stethoscope-microphones,” Sensors, vol. 20, no. 17, p. 4904, 2020.
- H. Bello, L. A. S. Marin, S. Suh, B. Zhou, and P. Lukowicz, “Inmyface: Inertial and mechanomyography-based sensor fusion for wearable facial activity recognition,” Information Fusion, p. 101886, 2023.
- H. Bello, S. Suh, B. Zhou, and P. Lukowicz, “Faceeat: Facial and eating activities recognition with inertial and mechanomyography fusion using a glasses-based design for real-time and on-the-edge inference,” in Adjunct Proceedings of the 2023 ACM International Joint Conference on Pervasive and Ubiquitous Computing & the 2023 ACM International Symposium on Wearable Computing, 2023, pp. 199–199.
- H. Bello, S. Suh, B. Zhou, and L. Paul, “Meciface: Mechanomyography and inertial fusion based glasses for edge real-time recognition of facial and eating activities,” arXiv preprint arXiv:2306.13674, 2023.
- H. Bello, J. Rodriguez, and P. Lukowicz, “Vertical hand position estimation with wearable differential barometery supported by rfid synchronization,” in EAI International Conference on Body Area Networks. Springer, 2019, pp. 24–33.
- H. Bello, B. Zhou, S. Suh, and P. Lukowicz, “Mocapaci: Posture and gesture detection in loose garments using textile cables as capacitive antennas,” in Proceedings of the 2021 ACM International Symposium on Wearable Computers, 2021, pp. 78–83.
- H. Bello, B. Zhou, S. Suh, L. A. Sanchez Marin, and P. Lukowicz, “Move with the theremin: Body posture and gesture recognition using the theremin in loose-garment with embedded textile cables as antennas,” Frontiers in Computer Science, vol. 4, p. 915280, 2022.
- H. Bello, S. Suh, D. Geißler, L. S. S. Ray, B. Zhou, and P. Lukowicz, “Captainglove: Capacitive and inertial fusion-based glove for real-time on edge hand gesture recognition for drone control,” in Adjunct Proceedings of the 2023 ACM International Joint Conference on Pervasive and Ubiquitous Computing & the 2023 ACM International Symposium on Wearable Computing, 2023, pp. 165–169.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
