MM-Fi: Multi-Modal Non-Intrusive 4D Human Dataset for Versatile Wireless Sensing (2305.10345v2)

Abstract: 4D human perception plays an essential role in a myriad of applications, such as home automation and metaverse avatar simulation. However, existing solutions which mainly rely on cameras and wearable devices are either privacy intrusive or inconvenient to use. To address these issues, wireless sensing has emerged as a promising alternative, leveraging LiDAR, mmWave radar, and WiFi signals for device-free human sensing. In this paper, we propose MM-Fi, the first multi-modal non-intrusive 4D human dataset with 27 daily or rehabilitation action categories, to bridge the gap between wireless sensing and high-level human perception tasks. MM-Fi consists of over 320k synchronized frames of five modalities from 40 human subjects. Various annotations are provided to support potential sensing tasks, e.g., human pose estimation and action recognition. Extensive experiments have been conducted to compare the sensing capacity of each or several modalities in terms of multiple tasks. We envision that MM-Fi can contribute to wireless sensing research with respect to action recognition, human pose estimation, multi-modal learning, cross-modal supervision, and interdisciplinary healthcare research.

• The paper introduces a comprehensive non-intrusive MM-Fi dataset that fuses five sensing modalities for accurate 4D human pose estimation and activity recognition.
• The dataset comprises over 320,000 synchronized frames capturing 27 diverse actions from 40 subjects, addressing privacy and practical constraints in conventional methods.
• Experiments reveal that multi-sensor fusion, especially combining RGB, LiDAR, and mmWave radar data, significantly enhances pose estimation accuracy and overall robustness.

A Comprehensive Examination of MM-Fi: Multi-Modal Non-Intrusive 4D Human Dataset for Versatile Wireless Sensing

The research paper titled "MM-Fi: Multi-Modal Non-Intrusive 4D Human Dataset for Versatile Wireless Sensing" introduces a pioneering dataset, MM-Fi, that addresses limitations in current human sensing methods, offering a robust resource for the development of non-intrusive wireless sensing technologies. The authors argue that existing methods relying on cameras and wearable devices face notable challenges related to privacy concerns and practicality, particularly in realistic applications, while alternative solutions using non-intrusive sensors such as LiDAR, mmWave radar, and Wi-Fi remain underexplored, especially for comprehensive human pose estimation and activity recognition. The MM-Fi dataset aims to bridge this significant gap by providing a multi-modal, detailed, and synchronized dataset featuring five different sensing modalities: RGB images, depth images, LiDAR point clouds, mmWave radar point clouds, and Wi-Fi Channel State Information (CSI).

Dataset Composition and Methodology

MM-Fi comprises over 320,000 synchronized frames from 40 diverse human subjects, each performing 27 categorized actions—14 daily activities and 13 rehabilitation exercises. This diversity of actions underscores the dataset's potential utility across ubiquitous computing and healthcare sectors. Several key annotations are included within the dataset, such as 2D and 3D pose landmarks, 3D dense pose estimation, and comprehensive action categories, providing a wealth of data for multi-modal fusion and cross-modal supervision research tasks.

The dataset is collected using an innovative synchronized sensor platform, which integrates various sensors and synchronizes data capture using the Robot Operating System (ROS). This platform allows for the reconstruction of detailed and precise human poses by exploiting the complementary strengths of each modality and is engineered to circumvent environmental variables such as lighting conditions and user compliance issues associated with camera-based systems.

Experimental Setup and Results

Extensive experiments were conducted to evaluate different modalities' efficacy, either singly or in combination, for various tasks, including 3D human pose estimation and action recognition. One critical finding highlighted by the authors is the superiority of multi-sensor fusion strategies over single modality applications; for instance, fusion involving RGB, LiDAR, and mmWave radar achieves marked improvements in pose estimation metrics such as Mean Per Joint Position Error (MPJPE) and Procrustes Analysis MPJPE (PA-MPJPE). These results emphasize MM-Fi's potential in enhancing the accuracy and robustness of human sensing applications when leveraging multi-modal datasets.

Implications and Future Directions

MM-Fi's multi-modal data offers expansive avenues for future research. The comprehensive nature and the non-intrusive design hold promise for developing intelligent environments that preserve privacy and operate seamlessly in complex settings. Additionally, the dataset can catalyze advancements in domain adaptation and generalization techniques, vital for building robust sensing systems that maintain accuracy despite subject and environmental variations.

The authors identify several limitations within the current dataset version, including manual annotation processes and controlled environmental data collection, now actively addressed in their subsequent dataset versions. Future iterations are suggested to expand into multi-orientation scenarios with richer environments, promising greater applicability. This ongoing work reflects a strong commitment to evolving the dataset's scope and accessibility, encouraging broader adoption within the research community and fostering developments in AI-driven human interaction technologies.

In conclusion, the MM-Fi dataset stands as a critical contribution to the field of wireless human sensing, laying the groundwork for extensive research and potential real-world deployment of advanced sensing systems. The dataset's structured framework and detailed annotations are poised to accelerate innovation in both academic research and practical applications within smart environments and healthcare monitoring sectors.