mm-Pose: Real-Time Human Skeletal Posture Estimation using mmWave Radars and CNNs (1911.09592v1)

Abstract: In this paper, mm-Pose, a novel approach to detect and track human skeletons in real-time using an mmWave radar, is proposed. To the best of the authors' knowledge, this is the first method to detect >15 distinct skeletal joints using mmWave radar reflection signals. The proposed method would find several applications in traffic monitoring systems, autonomous vehicles, patient monitoring systems and defense forces to detect and track human skeleton for effective and preventive decision making in real-time. The use of radar makes the system operationally robust to scene lighting and adverse weather conditions. The reflected radar point cloud in range, azimuth and elevation are first resolved and projected in Range-Azimuth and Range-Elevation planes. A novel low-size high-resolution radar-to-image representation is also presented, that overcomes the sparsity in traditional point cloud data and offers significant reduction in the subsequent machine learning architecture. The RGB channels were assigned with the normalized values of range, elevation/azimuth and the power level of the reflection signals for each of the points. A forked CNN architecture was used to predict the real-world position of the skeletal joints in 3-D space, using the radar-to-image representation. The proposed method was tested for a single human scenario for four primary motions, (i) Walking, (ii) Swinging left arm, (iii) Swinging right arm, and (iv) Swinging both arms to validate accurate predictions for motion in range, azimuth and elevation. The detailed methodology, implementation, challenges, and validation results are presented.

• The paper introduces mm-Pose, a framework that uses mmWave radar and CNNs for real-time 3D skeletal posture estimation.
• It transforms sparse radar data into high-resolution images via a forked CNN architecture, achieving average errors of 3.2 cm in depth and 2.7 cm in elevation.
• This approach enables reliable tracking under adverse conditions, opening new paths for applications in healthcare, autonomous vehicles, and privacy-preserving monitoring.

mm-Pose: Real-Time Human Skeletal Posture Estimation using mmWave Radars and CNNs

The paper in review presents mm-Pose, a novel framework designed for real-time estimation and tracking of human skeletal posture using millimeter-wave (mmWave) radar technology in conjunction with convolutional neural networks (CNNs). The primary contribution of this work is the utilization of mmWave radar for detecting and localizing more than 15 distinct skeletal joints, a challenging feat given the traditionally sparse nature of radar signal data.

Overview of the Method

The authors' approach capitalizes on the resilience of mmWave radar systems under lighting and weather adversities, making them suitable for applications such as traffic monitoring, autonomous vehicles, healthcare, and defense. The methodological core involves processing radar reflection signals to produce a point cloud, which is subsequently resolved into range, azimuth, and elevation metrics. This point cloud is then transformed into a novel radar-to-image representation to mitigate data sparsity and enhance feature quality for machine learning models.

The radar-to-image transformation assigns RGB values representing range, azimuth/elevation, and reflection power intensity to provide a compact, high-resolution input to the CNN architecture. This approach is notably efficient, as it reduces the input size from a potentially large 3D voxel grid to two 2D image projections, addressing both computational complexity and the sparsity of features.

A forked CNN architecture is employed to predict the position of skeletal joints in 3D space. This model consists of separate CNN branches for the radar projections from the XY and XZ planes, merging into a multi-layer perceptron (MLP) to refine the predictions of the joints’ spatial coordinates. Notably, radar reflections are used in lieu of the vision-based data often required for such estimations, presenting a significant advancement in the field.

Experimental Validation and Results

The paper details rigorous experiments carried out to validate the efficacy of mm-Pose. The researchers used the Texas Instruments AWR 1642 mmWave radars positioned perpendicularly to capture both the azimuth and elevation data of subjects performing various simple motions such as walking and arm swinging.

The evaluation metrics showcase promising accuracy, with an average localization error of 3.2 cm in depth and 2.7 cm in elevation, outperforming previous benchmarks like RF-Pose3D, albeit with slightly higher error in the azimuth dimension due to data variability in joint positions. The results validate the hypothesis that radar-based skeletal tracking is achievable with a level of precision comparable to image-based systems, without the associated limitations in visibility and privacy concerns.

Implications and Future Work

This research has significant practical implications, demonstrating that mmWave radar technology can be developed into robust systems for real-time human interaction monitoring in various environments, from indoor healthcare settings to outdoor urban traffic applications. By leveraging the unique properties of mmWave radar, it opens new avenues for further research into privacy-preserving, vision-independent human tracking solutions.

The authors acknowledge limitations such as data acquisition constraints and the inherent challenges in discerning fine skeletal details with existing radar technology. Future work could involve expanding motion datasets to enhance training robustness and exploring dual-radar or multi-sensor fusion systems to further refine joint localization accuracy.

In conclusion, the mm-Pose framework advances the discourse on non-visual human posture estimation methodologies, establishing a foundation for enhanced, flexible deployment of mmWave-based skeletal tracking solutions across diverse application domains.