Bodies at Rest: 3D Human Pose and Shape Estimation from a Pressure Image using Synthetic Data (2004.01166v1)

Abstract: People spend a substantial part of their lives at rest in bed. 3D human pose and shape estimation for this activity would have numerous beneficial applications, yet line-of-sight perception is complicated by occlusion from bedding. Pressure sensing mats are a promising alternative, but training data is challenging to collect at scale. We describe a physics-based method that simulates human bodies at rest in a bed with a pressure sensing mat, and present PressurePose, a synthetic dataset with 206K pressure images with 3D human poses and shapes. We also present PressureNet, a deep learning model that estimates human pose and shape given a pressure image and gender. PressureNet incorporates a pressure map reconstruction (PMR) network that models pressure image generation to promote consistency between estimated 3D body models and pressure image input. In our evaluations, PressureNet performed well with real data from participants in diverse poses, even though it had only been trained with synthetic data. When we ablated the PMR network, performance dropped substantially.

• The paper introduces a framework using synthetic data to train a deep learning model for estimating 3D human pose and shape from pressure images, addressing occlusion challenges.
• A key contribution is the large PressurePose dataset of synthetic pressure images, generated through physics-based simulations of human body models on a pressure mat.
• The PressureNet model, trained exclusively on this synthetic data, accurately predicts complex human poses from low-resolution pressure images and shows promise for applications like patient monitoring and sleep studies.

3D Human Pose and Shape Estimation from Pressure Images Using Synthetic Data

This paper introduces an innovative framework for estimating 3D human pose and shape using pressure images, offering a detailed examination of a unique methodology that leverages synthetic data for deep learning model training. The authors, Henry M. Clever et al., propose a system that stands out by circumventing the typical challenges faced by traditional line-of-sight perception techniques, especially in scenarios where body occlusion is prevalent, such as when individuals are resting in bed.

PressurePose Dataset and Synthetic Data Generation

The cornerstone of the approach is the PressurePose dataset, comprised of 206,000 synthetic pressure images paired with 3D human poses and shapes. The authors employ a physics-based simulation framework to generate this dataset. Human body models, both articulated and soft body, are subjected to physics simulations to produce realistic pressure images. This process involves simulation tools, DART and FleX, to model the interactions between human bodies and a simulated pressure mat.

The generation process includes sampling body poses and shapes from a high-dimensional parameter space, followed by a sequence of physics simulations aimed at attaining a resting pose, and culminating in the creation of synthetic pressure images. The simulations use SMPL models to generate realistic human body meshes, with the ultimate objective being a dataset rich in diversity and annotative detail.

PressureNet: Deep Learning Model

PressureNet, the deep learning model developed from this dataset, is a novel architecture designed to estimate 3D human body models, considering both pose and shape from low-resolution pressure images. Notably, it integrates a pressure map reconstruction (PMR) network to enhance the consistency between estimated 3D models and the input pressure images. This component, the PMR network, serves as an innovative mechanism for maintaining fidelity in the synthetic pressure images and improving the robustness of the model.

The model's training is exclusively on synthetic data, demonstrating significant adaptability and effectiveness when applied to real-world data despite this. Evaluation against real data from human participants showed that PressureNet can accurately predict complex human poses that other models find challenging, such as supine poses with distinct hand placements.

Evaluation and Implications

The performance of PressureNet was rigorously evaluated comparing synthetic and real datasets. Ablation studies confirmed the importance of components like the PMR network in maintaining estimation accuracy. The model's efficacy in diverse human poses underscores its potential applications in medical fields—such as patient monitoring, sleep studies, and assistive robotics—where non-intrusive, accurate human pose detection under occlusion conditions is valuable.

Future Directions

The paper opens several avenues for future research. Enhancements in the realism of synthetic data and more extensive validation on diverse real-world scenarios could further reduce the domain gap. The methods presented could be adapted for more complex environments or incorporated into multimodal sensing frameworks combining pressure data with other types of sensor input.

Further exploration into the biomechanics of human resting poses could yield refined dynamics that enhance the generation of synthetic datasets. These developments would not only improve healthcare applications but could also extend into areas like animation and interactive robotics where accurate human modeling is crucial.

In conclusion, the paper represents a significant step forward in using synthetic data for training deep learning models to estimate 3D human pose and shape in contexts typically hampered by occlusions. The methodologies and findings outlined hold promise for expanding the utility of non-visual sensor modalities in computer vision and human-computer interaction fields.