EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devices (2112.07642v3)

Abstract: Understanding social interactions from egocentric views is crucial for many applications, ranging from assistive robotics to AR/VR. Key to reasoning about interactions is to understand the body pose and motion of the interaction partner from the egocentric view. However, research in this area is severely hindered by the lack of datasets. Existing datasets are limited in terms of either size, capture/annotation modalities, ground-truth quality, or interaction diversity. We fill this gap by proposing EgoBody, a novel large-scale dataset for human pose, shape and motion estimation from egocentric views, during interactions in complex 3D scenes. We employ Microsoft HoloLens2 headsets to record rich egocentric data streams (including RGB, depth, eye gaze, head and hand tracking). To obtain accurate 3D ground truth, we calibrate the headset with a multi-Kinect rig and fit expressive SMPL-X body meshes to multi-view RGB-D frames, reconstructing 3D human shapes and poses relative to the scene, over time. We collect 125 sequences, spanning diverse interaction scenarios, and propose the first benchmark for 3D full-body pose and shape estimation of the social partner from egocentric views. We extensively evaluate state-of-the-art methods, highlight their limitations in the egocentric scenario, and address such limitations leveraging our high-quality annotations. Data and code are available at https://sanweiliti.github.io/egobody/egobody.html.

• The paper introduces EgoBody, a dataset capturing 3D body shape, pose, and dynamics from synchronized egocentric and third-person views using diverse capture systems.
• It employs a marker-less SMPL-X motion capture approach combined with rigorous calibration protocols to accurately reconstruct human interactions in indoor environments.
• Benchmark evaluations reveal significant reductions in mean per-joint errors when models fine-tuned with EgoBody, demonstrating its effectiveness for egocentric pose estimation.

Overview of "EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devices"

The paper "EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devices" by Siwei Zhang et al. introduces an innovative dataset designed to enhance the understanding and analysis of egocentric social interactions in complex 3D environments. This dataset, named EgoBody, bridges a critical gap in the field of human pose and motion estimation by providing comprehensive, high-quality data from egocentric perspectives, particularly using data captured from head-mounted devices (HMDs) like the Microsoft HoloLens2.

Dataset Composition and Collection Methodology

EgoBody stands out by capturing synchronized multi-modal data from both egocentric views and third-person perspectives. The dataset encompasses 125 sequences recorded from 36 subjects engaged in 2-person interaction scenarios across 15 indoor environments. Key components of the capture setup include the use of Microsoft HoloLens2 for capturing RGB, depth, head, hand, and eye gaze tracking from a first-person perspective, along with a multi-camera rig of Azure Kinect devices for third-person views. The authors employ an intricate calibration protocol to align the diverse data streams, addressing calibration inaccuracies and synchronization issues rigorously.

A significant contribution of EgoBody is its rich annotation, which includes the 3D ground-truth body shape, pose, and dynamics for all interacting individuals over time, as well as detailed 3D scene reconstructions. The dataset extensively leverages the SMPL-X body model to provide detailed mesh representations of human shapes adjusted for various poses and expressions, benefiting from a marker-less motion capture approach.

Benchmark Contributions and Analysis

For benchmarking, this work presents the first evaluation framework aimed at assessing the performance of contemporary 3D human pose and shape estimation (3DHPS) models within egocentric settings, as traditionally models have been predominantly tested in third-person scenarios. Evaluated methods include regression-based techniques like SPIN, EFT, and model-free approaches such as METRO, with explicit emphasis on the challenges imposed by egocentric data views, such as motion blur and body part truncation.

Notably, the dataset facilitates a critical performance increase when models are fine-tuned using EgoBody. The paper shows substantial improvements across several baseline models, demonstrating a pronounced reduction in mean per-joint position error (MPJPE) and vertex-to-vertex distance (V2V) when the models are adapted with their dataset, thus validating its practical value for enhancing existing 3DHPS methods' robustness and accuracy in egocentric conditions.

Implications and Future Directions

The introduction of EgoBody marks a notable advancement in datasets enabling the paper of human interaction from an egocentric perspective. This dataset will likely catalyze new research areas and encourage the development of more sophisticated models capable of accurately interpreting human pose and interaction in scenarios that closely mimic real-world conditions, such as virtual reality or assistive robotics scenarios.

The implications of this research are expansive, as understanding interactions from an egocentric view may enhance virtual and augmented reality applications, improve social assistive technology, and lead to more intuitive human-computer interaction systems. Future work inspired by this dataset could include extending the dataset with additional modalities, such as audio data, or exploring real-time applications that leverage the insights gained from this framework.

The paper adeptly highlights how inadequate training data in egocentric scenarios has limited existing methodologies' performance and proposes a rigorous, multi-faceted dataset to address these gaps. Going forward, further studies could build upon this foundation to improve the generalization of pose estimation models, exploring techniques that effectively handle egocentric data's distinct challenges while considering diverse social interaction nuances.