Co-Evolution of Pose and Mesh for 3D Human Body Estimation from Video (2308.10305v1)

Abstract: Despite significant progress in single image-based 3D human mesh recovery, accurately and smoothly recovering 3D human motion from a video remains challenging. Existing video-based methods generally recover human mesh by estimating the complex pose and shape parameters from coupled image features, whose high complexity and low representation ability often result in inconsistent pose motion and limited shape patterns. To alleviate this issue, we introduce 3D pose as the intermediary and propose a Pose and Mesh Co-Evolution network (PMCE) that decouples this task into two parts: 1) video-based 3D human pose estimation and 2) mesh vertices regression from the estimated 3D pose and temporal image feature. Specifically, we propose a two-stream encoder that estimates mid-frame 3D pose and extracts a temporal image feature from the input image sequence. In addition, we design a co-evolution decoder that performs pose and mesh interactions with the image-guided Adaptive Layer Normalization (AdaLN) to make pose and mesh fit the human body shape. Extensive experiments demonstrate that the proposed PMCE outperforms previous state-of-the-art methods in terms of both per-frame accuracy and temporal consistency on three benchmark datasets: 3DPW, Human3.6M, and MPI-INF-3DHP. Our code is available at https://github.com/kasvii/PMCE.

• The paper presents the PMCE network that decouples 3D pose estimation and mesh regression, achieving a 12.1% reduction in MPJPE.
• The methodology employs a two-stream encoder with a Spatial-Temporal Transformer and GRU to robustly extract mid-frame features.
• Adaptive Layer Normalization in the co-evolution decoder refines pose and mesh jointly, ensuring improved temporal consistency in video.

Co-Evolution of Pose and Mesh for 3D Human Body Estimation from Video

The paper "Co-Evolution of Pose and Mesh for 3D Human Body Estimation from Video" presents an advanced methodology for accurately and consistently recovering 3D human motion from video inputs. The authors address the challenges inherent in this task, particularly focusing on the need to balance per-frame accuracy with temporal consistency in motion capture.

Traditional methods relying on single-image estimation or static feature extraction from video frames have struggled with high complexity and low representation capacity. Such methods typically exhibit limitations in smooth pose motion and feature a restricted range of shape patterns. The proposed model, the Pose and Mesh Co-Evolution Network (PMCE), innovatively decouples the task into two distinct yet interacting parts: video-based 3D human pose estimation and mesh vertices regression.

Key Components and Methodology

The PMCE network is characterized by a two-stream encoder and a co-evolution decoder architecture.

• Two-Stream Encoder:

1. 3D Pose Estimation Stream: This stream leverages a Spatial-Temporal Transformer to estimate the mid-frame 3D pose from a sequence of 2D poses detected across several video frames. Notably, pose normalization is performed relative to the full image instead of cropped bounding boxes, aiding in maintaining global spatial orientation and enhancing robustness.
2. Image Feature Aggregation Stream: This stream extracts static image features using a pre-trained ResNet-50 and passes them through a GRU network to obtain a comprehensive temporal image feature for the mid-frame.

• Co-Evolution Decoder:

The decoder facilitates interaction between pose and mesh, using Adaptive Layer Normalization (AdaLN) to dynamically adjust joint and vertex features based on image features. This mechanism ensures that mesh and pose data are co-evolved to fit the specific body shape accurately. The network achieves this through symmetric attention mechanisms allowing for mesh and pose refinement in both cross-attention and self-attention layers.

Results and Evaluation

The PMCE demonstrates substantial improvement over previous state-of-the-art methods, achieving better accuracy and consistency in motion estimation across challenging datasets such as 3DPW, Human3.6M, and MPI-INF-3DHP. Key performance metrics indicate a reduction in MPJPE by 12.1% compared to other leading approaches, coupled with improvements in PVE and acceleration error, highlighting its robust handling of temporal consistency.

Implications and Future Directions

The integration of 3D pose estimation and mesh vertex regression through co-evolving processes presents a compelling advancement in 3D human body reconstruction. PMCE’s approach of leveraging detailed pose sequences alongside temporal image features to refine mesh estimations sets a high benchmark in the field of human motion analysis.

Practically, this work offers significant promise for applications in motion capture in complex environments where occlusion and high motion variability are prevalent. Theoretically, the adaptive normalization mechanism and the dual-stream processing contribute valuable strategies for future research, possibly extending to areas beyond human pose estimation, such as robotic perception and virtual reality simulations.

Looking forward, enhancements may include further refinement in dealing with self-occlusions and complex pose articulations, as well as expanding the adaptability of the model across different domains and camera perspectives. Expanding the training datasets to include a broader range of body shapes and movements could also enhance model generalization, thus further reinforcing its applicability in diverse real-world scenarios.

Through meticulous engineering and integration of different processing stages, the paper provides a comprehensive model that advances the precision and fluidity of 3D human body motion estimation in video analysis.