Shape-Aware Human Pose and Shape Reconstruction Using Multi-View Images (1908.09464v1)

Abstract: We propose a scalable neural network framework to reconstruct the 3D mesh of a human body from multi-view images, in the subspace of the SMPL model. Use of multi-view images can significantly reduce the projection ambiguity of the problem, increasing the reconstruction accuracy of the 3D human body under clothing. Our experiments show that this method benefits from the synthetic dataset generated from our pipeline since it has good flexibility of variable control and can provide ground-truth for validation. Our method outperforms existing methods on real-world images, especially on shape estimations.

• The paper presents a shape-aware multi-view framework for 3D human pose and shape reconstruction, improving accuracy with direct shape supervision.
• The framework uses a large synthetic dataset with ground-truth shapes, enabling accurate shape recovery under garment occlusion in real images.
• The multi-view method improves pose estimation and achieves better shape recovery on real-world images compared to single-view baselines like HMR.

The paper addresses the problem of 3D human body reconstruction from multi-view images, leveraging the Skinned Multi-Person Linear (SMPL) model to represent pose and shape. The core idea is that using multiple views reduces projection ambiguity and improves reconstruction accuracy, especially for clothed humans.

Here's a breakdown:

• The paper introduces a learning-based, shape-aware framework for human body mesh reconstruction that uses SMPL parameters for pose and shape estimation and is directly supervised on shape parameters.
• A scalable, end-to-end, multi-view multi-stage learning framework accounts for the ambiguity of the 3D human body (geometry) reconstruction problem from 2D images, achieving improved estimation results.
• A large simulated dataset, including clothed human bodies and the corresponding ground-truth parameters, enhances the reconstruction accuracy, especially in shape estimation, where no ground-truth or supervision is provided in the real-world dataset.
• Accurate shape recovery under garment occlusion is achieved by providing corresponding supervision and deepening the model using the multi-view framework.

The paper uses a multi-view multi-stage network structure to capture visual features on garments, which are considered important for shape estimation. The model uses the SMPL model, representing human body shapes and poses with Principal Component Analysis (PCA) coefficients.

To train the model, the authors generate a synthetic dataset of multi-view human motion sequences with varying poses, shapes, and clothes using physically-based simulation. The use of this dataset makes the model "shape-aware," capturing the statistical correlation between garment features and human body shapes.

Model Architecture:

• The network iteratively refines pose and shape estimates through multiple stages, processing multi-view images one at a time.
• A shared-parameter prediction block computes corrections based on image features and previous estimates.
• Camera and human body parameters are estimated simultaneously, and predicted 3D joints are projected back to 2D for loss computation.
• Pose and shape parameters are shared among views, while each view maintains its camera calibration and global rotation.
• The loss function combines 2D joint loss, 3D joint loss, and SMPL parameter loss:

  $$L_i=\lambda_0 L_{2Djoint}+\lambda_1L_{3Djoint}+L_{SMPL}$$

  where:

  • $L_i$ is the loss at stage $i$
  • $L_{2Djoint}$ is the 2D joint loss
  • $L_{3Djoint}$ is the 3D joint loss
  • $L_{SMPL}$ is the SMPL parameter loss
  • $\lambda_0$ and $\lambda_1$ are weighting factors.
• The SMPL model generates a human body mesh based on pose and shape parameters:

  $$\mathbf{X}(\theta,\beta)=\mathbf{W}\mathbf{G}(\theta) (\mathbf{X}_0+\mathbf{S}\beta+\mathbf{P} \mathbf{R}(\theta))$$

  where:

  • $\mathbf{X}$ are the computed vertices
  • $\theta$ are the rotations of each joint plus the global rotation
  • $\beta$ are the PCA coefficients
  • $\mathbf{W}, \mathbf{S}$ and $\mathbf{P}$ are trained matrices
  • $\mathbf{G}(\theta)$ is the global transformation
  • $\mathbf{X_0}$ are the mean body vertices
  • $\mathbf{R}(\theta)$ is the relative rotation matrix.
• The 3D body is projected back to 2D using orthogonal projection:

  $$\mathbf{x}=s\mathbf{X}(\theta,\beta)\mathbf{R}^T+\mathbf{t}$$

  where:

  • $\mathbf{x}$ are the 2D projected vertices
  • $s$ is the scale
  • $\mathbf{R}$ is the orthogonal projection matrix
  • $\mathbf{t}$ is the translation.

The framework uses a recurrent structure, making it applicable to any number of views. A multi-view multi-stage framework couples multiple image inputs with shared parameters across regression blocks. The shared information is expressed as the predicted human body parameter. The corrective values, instead of the updated parameters, are predicted at each regression block to prevent gradient vanishing, inspired by residual networks.

Data Preparation:

• The authors generate a synthetic dataset with ground-truth human body shapes where garments are dressed rather than pasted on the skin.
• The CMU MoCap dataset is used as the pose subspace, and shape parameters are uniformly sampled.
• An optimization scheme avoids inter-penetration in the generated human mesh.
• Cloth registration and simulation are performed using ArcSim, with random sampling of cloth tightness.

Results:

The model was evaluated on the Human3.6M and MPI_INF_3DHP datasets and compared against the HMR method. The results showed that the multi-view approach achieves higher accuracy, especially in shape estimation. The authors use Mean Per Joint Position Error (MPJPE), Percentage of Correct Keypoints (PCK), and Area Under the Curve (AUC) as metrics for pose estimation. They use the Hausdorff distance between meshes to capture shape differences.

The paper finds that multi-view input improves pose estimation accuracy significantly. The synthetic training data improves shape estimation without overfitting. It is demonstrated that the model performs better on real-world images, capturing body shape more accurately than the baseline method.

A key advantage of the method is that multi-view input does not need to be taken with the exact same pose. The error correction structure of the model allows it to be applied as long as the poses of the views are not significantly different.