Hierarchical Kinematic Probability Distributions for 3D Human Shape and Pose Estimation from Images in the Wild (2110.00990v2)

Abstract: This paper addresses the problem of 3D human body shape and pose estimation from an RGB image. This is often an ill-posed problem, since multiple plausible 3D bodies may match the visual evidence present in the input - particularly when the subject is occluded. Thus, it is desirable to estimate a distribution over 3D body shape and pose conditioned on the input image instead of a single 3D reconstruction. We train a deep neural network to estimate a hierarchical matrix-Fisher distribution over relative 3D joint rotation matrices (i.e. body pose), which exploits the human body's kinematic tree structure, as well as a Gaussian distribution over SMPL body shape parameters. To further ensure that the predicted shape and pose distributions match the visual evidence in the input image, we implement a differentiable rejection sampler to impose a reprojection loss between ground-truth 2D joint coordinates and samples from the predicted distributions, projected onto the image plane. We show that our method is competitive with the state-of-the-art in terms of 3D shape and pose metrics on the SSP-3D and 3DPW datasets, while also yielding a structured probability distribution over 3D body shape and pose, with which we can meaningfully quantify prediction uncertainty and sample multiple plausible 3D reconstructions to explain a given input image. Code is available at https://github.com/akashsengupta1997/HierarchicalProbabilistic3DHuman .

• The paper presents a novel probabilistic framework that predicts a distribution of 3D human poses using hierarchical matrix-Fisher and Gaussian models.
• It employs a differentiable rejection sampler and reprojection loss to align multiple plausible 3D configurations with 2D image evidence.
• The method achieves competitive accuracy on benchmark datasets by significantly reducing per-vertex and joint position errors.

Overview of Hierarchical Kinematic Probability Distributions for 3D Human Shape and Pose Estimation

The paper "Hierarchical Kinematic Probability Distributions for 3D Human Shape and Pose Estimation from Images in the Wild" presents a sophisticated approach to estimating 3D human body shape and pose from single RGB images. The authors propose a method to address the inherent uncertainty and ill-posed nature of the problem by predicting a distribution of possible 3D configurations rather than a single deterministic prediction. This approach leverages the human body's kinematic structure and advances beyond preceding methodologies that often produced overly deterministic predictions.

Methodological Insights

At the core of the proposed method lies the use of a hierarchical matrix-Fisher distribution for 3D joint rotations, paired with a Gaussian distribution for the SMPL body shape parameters. The matrix-Fisher distribution is well-suited for modeling the space of 3D rotations due to its basis in the special orthogonal group $SO(3)$, providing a theoretically sound framework for handling rotations. The adoption of a hierarchical structure allows the model to encapsulate dependencies among joint rotations that naturally arise from the human anatomy's kinematic tree.

The training framework is enhanced by incorporating a differentiable rejection sampler, facilitating the imposition of a reprojection loss. This design ensures that distribution samples are consistent with 2D observations in the input images. In terms of architecture, the network predicts hierarchically-organized probabiliy distributions for joint poses, supporting the production of multiple plausible 3D body configurations.

Training data comprises synthetic image samples, where the network learns to generalize from these to 'in-the-wild' conditions. The authors notably avoid reliance on accurately segmented silhouettes, opting instead for edge-based proxy representations that better simulate the shape information from synthetic images. This method is shown to improve robustness to domain shifts between synthetic and natural images.

Numerical Results and Claims

The model demonstrates competitive performance against state-of-the-art counterparts on 3DPW and SSP-3D datasets by achieving commendable metrics in both 3D shape and pose accuracy. Specifically, the result encompasses improvements in per-vertex Euclidean error (PVE-T-SC) and mean-per-joint-position-error (MPJPE-SC), reflecting the ability to capture a broad range of pose and shape variations.

The paper reports robust empirical outcomes, revealing that both the hierarchical model and the inclusion of the differentiated rejection sampler substantially enhance the alignment of predictions with visual evidence, particularly under conditions involving occlusion or depth ambiguity. This capability to handle uncertainty is quantitatively demonstrated through reductions in both pose estimation errors and per-vertex uncertainty measures.

Implications and Future Directions

The implications of this work are multifold, extending both practical and theoretical dimensions in the domain of 3D human estimation. Practically, the ability to predict multiple plausible configurations offers significant utility in applications like animation, virtual reality, and human-computer interaction, where reliability in uncertain conditions is paramount. Theoretically, the application of hierarchical probabilistic models opens avenues for further research into high-dimensional kinematic estimation problems and could inspire methodologies for other articulated systems beyond the human body.

Future developments may focus on further enriching the model's multi-modal predictive capabilities, including more diverse clothed human datasets that consider the shape effects of garments. Another promising direction lies in enhancing the global optimization of body parameters through integration with temporal data, which could refine pose continuation over video sequences.

In summary, this paper provides a substantial contribution to the field of computer vision by elegantly combining neural network distribution estimation with a nuanced understanding of human biomechanics.