Combining Implicit Function Learning and Parametric Models for 3D Human Reconstruction (2007.11432v2)

Abstract: Implicit functions represented as deep learning approximations are powerful for reconstructing 3D surfaces. However, they can only produce static surfaces that are not controllable, which provides limited ability to modify the resulting model by editing its pose or shape parameters. Nevertheless, such features are essential in building flexible models for both computer graphics and computer vision. In this work, we present methodology that combines detail-rich implicit functions and parametric representations in order to reconstruct 3D models of people that remain controllable and accurate even in the presence of clothing. Given sparse 3D point clouds sampled on the surface of a dressed person, we use an Implicit Part Network (IP-Net)to jointly predict the outer 3D surface of the dressed person, the and inner body surface, and the semantic correspondences to a parametric body model. We subsequently use correspondences to fit the body model to our inner surface and then non-rigidly deform it (under a parametric body + displacement model) to the outer surface in order to capture garment, face and hair detail. In quantitative and qualitative experiments with both full body data and hand scans we show that the proposed methodology generalizes, and is effective even given incomplete point clouds collected from single-view depth images. Our models and code can be downloaded from http://virtualhumans.mpi-inf.mpg.de/ipnet.

• The paper’s main contribution is IP-Net, which integrates implicit function learning with parametric body models to accurately reconstruct detailed 3D human forms from sparse input data.
• The method predicts both an outer dressed surface and an inner body shape, mitigating inter-penetration issues and enhancing reconstruction realism.
• Evaluations show high outer surface accuracy and robust inner shape estimates, demonstrating the approach’s effectiveness even with limited training data.

Combining Implicit Function Learning and Parametric Models for 3D Human Reconstruction

The paper "Combining Implicit Function Learning and Parametric Models for 3D Human Reconstruction" presents an innovative approach to 3D human model reconstruction by integrating implicit function learning with parametric body models. This integration seeks to exploit the strengths of both methodologies to produce detailed, controllable 3D human models from sparse input data like point clouds.

Implicit Function Learning

Implicit function learning using deep learning approximations has emerged as a powerful method for reconstructing 3D surfaces. A typical limitation, however, is their static nature, which hampers post-reconstruction control over pose and shape. This paper addresses this challenge by combining implicit methods with parametric models that offer the necessary flexibility and control.

IP-Net Architecture

The core of the proposed methodology is IP-Net, an architecture that leverages implicit functions and parametric representation to predict multiple surfaces:

• Outward Surface: The general shape of the dressed human.
• Inner Surface: The underlying body shape without clothing.

IP-Net is tasked with generating these reconstructions from sparse point clouds collected using depth sensors. This method also predicts semantic correspondences for fitting with the Skinned Multi-Person Linear (SMPL) model.

Methodological Advancements

IP-Net introduces several novel contributions to the field:

• Double-Layered Surface Prediction: By predicting both outer and inner layers of the human model, IP-Net mitigates typical inter-penetration issues encountered by naive reconstructions.
• Implicit Feature Encoding: IP-Net computes an implicit feature grid from the input point cloud, which is used to make informed predictions at continuous point locations.
• Parameter Integration: The method allows for the SMPL+D model to be registered to the predicted surfaces for enhanced control over the model's pose and shape.

Quantitative and Qualitative Analysis

The paper extensively evaluates the performance of IP-Net using both existing and newly collected datasets:

• Outer Surface Accuracy: The IP-Net's outer surface predictions align closely with ground truth scans, yielding a vertex-to-surface error comparable to or better than current implicit method benchmarks.
• Body Shape Under Clothing: On datasets like BUFF, IP-Net's predicted inner surfaces provide robust estimates of body shape, even when trained without the specific dataset.

Practical Implications and Future Work

IP-Net holds significant implications for both computer graphics and computer vision, notably in fields that require highly detailed human models that are re-poseable and re-shapeable. Some of the practical applications could include virtual reality, animation, and gaming industries. Looking forward, expanding IP-Net to handle dynamic sequences or enhance detail prediction particularly around expressive facial features are potential areas for further research. Additionally, exploring the integration of temporal data could streamline body shape predictions across frames, further improving accuracy.

In summary, the paper successfully unifies two paradigms of 3D reconstruction, surpassing the limitations of traditional implicit models by introducing a controllable model through parametric fitting. The proposed method enhances the usability and realism of reconstructed models, thereby contributing significant value to ongoing work in 3D human modeling.