Tex2Shape: Detailed Full Human Body Geometry From a Single Image (1904.08645v2)

Abstract: We present a simple yet effective method to infer detailed full human body shape from only a single photograph. Our model can infer full-body shape including face, hair, and clothing including wrinkles at interactive frame-rates. Results feature details even on parts that are occluded in the input image. Our main idea is to turn shape regression into an aligned image-to-image translation problem. The input to our method is a partial texture map of the visible region obtained from off-the-shelf methods. From a partial texture, we estimate detailed normal and vector displacement maps, which can be applied to a low-resolution smooth body model to add detail and clothing. Despite being trained purely with synthetic data, our model generalizes well to real-world photographs. Numerous results demonstrate the versatility and robustness of our method.

• The paper reframes detailed body reconstruction as an image-to-image translation problem, achieving rapid and accurate 3D shape recovery in about 50ms per frame.
• It converts partial texture maps into detailed normal and displacement maps using a U-Net architecture with a PatchGAN discriminator.
• The approach generalizes well to real photos and has significant implications for VR, AR, and digital avatar creation.

Tex2Shape: Detailed Full Human Body Geometry From a Single Image

The paper "Tex2Shape: Detailed Full Human Body Geometry From a Single Image" proposes a novel method for reconstructing detailed human body shape from a single photograph using a technique that is both efficient and effective at capturing intricate details. The primary mechanism introduced involves converting the complex task of shape regression into an image-to-image translation problem. This approach enables the capture of detailed features of the human body, such as facial nuances, hair, and clothing textures, with promising results even on body parts obscured in the original image.

The methodology involves generating partial texture maps of the visible body areas from images using available off-the-shelf methods. These textures are subsequently translated into detailed normal and displacement maps via the Tex2Shape network. The resultant maps enhance a basic body model by integrating fine details, including clothing and hair. The authors' approach leverages a pose-independent UV mapping of the SMPL body model, diffusing the complexity of mapping 2D image pixels to 3D mesh displacements. This alignment simplifies the training process and enhances the accuracy of detailed 3D reconstructions.

Significant insights include several key innovations in capturing high-resolution detail from minimal input data. First, the paper claims to be the first to frame detailed body shape recovery as an image-to-image translation problem, leading to simplicity and efficiency in results. The Tex2Shape network, utilizing a U-Net architecture with a PatchGAN discriminator, manages to estimate detailed body shapes quickly (approximately 50 milliseconds per frame) while maintaining high graphical fidelity.

The results, strengthened by training on a substantial synthetic dataset of 2043 3D scans augmented for realism via spherical harmonic lighting, demonstrate the robustness of the model in handling real-world images. Notably, despite the Tex2Shape model's reliance on synthetic training data, it shows commendable generalization to actual photographs.

Implications of this research are noteworthy for fields such as virtual reality (VR), augmented reality (AR), and digital avatar creation, offering a robust method for generating detailed and authentic digital representations of humans. As VR and AR applications continue to grow, this research could streamline the process of creating realistic, interactive digital presences, enhancing user identification and immersion.

Looking to the future, advancements in AI and machine learning could further enhance Tex2Shape, potentially incorporating semi-supervised learning techniques to improve results with limited labeled data. Additionally, expanding the model to encompass a wider variety of clothing types and hairstyles, as well as incorporating dynamic pose estimation, could extend its applicability and robustness.

In conclusion, this paper presents Tex2Shape, a practical solution for human body reconstruction from a single image, offering a blend of simplicity, speed, and detail that holds significant potential for various practical applications in digital media and interactive technology domains.