Multi-Garment Net: Learning to Dress 3D People from Images (1908.06903v2)

Abstract: We present Multi-Garment Network (MGN), a method to predict body shape and clothing, layered on top of the SMPL model from a few frames (1-8) of a video. Several experiments demonstrate that this representation allows higher level of control when compared to single mesh or voxel representations of shape. Our model allows to predict garment geometry, relate it to the body shape, and transfer it to new body shapes and poses. To train MGN, we leverage a digital wardrobe containing 712 digital garments in correspondence, obtained with a novel method to register a set of clothing templates to a dataset of real 3D scans of people in different clothing and poses. Garments from the digital wardrobe, or predicted by MGN, can be used to dress any body shape in arbitrary poses. We will make publicly available the digital wardrobe, the MGN model, and code to dress SMPL with the garments.

• The paper introduces Multi-Garment Net, which learns separate models for 3D bodies and garments from images using distinct mesh representations.
• It employs a multi-step registration process with 2D and 3D supervision to accurately reconstruct human shape and garment details.
• The results demonstrate superior garment transfer and realistic texture rendering, promising applications in virtual dressing and digital fashion.

A Technical Overview of "Multi-Garment Net: Learning to Dress 3D People from Images"

The paper "Multi-Garment Net: Learning to Dress 3D People from Images" presents a methodological advancement in computer vision, specifically tailored for the reconstruction of human body shapes and clothing from image data. The authors propose the Multi-Garment Network (MGN), which leverages the SMPL model to infer 3D body and clothing geometry from a sequence of images. By addressing several shortcomings of previous works that used singular mesh representations, MGN provides enhanced control and realism in modeling human figures.

Core Contributions

MGN uniquely learns separate models for human bodies and layered garments, which are represented as distinct meshes derived from images. The ability to manipulate and transfer garments between different body shapes and poses highlights its versatility. The research extends beyond the basic reconstruction, allowing for practical applications such as virtual dressing, which can be particularly useful in VR, AR, and digital fashion.

To train MGN, the research exploits a comprehensive digital wardrobe embracing 712 digital garments. This is constructed using a novel garment registration approach that aligns templates of clothing onto 3D scans of clothed people. This extensive dataset enables the network to effectively capture the complex interplay between body shapes and garments.

Methodological Advancements

A salient feature of the MGN approach is its multi-step registration process, which separates clothing and body models from individual frames, thereby overcoming the blending issue of clothing details with body representations seen in prior techniques. The paper introduces a garment-specific template registration process. This method considers variations in shapes within each garment category through PCA-based models complemented by displacement fields to detail clothing geometry.

The approach utilizes a combination of 2D and 3D supervision in the training phase. This allows for leveraging both direct 3D scan data and indirect knowledge from image projections, refining the model's predictions to align closely with real-world appearances. Notably, differentiable rendering techniques enhance this alignments, maintaining garment silhouette coherence while confronting contextual variations in pose and shape.

Quantitative and Qualitative Results

Quantitatively, MGN achieves significant precision in garment reconstruction, with an error margin of approximately 5.78mm when eight frames are used. While this is comparable to results from the existing octopus model, the ability of MGN to profound separate clothing layers renders it superior in qualitative evaluations. The model excels in rendering lifelike textures and transferring garment designs across different model shapes, with lower chances of distortions or self-intersections.

Implications and Future Directions

Practically, MGN can transform industries reliant on 3D modeling, providing enhanced control in digital apparel markets and realistic human avatars for entertainment or training simulations. Theoretically, it complements the trajectory of deep learning models capable of nuanced image-to-3D translations, setting a precedent for layers in model representations.

Looking to the future, the paper encourages further exploration into enhancing texture realism and garment dynamics under real-world conditions. Additionally, expansion to a wider variety of garment types and optimizations to reduce computation could make the technique accessible for real-time applications.

Conclusion

In summary, this paper delineates a comprehensive approach for modeling human figures with layered garments from image data. MGN stands as a methodological leap, introducing detailed modeling and garment transfer capabilities that offer substantial utility across various applications. The accompanying dataset and model release promote continued research in enhancing 3D garment modeling fidelity and realism in digital environments.