GarmentDreamer: 3DGS Guided Garment Synthesis with Diverse Geometry and Texture Details (2405.12420v1)

Abstract: Traditional 3D garment creation is labor-intensive, involving sketching, modeling, UV mapping, and texturing, which are time-consuming and costly. Recent advances in diffusion-based generative models have enabled new possibilities for 3D garment generation from text prompts, images, and videos. However, existing methods either suffer from inconsistencies among multi-view images or require additional processes to separate cloth from the underlying human model. In this paper, we propose GarmentDreamer, a novel method that leverages 3D Gaussian Splatting (GS) as guidance to generate wearable, simulation-ready 3D garment meshes from text prompts. In contrast to using multi-view images directly predicted by generative models as guidance, our 3DGS guidance ensures consistent optimization in both garment deformation and texture synthesis. Our method introduces a novel garment augmentation module, guided by normal and RGBA information, and employs implicit Neural Texture Fields (NeTF) combined with Score Distillation Sampling (SDS) to generate diverse geometric and texture details. We validate the effectiveness of our approach through comprehensive qualitative and quantitative experiments, showcasing the superior performance of GarmentDreamer over state-of-the-art alternatives. Our project page is available at: https://xuan-li.github.io/GarmentDreamerDemo/.

A Deep Dive into GarmentDreamer: 3D Garment Synthesis with Text Prompts

Introduction

Creating realistic 3D digital garments is an essential aspect of fashion design, virtual reality, and gaming, but traditional methods are time-consuming and labor-intensive. Enter GarmentDreamer, a novel framework designed to generate high-quality 3D garment meshes from simple text prompts. This new methodology leverages advanced 3D Gaussian Splatting (3DGS) techniques to produce garments with intricate geometry and rich textures, making them ready for simulation and animation.

Key Concepts

Traditional vs. Generative Approaches

Traditional 3D garment creation includes several meticulous steps:

• Sketching and Modeling: Initial design via sketching.
• UV Mapping: Unwrapping the 3D model to a 2D image for texturing.
• Texturing: Adding color and material details.

These steps collectively demand significant manual effort and expertise, often resulting in high production costs.

GarmentDreamer shifts from these practices by using generative models based on 3DGS and diffusion techniques. Two common generative approaches are:

1. 2D Pattern Reconstruction: Translating 2D sewing patterns into 3D garments.
2. Direct Shape Generation: Employing generative models to predict the 3D shape and texture directly from text or images.

While the former relies heavily on extensive paired training data, the latter can suffer from inconsistencies and lack of detail. GarmentDreamer overcomes these drawbacks to offer a streamlined, robust solution.

Methodology

Garment Dreamer's Workflow

GarmentDreamer employs a multi-step process guided by 3D Gaussian Splatting models and diffusion-based principles:

1. Generating the Template Mesh:
   • Latent Diffusion Model: Garment geometry is encoded into a compact vector form, which is decoded later to initiate the mesh.
   • Simulation-based Preprocessing: Ensures the generation of a smooth, noise-free mesh by removing high-frequency details using physical simulation techniques.
2. 3D Gaussian Splatting:
   • 3D Gaussian kernels act as a scaffolding to imbue the garment templates with intricate geometric details using Score Distillation Sampling (SDS).
   • These Gaussian kernels are refined through RGB and mask renderings from multiple views, ensuring high fidelity and detailed final outputs.
3. Geometry Refinement:
   • Coarse Stage: Ensures basic shape and important openings (e.g., for arms, head).
   • Fine Stage: Adds nuanced geometric details using neural deferred shading and normal map guidance.
4. Texture Synthesis:
   • Utilizes an implicit Neural Texture Field (NeTF) to extract and optimize textures, ensuring high-quality outputs that can be directly applied to a simulation.

Numerical and Qualitative Strengths

• FashionCLIP Similarity Score (FCSS): GarmentDreamer achieves higher FCSS scores compared to existing methods, demonstrating its superior ability to align generated garments with text prompts.
• Computational Efficiency: Faster generation times (around 15 minutes per model) compared to traditional deformation-based approaches and competing generative models.
• Complexity and Quality: Produces garments with rich textures and detailed geometry, making them suitable for immediate use in fashion design, animation, and virtual reality.

Implications and Future Work

GarmentDreamer's ability to generate high-quality, simulation-ready garments from text prompts holds significant implications:

• Design Efficiency: Speeds up the prototyping phase in fashion design, allowing designers to iterate quickly on new concepts.
• Accessibility: Lowers the barrier for 3D garment creation, enabling smaller studios and independent designers to produce professional-level apparel.
• Virtual Try-on and Gaming: Enhances the realism and engagement in virtual environments.

Future Directions:

1. Efficiency Improvements: Reducing generation time further to make it viable for large-scale fashion applications.
2. Dynamic Simulation Integration: Incorporating physics-based simulations could enhance realism, especially for dynamic and interactive applications.
3. Further Detail in Textures: Ensuring textures maintain high fidelity without lighting artifacts, potentially by learning Physically-Based Rendering (PBR) materials.
4. Compatibility with Traditional Methods: Introducing parameterized 2D sewing patterns could bridge the gap between traditional garment manufacturing and digital generation techniques.

Conclusion

GarmentDreamer represents a significant advancement in the field of generative garment design, blending the speed and efficiency of modern AI techniques with the detail and quality necessary for professional applications. By optimizing and refining 3D garment generation, GarmentDreamer stands poised to transform industries relying on digital apparel, from virtual reality to fashion and beyond.