Gaussian Garments: Reconstructing Simulation-Ready Clothing with Photorealistic Appearance from Multi-View Video (2409.08189v1)

Abstract: We introduce Gaussian Garments, a novel approach for reconstructing realistic simulation-ready garment assets from multi-view videos. Our method represents garments with a combination of a 3D mesh and a Gaussian texture that encodes both the color and high-frequency surface details. This representation enables accurate registration of garment geometries to multi-view videos and helps disentangle albedo textures from lighting effects. Furthermore, we demonstrate how a pre-trained graph neural network (GNN) can be fine-tuned to replicate the real behavior of each garment. The reconstructed Gaussian Garments can be automatically combined into multi-garment outfits and animated with the fine-tuned GNN.

• The paper presents a novel pipeline combining multi-view video and 3D Gaussian splatting to capture detailed garment geometry.
• It employs advanced registration methods and neural networks to separate albedo from lighting and simulate realistic garment dynamics.
• Quantitative results show an F-score of 89.6%, with high fidelity in appearance metrics such as SSIM and PSNR, underscoring its robustness.

Gaussian Garments for Photorealistic Simulation-Ready Clothing Reconstruction

The paper "Gaussian Garments: Reconstructing Simulation-Ready Clothing with Photorealistic Appearance from Multi-View Video" presents a comprehensive new methodology for obtaining highly realistic, simulation-ready garment models. This is achieved by leveraging multi-view video inputs and a novel 3D Gaussian splatting technique. Gaussian splatting, coupled with conventional 3D mesh representations, allows for the accurate capture of garment geometry, fine surface details, and dynamic behavior in a manner suitable for numerous computer graphics applications.

Methodology

The core of the paper introduces a pipeline consisting of four stages:

1. Initial Mesh Reconstruction: This initial step involves capturing the garment’s 3D geometry from a template frame using a combination of multi-view stereo, surface reconstruction, and remeshing.
2. Geometry Registration: Here, the pre-obtained mesh is dynamically aligned with multi-view video sequences. The authors optimize a combination of photometric errors and physical regularities, including bending and strain energies. An innovative aspect of the registration process is substituting a body penetration term with a virtual edges regularization mechanism in early optimization phases to prevent divergence during highly dynamic sequences.
3. Appearance Modeling: This stage deals with disentangling albedo textures from lighting effects to achieve photorealism. A neural network is employed to predict per-frame texture adjustments, aiding in the preservation of high-frequency details and occlusion handling.
4. Behavior Fine-Tuning: A pre-trained graph neural network (GNN) is fine-tuned to replicate real-world garment behavior. This involves not only node positions and velocities but also material vectors and the resting state of garment edges to simulate dynamic draping accurately.

Numerical Results

The authors highlight several numerical results that underline the strength of their methodologies:

• Their registration process achieved an F-score of 89.6%, Chamfer Distance (CD) of 1.04 cm, and point-to-mesh distance of 0.504 cm when using the full proposed pipeline. This was compared against simpler ablations which produced higher error metrics.
• The appearance modeling, inclusive of lighting effects and geometry translations, resulted in LPIPS of 0.00812, SSIM of 0.992, and PSNR of 38.8 dB—metrics that demonstrate a significant improvement over baseline comparisons.

Implications and Future Directions

Practical Implications: The Gaussian Garments methodology has broad implications for industries reliant on high-quality garment visualization. The ability to produce photorealistic digital clothing assets efficiently from video data opens new avenues in virtual try-ons, movie CGI, and video game development. The flexibility to resize and combine garments into new multi-layer outfits further enhances its applicability.

Theoretical Contributions: From a theoretical perspective, the paper bridges the gap between neural implicit modeling and explicit geometric representations. The mesh and 3D Gaussian combination offers a robust framework for capturing the intricate details and realistic behavior of garments. The effective use of a GNN for simulation and the novel registration regularization techniques are significant contributions to computer graphics and vision fields.

Limitations and Future Work

While the proposed method is robust, there are several limitations noted by the authors. These include an assumption of uniform lighting conditions, challenges in capturing extremely high-frequency details like fur, fixed resolution constraints of Gaussian textures, and the simplification of garment details such as collars and pockets.

Future work should aim to:

1. Extend the method to accommodate dynamic relighting scenarios.
2. Incorporate multi-resolution strategies to tackle magnification and minification artifacts.
3. Enhance the geometric awareness of the method to represent complex garment features more accurately.
4. Explore the integration of differentiable physics simulators to broaden the capabilities of the proposed system further.

In conclusion, the Gaussian Garments paper presents a methodical and numerically validated approach to digital garment reconstruction. Combining multi-view video data, Gaussian splatting, and advanced neural network techniques, it sets a new standard in achieving photorealistic and simulation-ready digital clothing. The method's robust design and high versatility highlight its potential for broad application and further research in the domain of realistic garment modeling.