LayGA: Layered Gaussian Avatars for Animatable Clothing Transfer (2405.07319v1)

Abstract: Animatable clothing transfer, aiming at dressing and animating garments across characters, is a challenging problem. Most human avatar works entangle the representations of the human body and clothing together, which leads to difficulties for virtual try-on across identities. What's worse, the entangled representations usually fail to exactly track the sliding motion of garments. To overcome these limitations, we present Layered Gaussian Avatars (LayGA), a new representation that formulates body and clothing as two separate layers for photorealistic animatable clothing transfer from multi-view videos. Our representation is built upon the Gaussian map-based avatar for its excellent representation power of garment details. However, the Gaussian map produces unstructured 3D Gaussians distributed around the actual surface. The absence of a smooth explicit surface raises challenges in accurate garment tracking and collision handling between body and garments. Therefore, we propose two-stage training involving single-layer reconstruction and multi-layer fitting. In the single-layer reconstruction stage, we propose a series of geometric constraints to reconstruct smooth surfaces and simultaneously obtain the segmentation between body and clothing. Next, in the multi-layer fitting stage, we train two separate models to represent body and clothing and utilize the reconstructed clothing geometries as 3D supervision for more accurate garment tracking. Furthermore, we propose geometry and rendering layers for both high-quality geometric reconstruction and high-fidelity rendering. Overall, the proposed LayGA realizes photorealistic animations and virtual try-on, and outperforms other baseline methods. Our project page is https://jsnln.github.io/layga/index.html.

• The paper presents a novel two-stage process using layered 3D Gaussian splatting to distinguish body and clothing for animatable avatars.
• The approach improves photorealism and dynamic garment simulation by decoupling geometry computation from rendering.
• Results demonstrate superior clothing transfer across diverse body shapes and garment types, with strong implications for digital fashion.

Exploring Layered Gaussian Avatars for Animatable Clothing Transfer

Introduction to Layered Gaussian Avatars (LayGA)

The challenge of animatable clothing transfer in creating virtual avatars involves seamlessly dressing and animating digital garments across different virtual identities. Traditional methods often struggle with problems like the entanglement of body and clothing representations, leading to difficulties in accurate garment movement simulation when transferring clothing between models. The paper introduces Layered Gaussian Avatars (LayGA), a novel approach aimed at separating the body and clothing layers to enhance the quality and functionality of virtual try-on systems.

Key Innovations and Technique

LayGA achieves its breakthrough by treating the body and clothing as separate entities, allowing detailed and realistic clothing animation across different avatars. Here's how the approach breaks down:

Gaussian Splatting for Detail

Instead of traditional methods that use a unified model for body and clothing, LayGA utilizes 3D Gaussian Splatting, a technique that includes using Gaussian distributions to represent both body and clothing details distinctly. This method not only enables efficient rendering but also accommodates high-resolution details which are crucial for realistic animations.

Two-Stage Training Process

• Single-Layer Reconstruction: The first stage focuses on reconstructing a unified avatar using geometric constraints which help in distinguishing between body and clothing at the foundational level.
• Multi-Layer Fitting: Here, separate models are trained for both clothing and body using the pre-defined layered information. This stage ensures accurate tracking of clothing dynamics relative to the body's movement.

Geometry and Rendering Enhancements

To overcome the challenge of maintaining high-quality visual output while adhering to the geometric constraints, the researchers propose separating the geometry calculation from the rendering process. This separation allows the system to maintain surface smoothness for collision handling (important for different body shapes interaction) without sacrificing the rendering quality which is crucial for the visual realism of the avatars.

Practical Implications and Results

The implementation of LayGA showed excellent performance in terms of both the fidelity of the clothing animation and the quality of the virtual try-on applications:

• Photorealism: The approach outperformed baseline methods in rendering realistic animations.
• Flexibility in Clothing Transfer: The system effectively handles clothing transfer across multiple identities, managing different body sizes and garment types.

Future Prospects in Digital Fashion and Avatars

Looking forward, the implications of this research are vast. In the sphere of digital fashion, LayGA opens up possibilities for more interactive and customizable virtual try-ons, potentially transforming online retail experiences. Furthermore, as virtual reality and digital avatars become commonplace, the ability to dynamically change one's digital appearance with realistic animations can greatly enhance user experience in social and professional virtual environments.

Conclusion

The development and implementation of Layered Gaussian Avatars represent a significant advance in the technology of digital avatars and virtual clothing. By addressing the problems associated with traditional methods and leveraging the power of Gaussian maps and careful layer separation, LayGA provides a robust solution for animatable clothing transfer, marking a step forward in the evolution of virtual identity and digital fashion industries. As technology progresses, the integration of such systems in everyday applications seems both promising and inevitable.