HeadStudio: Text to Animatable Head Avatars with 3D Gaussian Splatting
(2402.06149)Abstract
Creating digital avatars from textual prompts has long been a desirable yet challenging task. Despite the promising outcomes obtained through 2D diffusion priors in recent works, current methods face challenges in achieving high-quality and animated avatars effectively. In this paper, we present $\textbf{HeadStudio}$, a novel framework that utilizes 3D Gaussian splatting to generate realistic and animated avatars from text prompts. Our method drives 3D Gaussians semantically to create a flexible and achievable appearance through the intermediate FLAME representation. Specifically, we incorporate the FLAME into both 3D representation and score distillation: 1) FLAME-based 3D Gaussian splatting, driving 3D Gaussian points by rigging each point to a FLAME mesh. 2) FLAME-based score distillation sampling, utilizing FLAME-based fine-grained control signal to guide score distillation from the text prompt. Extensive experiments demonstrate the efficacy of HeadStudio in generating animatable avatars from textual prompts, exhibiting visually appealing appearances. The avatars are capable of rendering high-quality real-time ($\geq 40$ fps) novel views at a resolution of 1024. They can be smoothly controlled by real-world speech and video. We hope that HeadStudio can advance digital avatar creation and that the present method can widely be applied across various domains.
Overview
-
HeadStudio introduces a novel framework for generating realistic and animatable digital head avatars directly from text prompts using 3D Gaussian Splatting and the FLAME statistical head model.
-
The technique combines FLAME-based 3D Gaussian Splatting (F-3DGS) for accurate facial deformations with FLAME-based Score Distillation Sampling (F-SDS) for high semantic fidelity in animations.
-
HeadStudio achieves over 40 frames per second at 1024 resolution, setting a new standard for performance and quality in avatar generation.
-
It opens new avenues for applications in virtual/augmented reality, gaming, and online communications, with potential for expanding into more sophisticated avatar control and representation.
Leveraging 3D Gaussian Splatting for Realistic and Animatable Head Avatars from Text Prompts
Introduction to HeadStudio
In the realm of digital head avatars, generating high-quality and animated representations directly from text prompts is a formidable challenge. Recent advancements have pivoted towards text-based generation methods, showing promise over traditional image-based approaches due to their convenience and generalization capabilities. However, a recurring issue has been the trade-off between static quality and dynamism in animation. In response, we introduce HeadStudio, a cutting-edge framework designed to produce realistic and animatable avatars using 3D Gaussian Splatting (3DGS) and leveraging the FLAME statistical head model for semantic deformation and score distillation guidance.
Technical Foundation and Innovations of HeadStudio
HeadStudio stands at the intersection of 3D Gaussian Splatting and FLAME-based methodologies. The approach consists of two pivotal components:
- FLAME-based 3D Gaussian Splatting (F-3DGS): This technique rigs 3D Gaussian points to a FLAME mesh, ensuring that deformations adhere to facial expressions accurately. It capitalizes on FLAME's robust morphological control to drive the adaptation of 3D Gaussian points, factoring in facial movements and expressions seamlessly.
- FLAME-based Score Distillation Sampling (F-SDS): Leveraging a fine-grained FLAME-based control signal derived from the MediaPipe facial landmark map, F-SDS guides the distillation process. This ensures a high degree of semantic fidelity, enabling the generated avatars to perform realistic animations, driven by real-world speech and video inputs.
A detailed evaluation demonstrates HeadStudio's capability to generate animatable avatars exceeding 40 frames per second at 1024 resolution, marking a significant advancement in both performance and quality.
Practical Implications and Future Directions
HeadStudio not only broadens the scope of digital avatar creation but also introduces a novel methodological approach that could be extended to other domains. The integration of FLAME into both 3D representation and score distillation reflects a nuanced understanding of the underlying statistical model, paving the way for more sophisticated avatar manipulation and control. Furthermore, the ability to generate avatars that can be dynamically controlled in real-time opens new possibilities for applications in virtual and augmented reality, gaming, and online communication platforms.
The success of HeadStudio predicates further exploration into the amalgamation of 3D representation techniques and statistical models for even more detailed and expressive avatars. Future work may delve into enhancing the diversity of avatars, exploring other control signals for animation, and refining the balance between static fidelity and dynamic expression.
Conclusion
HeadStudio represents a notable stride towards resolving the long-standing challenge of generating high-fidelity, animatable head avatars from text prompts. By harnessing the power of 3D Gaussian Splatting and the FLAME model, it establishes a new benchmark for realism and animation capability in digital avatars. As the field of generative AI continues to evolve, approaches like HeadStudio underscore the potential for innovative cross-disciplinary applications, heralding a new era of digital representation.