3D Gaussian Blendshapes for Head Avatar Animation (2404.19398v2)

Abstract: We introduce 3D Gaussian blendshapes for modeling photorealistic head avatars. Taking a monocular video as input, we learn a base head model of neutral expression, along with a group of expression blendshapes, each of which corresponds to a basis expression in classical parametric face models. Both the neutral model and expression blendshapes are represented as 3D Gaussians, which contain a few properties to depict the avatar appearance. The avatar model of an arbitrary expression can be effectively generated by combining the neutral model and expression blendshapes through linear blending of Gaussians with the expression coefficients. High-fidelity head avatar animations can be synthesized in real time using Gaussian splatting. Compared to state-of-the-art methods, our Gaussian blendshape representation better captures high-frequency details exhibited in input video, and achieves superior rendering performance.

• The paper introduces a novel 3D Gaussian blendshape representation that enhances photorealistic head animations.
• It utilizes a neutral base model with expression blendshapes and an optimized training strategy to capture high-frequency details.
• Experimental results demonstrate superior PSNR, SSIM, and rendering speeds up to 370 FPS compared to state-of-the-art methods.

3D Gaussian Blendshapes for Head Avatar Animation

This paper introduces an innovative method for creating photorealistic head avatars using 3D Gaussian blendshapes, significantly enhancing real-time avatar animations in terms of fidelity and speed. The authors develop a novel representation and learning mechanism leveraging 3D Gaussians, representing a neutral head model and a set of expression blendshapes. These blendshapes allow linear blending with expression coefficients to facilitate real-time head animation, focusing on capturing high-frequency details from input monocular videos.

Technical Contributions and Methodology

The core contribution lies in the use of 3D Gaussian blendshapes, which offer a compelling alternative to traditional mesh-based approaches. The representation comprises:

1. Neutral Base Model: Represented using 3D Gaussians that encapsulate basic properties like position, opacity, rotation, and color.
2. Expression Blendshapes: These complement the neutral model, allowing the construction of diverse facial expressions through linear blending.
3. Optimization Strategy: The authors propose a training method that ensures the difference between Gaussian blendshapes aligns semantically with the corresponding mesh blendshapes. This is achieved using an intermediate variable that scales Gaussian differences proportionally to mesh positional displacements.
4. Mouth Interior Gaussians: A dedicated set of Gaussians manage the mouth interior, improving the rendering of teeth and internal mouth movements.

The paper details how to initialize and optimize these components using monocular video inputs to produce a dynamic and photorealistic avatar model that can be animated in real-time.

Experimental Validation

The authors provide extensive experimental results demonstrating the superiority of their approach over state-of-the-art methods such as INSTA, PointAvatar, and NeRFBlendShape. When compared, their method consistently achieves higher PSNR and SSIM scores across various datasets, while maintaining a significant performance advantage with rendering speeds of 370 frames per second.

Implications and Future Directions

The development and application of Gaussian blendshapes represent a significant step forward in avatar animation, offering a more efficient and detailed representation. For experts in the field, this paper suggests new possibilities for head avatar synthesis, particularly in the domains of telepresence and virtual reality, where high fidelity and real-time performance are crucial.

Future research could explore expanding this approach to incorporate more complex deformations or integrate it with other neural techniques, potentially enhancing realism and performance further. While the current method excels at reproducing expressions seen during training, handling exaggerated or completely novel expressions remains a challenge, indicating a potential area for further exploration.

Conclusion

This paper provides a thorough exploration of 3D Gaussian blendshapes, offering valuable insights into advanced avatar animation techniques. The integration of Gaussian splatting for rendering purposes, coupled with effective training methodologies, marks a notable advancement in the field, presenting both practical and theoretical contributions to computer graphics and interactive techniques.