Expressive Whole-Body 3D Gaussian Avatar (2407.21686v1)

Abstract: Facial expression and hand motions are necessary to express our emotions and interact with the world. Nevertheless, most of the 3D human avatars modeled from a casually captured video only support body motions without facial expressions and hand motions.In this work, we present ExAvatar, an expressive whole-body 3D human avatar learned from a short monocular video. We design ExAvatar as a combination of the whole-body parametric mesh model (SMPL-X) and 3D Gaussian Splatting (3DGS). The main challenges are 1) a limited diversity of facial expressions and poses in the video and 2) the absence of 3D observations, such as 3D scans and RGBD images. The limited diversity in the video makes animations with novel facial expressions and poses non-trivial. In addition, the absence of 3D observations could cause significant ambiguity in human parts that are not observed in the video, which can result in noticeable artifacts under novel motions. To address them, we introduce our hybrid representation of the mesh and 3D Gaussians. Our hybrid representation treats each 3D Gaussian as a vertex on the surface with pre-defined connectivity information (i.e., triangle faces) between them following the mesh topology of SMPL-X. It makes our ExAvatar animatable with novel facial expressions by driven by the facial expression space of SMPL-X. In addition, by using connectivity-based regularizers, we significantly reduce artifacts in novel facial expressions and poses.

• The paper introduces ExAvatar, a hybrid model combining SMPL-X and 3D Gaussian Splatting to generate expressive whole-body 3D avatars from short monocular videos.
• It mitigates data limitations using connectivity-based regularizers that reduce artifacts in facial expressions and dynamic poses.
• The comprehensive system achieves high-fidelity rendering, outperforming state-of-the-art methods in metrics like PSNR, SSIM, and LPIPS.

Expressive Whole-Body 3D Gaussian Avatar: An Expert Overview

The paper "Expressive Whole-Body 3D Gaussian Avatar" by Moon et al. introduces ExAvatar, an expressive whole-body 3D human avatar generated from short monocular videos. This work integrates the SMPL-X model and 3D Gaussian Splatting (3DGS) to overcome common challenges in creating 3D human avatars from limited and unstructured video data.

Key Contributions

1. Hybrid Representation: The paper presents a hybrid representation that enhances both the generative and control capabilities of the avatar. By treating each 3D Gaussian as a vertex on the surface with pre-defined connectivity, ExAvatar is capable of advanced animations driven by the facial expression space of SMPL-X.
2. Mitigating Data Limitations: The approach addresses major challenges, such as limited diversity of facial expressions and poses in video and the absence of 3D observations. Connectivity-based regularizers are introduced to reduce artifacts in novel expressions and poses, ensuring more robust and visually stable animations.
3. Comprehensive System: ExAvatar combines the whole-body parametric mesh of SMPL-X and photorealistic rendering powered by 3DGS. This system achieves high fidelity in representing identity-specific details and environment-dependent attributes without posing dependency.

Implementation Details

• Accurate Co-Registration:

The system begins with an accurate co-registration of SMPL-X using offsets for joints and facial vertices to handle human parts with limited datasets. This ensures more detailed and precise facial and hand modeling.

• Architecture:

ExAvatar utilizes a canonical mesh from SMPL-X, enhanced by a learnable triplane for extracting per-vertex features. Two sets of MLPs are employed: one for geometry and another for pose-dependent offsets, improving the overall expressiveness and adaptability of the model.

• Animation and Rendering:

The model seamlessly animates using SMPL-X’s facial expression codes and 3D poses. A novel integration with 3DGS ensures efficient and photorealistic rendering, maintaining consistent texture and geometry, especially crucial for dynamic scenes.

Results and Evaluation

The efficacy of ExAvatar is substantiated through extensive evaluations on datasets like NeuMan and X-Humans. Numerical results demonstrate superior performance in PSNR, SSIM, and LPIPS metrics against state-of-the-art methods such as Vid2Avatar and GaussianAvatar, reinforcing the robustness and quality of ExAvatar's outputs. Qualitative assessments validate the model's capability to maintain high fidelity in novel views and poses, especially around expressive regions such as faces and hands.

Theoretical and Practical Implications

• Theoretical:

The hybrid representation technique paves the way for future research integrating volumetric and mesh-based models. This method exemplifies a significant step forward in mitigating occlusion ambiguities and ensuring high-quality 3D reconstructions from minimal data.

• Practical:

ExAvatar has immediate applications in virtual reality, gaming, and digital avatar creation for social media and teleconferencing platforms. The model's ability to work with short video clips without requiring extensive, structured 3D observations makes it highly adaptable and practical for real-world uses.

Future Directions and Limitations

For future research, addressing the unobserved parts of the human model, such as the mouth's interior and dynamic clothing, could further enhance authenticity. Incorporating generative priors using techniques like Score Distillation Sampling could also improve hallucination of unobserved regions, thus boosting the realism of the generated avatars.

Conclusion

ExAvatar represents a significant leap in the modeling of expressive, whole-body 3D avatars from limited video data. By leveraging a hybrid mesh and 3D Gaussian representation, it achieves a nuanced balance between expressiveness, fidelity, and computational efficiency, setting a new benchmark in the field of 3D human modeling.