SurFhead: Affine Rig Blending for Geometrically Accurate 2D Gaussian Surfel Head Avatars (2410.11682v2)

Abstract: Recent advancements in head avatar rendering using Gaussian primitives have achieved significantly high-fidelity results. Although precise head geometry is crucial for applications like mesh reconstruction and relighting, current methods struggle to capture intricate geometric details and render unseen poses due to their reliance on similarity transformations, which cannot handle stretch and shear transforms essential for detailed deformations of geometry. To address this, we propose SurFhead, a novel method that reconstructs riggable head geometry from RGB videos using 2D Gaussian surfels, which offer well-defined geometric properties, such as precise depth from fixed ray intersections and normals derived from their surface orientation, making them advantageous over 3D counterparts. SurFhead ensures high-fidelity rendering of both normals and images, even in extreme poses, by leveraging classical mesh-based deformation transfer and affine transformation interpolation. SurFhead introduces precise geometric deformation and blends surfels through polar decomposition of transformations, including those affecting normals. Our key contribution lies in bridging classical graphics techniques, such as mesh-based deformation, with modern Gaussian primitives, achieving state-of-the-art geometry reconstruction and rendering quality. Unlike previous avatar rendering approaches, SurFhead enables efficient reconstruction driven by Gaussian primitives while preserving high-fidelity geometry.

• The paper presents an affine rig blending method that leverages 2D Gaussian surfels to reconstruct detailed head geometry from RGB videos.
• It employs polar decomposition and Jacobian Blend Skinning to capture complex deformations and ensure smooth transitions in extreme poses.
• Experimental results demonstrate significant improvements in geometric fidelity over baseline models on both synthetic and real multi-view datasets.

SurFhead: Advancements in Gaussian Surfel Head Avatars

The development of personalized head avatars is a dynamic and complex field within computer graphics and computer vision. The paper presents SurFhead, a method designed to improve the geometric accuracy of head avatars rendered through Gaussian primitives. This model leverages 2D Gaussian surfels (sparse surface elements) to reconstruct detailed head geometry from RGB videos, which is a compelling and efficient approach for applications such as mesh reconstruction and relighting.

Technical Contributions

SurFhead addresses several challenges in high-fidelity avatar rendition, primarily focusing on the limitations imposed by similarity transformations. These transformations struggle with detailed geometric deformations, particularly those involving stretch and shear. SurFhead's key contribution lies in utilizing affine transformations and classical mesh-based techniques for effective deformation capture, thus allowing unprecedented geometric precision.

Central to SurFhead's innovation is the interpolation of affine transformations via polar decomposition. This process mitigates potential distortions in the normal directions of Gaussian primitives, which are critical for realistic rendering. Unlike prior rigid, isotropic transformation models, SurFhead employs affine transformations to accommodate complex deformations encountered in extreme poses, enabling the system to maintain high fidelity in rendered normals and images.

Methodology

The SurFhead framework binds 2D Gaussian surfel splats to a parametric morphable face model. This approach allows for comprehensive control over expressiveness and viewpoints. The Gaussian-based representation facilitates efficient reconstruction, as it provides precise depth from fixed ray intersections, alongside natural interpolation via Jacobian Blend Skinning (JBS). JBS effectively minimizes geometric discontinuities during extrapolation by linearly interpolating transformations across adjacent mesh triangles, providing a smooth animation transition.

Additionally, SurFhead’s regularization techniques address common issues observed with eyeball renderings, such as the hollow illusion effect, often leading to inaccurately concave geometry. Here, they utilize computationally efficient Anisotropic Spherical Gaussians (ASGs) to better handle high specularity and thereby model the convexity of the cornea more accurately.

Experimental Evaluation

The authors validated SurFhead using both synthetic datasets with ground truth normals and real, multi-view datasets, demonstrating significant advancements in geometry fidelity over baseline models. The results on NeRSemble datasets showcase SurFhead's superiority in extreme pose scenarios, where traditional methods exhibit rendering artifacts due to coarse deformation strategies.

Implications and Future Work

SurFhead represents a significant step forward in head avatar modeling, providing the tools necessary for precise and effective geometric transformations. Its approaches may lead to improved dynamic mesh reconstructions and more detailed relighting effects, aligning well with the ambitions of neural rendering fields.

In the future, further refinements can be anticipated in handling overly exaggerated expressions and integrating other facial components, such as hair and tongue, which remain a challenge under the current 3DMFM model framework. Notably, achieving efficiency in dense, real-time environments remains an ongoing endeavor, with potential optimizations in algorithmic components such as the polar decomposition routine.

SurFhead not only addresses key technical challenges but also offers a robust framework for further exploration and enhancement within AI-driven graphics and avatar rendering contexts.