InvertAvatar: Incremental GAN Inversion for Generalized Head Avatars (2312.02222v3)

Abstract: While high fidelity and efficiency are central to the creation of digital head avatars, recent methods relying on 2D or 3D generative models often experience limitations such as shape distortion, expression inaccuracy, and identity flickering. Additionally, existing one-shot inversion techniques fail to fully leverage multiple input images for detailed feature extraction. We propose a novel framework, \textbf{Incremental 3D GAN Inversion}, that enhances avatar reconstruction performance using an algorithm designed to increase the fidelity from multiple frames, resulting in improved reconstruction quality proportional to frame count. Our method introduces a unique animatable 3D GAN prior with two crucial modifications for enhanced expression controllability alongside an innovative neural texture encoder that categorizes texture feature spaces based on UV parameterization. Differentiating from traditional techniques, our architecture emphasizes pixel-aligned image-to-image translation, mitigating the need to learn correspondences between observation and canonical spaces. Furthermore, we incorporate ConvGRU-based recurrent networks for temporal data aggregation from multiple frames, boosting geometry and texture detail reconstruction. The proposed paradigm demonstrates state-of-the-art performance on one-shot and few-shot avatar animation tasks. Code will be available at https://github.com/XChenZ/invertAvatar.

• The paper introduces a novel incremental GAN inversion framework that rapidly reconstructs high-fidelity 3D facial avatars from one-shot or few-shot inputs.
• It integrates an animatable 3D GAN prior with a unique neural texture encoder and ConvGRU-based temporal aggregation to enhance detail and control.
• The approach outperforms traditional methods in photorealism and efficiency, advancing applications in AR/VR, 3D telepresence, and video conferencing.

InvertAvatar: Bridging the Gap in High-Fidelity 3D Facial Avatar Creation

The quest for creating photorealistic 3D avatars from simple 2D images has seen a significant breakthrough with the introduction of a framework known as InvertAvatar. This system facilitates the rapid conversion of source images into detailed 3D facial avatars, capable of expressing full-head rotations and nuanced expressions within a second. This is a noteworthy advancement for applications across augmented and virtual reality (AR/VR), 3D telepresence, and video conferencing, where the demand for high-fidelity and efficiency persists.

At the core of InvertAvatar is the implementation of a unique animatable 3D GAN (Generative Adversarial Network) prior. This sophisticated prior is enhanced with two key modifications aiming to improve upon the control of facial expressions. In addition, the framework introduces a novel neural texture encoder which organizes texture features based on UV parameterization. The strategy is distinct from contemporary methods that rely on complex networks to bridge the gap between the posed and canonical representations. Instead, focusing on pixel-aligned image-to-image translation ensures that the reconstruction retains meticulous detail.

Another remarkable component of the InvertAvatar methodology is the use of ConvGRU-based recurrent networks which excel at aggregating temporal data across multiple frames. This attribute allows the system to maintain or discard specific information during the reconstruction process. The result is a more refined geometry and detailed texturing of the avatars, with the final product's fidelity incrementally improving as the number of input frames increases.

InvertAvatar stands superior in one-shot and few-shot avatar animation tasks when compared to existing techniques. Its capabilities prove to outshine alternative methods in verisimilitude and control over finer details such as hair textures and subtle facial expressions. Most current methods either suffer from fidelity issues or depend heavily on optimization for each new identity introduced, leading to significant time consumption.

The workflow of InvertAvatar comprises a coarse-to-fine inversion architecture that operates in both latent and canonical feature spaces, executing the GAN inversion efficiently. Initially, a latent encoder projects the image onto the latent space of a pre-trained GAN. Following this, the one-shot learning-based reconstruction takes over, maximizing the extraction of personal details from a single frame. As a step further, the use of Recurrent Neural Networks enables temporal aggregation from multiple frames, consequently, escalating the accuracy of the avatar reconstruction considering sequential data.

Despite its significant accomplishments, the system exhibits limitations when dealing with extreme expressions beyond the 3D facial model's capacity, such as complex frowns or tongue movements. Additionally, instances such as dynamic lower teeth representation still pose a challenge. However, ongoing improvements in robust facial models are expected to mitigate these issues.

In conclusion, the InvertAvatar framework has charted a new territory in the field of GAN inversion methods, providing a path for both research and commercial applications in creating lifelike digital personas. It presents a flexible solution for generating personalized head avatars and is anticipated to influence the future direction of digital human modeling significantly. The foresight into potential misuse for creating "deepfakes" also suggests a need for ethical guardrails around the technology's deployment.