SAFA: Structure Aware Face Animation (2111.04928v1)

Abstract: Recent success of generative adversarial networks (GAN) has made great progress on the face animation task. However, the complex scene structure of a face image still makes it a challenge to generate videos with face poses significantly deviating from the source image. On one hand, without knowing the facial geometric structure, generated face images might be improperly distorted. On the other hand, some area of the generated image might be occluded in the source image, which makes it difficult for GAN to generate realistic appearance. To address these problems, we propose a structure aware face animation (SAFA) method which constructs specific geometric structures to model different components of a face image. Following the well recognized motion based face animation technique, we use a 3D morphable model (3DMM) to model the face, multiple affine transforms to model the other foreground components like hair and beard, and an identity transform to model the background. The 3DMM geometric embedding not only helps generate realistic structure for the driving scene, but also contributes to better perception of occluded area in the generated image. Besides, we further propose to exploit the widely studied inpainting technique to faithfully recover the occluded image area. Both quantitative and qualitative experiment results have shown the superiority of our method. Code is available at https://github.com/Qiulin-W/SAFA.

• The paper introduces SAFA as a novel method integrating 3D morphable models and GANs to tackle occlusions and pose challenges.
• It combines motion modeling, inpainting techniques, and geometrically-adaptive denormalization for improved identity and realism.
• Experimental results show SAFA outperforms state-of-the-art methods on metrics like AKD and FID, enabling robust facial animation.

Structure Aware Face Animation: An Advanced Approach

The paper "SAFA: Structure Aware Face Animation" presents an innovative approach to addressing challenges in face animation, particularly concerning occlusions and pose variations. Leveraging the capabilities of Generative Adversarial Networks (GANs) and 3D morphable models (3DMMs), the authors propose the SAFA method, which integrates detailed scene structure knowledge into the animation process.

Methodology

SAFA combines 2D and 3D modeling to address key challenges in face animation, such as pose preservation, identity preservation, realism, and occlusion awareness. The method leverages:

1. 3D Morphable Models (3DMM): Using the state-of-the-art FLAME model, SAFA captures the facial geometric structure to assist in creating accurate and realistic animations. The 3DMM aids in defining facial shape, expression, and motion, providing a robust structural framework for animation.
2. Motion Modeling: The method differentiates and separately models the face, other foreground elements like hair and beard, and the background. By employing a 3D morphable model for the face and affine transformations for other components, SAFA efficiently handles complex poses and occlusions.
3. Inpainting Techniques: The application of contextual attention modules enhances the ability of SAFA to reconstruct occluded areas in the animated image, allowing for seamless facial animations even amidst substantial occlusions.
4. Geometrically-Adaptive Denormalization (GADE): This novel layer integrates 3D geometric embeddings to further refine facial detail generation, leveraging the perceived geometry for enhanced realism.

Experimental Results

The experimental results, both qualitative and quantitative, demonstrate SAFA's superiority over existing methods such as Few-Shot Vid2Vid, Fast Bi-layer, and FOMM. Key metrics such as Average Keypoint Distance (AKD) and Fréchet Inception Distance (FID) indicate better pose accuracy and visual realism in generated videos. In contexts with significant pose shifts and occlusions, SAFA outperforms state-of-the-art methods in maintaining identity and achieving higher-quality visual outputs.

Implications and Speculations

The integration of 3DMMs with GANs and the development of novel layers like GADE highlights the potential of blending 2D and 3D techniques in animation tasks. The enhanced ability to deal with occlusions and large deviations in face poses opens avenues for more complex and realistic applications in entertainment, virtual reality, and telepresence.

This research also underscores the growing importance of structural awareness in neural network architectures, especially for tasks needing detailed geometric understanding. Future developments might focus on further improving efficiency and expanding the technique to handle even more dynamic scenarios or diverse datasets.

Overall, SAFA's contributions lie in its nuanced approach to face animation, pioneering the integration of explicit 3D structural knowledge with advanced machine learning techniques to yield superior results.