StyleMask: Disentangling the Style Space of StyleGAN2 for Neural Face Reenactment (2209.13375v1)

Abstract: In this paper we address the problem of neural face reenactment, where, given a pair of a source and a target facial image, we need to transfer the target's pose (defined as the head pose and its facial expressions) to the source image, by preserving at the same time the source's identity characteristics (e.g., facial shape, hair style, etc), even in the challenging case where the source and the target faces belong to different identities. In doing so, we address some of the limitations of the state-of-the-art works, namely, a) that they depend on paired training data (i.e., source and target faces have the same identity), b) that they rely on labeled data during inference, and c) that they do not preserve identity in large head pose changes. More specifically, we propose a framework that, using unpaired randomly generated facial images, learns to disentangle the identity characteristics of the face from its pose by incorporating the recently introduced style space $\mathcal{S}$ of StyleGAN2, a latent representation space that exhibits remarkable disentanglement properties. By capitalizing on this, we learn to successfully mix a pair of source and target style codes using supervision from a 3D model. The resulting latent code, that is subsequently used for reenactment, consists of latent units corresponding to the facial pose of the target only and of units corresponding to the identity of the source only, leading to notable improvement in the reenactment performance compared to recent state-of-the-art methods. In comparison to state of the art, we quantitatively and qualitatively show that the proposed method produces higher quality results even on extreme pose variations. Finally, we report results on real images by first embedding them on the latent space of the pretrained generator. We make the code and pretrained models publicly available at: https://github.com/StelaBou/StyleMask

• The paper introduces StyleMask, which leverages StyleGAN2’s style space to disentangle facial attributes without requiring paired training data.
• It employs a mask network with 3D model supervision to preserve source identity while accurately transferring target pose and expression.
• Quantitative and qualitative results show superior performance in identity preservation and pose transfer compared to state-of-the-art methods.

StyleMask: Disentangling the Style Space of StyleGAN2 for Neural Face Reenactment

The paper "StyleMask: Disentangling the Style Space of StyleGAN2 for Neural Face Reenactment" explores the application of the StyleGAN2 architecture in creating a framework for neural face reenactment. Specifically, it focuses on the challenges of transferring facial attributes such as pose and expression from a target image to a source image while maintaining the source identity. The unique contribution of this work is the utilization of a latent representation termed the style space $\mathcal{S}$ of StyleGAN2, which offers notable disentanglement properties conducive for this task.

Methodology

The authors propose a novel approach, StyleMask, which leverages the style space of StyleGAN2 to disentangle and transfer facial attributes effectively. The framework does not rely on paired data, where the source and target faces share the same identity. Instead, it learns from unpaired, randomly generated facial images. The method involves learning a mask network that operates in the style space to selectively combine the style codes of the source and target images, preserving the identity characteristics of the source and transferring the target's pose.

Key elements of the approach include:

• Style Space Utilization: StyleGAN2's style space is known for its ability to disentangle components of the image generation process, which is critical for isolating identity features from posture and expression.
• 3D Model Supervision: The framework employs a 3D model to supervise the disentanglement process, ensuring that the latent code mixes the correct components of the source and target styles.
• Unpaired Training: The method is trained on synthetic data without needing paired human image datasets, extending its applicability to faces with distinct identities.

Results

The authors report both qualitative and quantitative experiments demonstrating the proposed method's efficacy compared to existing state-of-the-art methods such as StyleFusion, ID-disentanglement models, and other GAN-based face reenactment techniques. Notable outcomes include:

• Higher scores in preserving source identity (CSIM) and transferring target pose (NME, Pose metrics).
• Superior performance in extreme pose variations, which are challenging for many existing face reenactment methods.
• Competitive quality metrics (e.g., FID scores) indicating high fidelity in the reenacted images.

Implications and Future Directions

This research provides significant insights into face reenactment and offers a path forward for developing more advanced and flexible GAN models capable of handling unpaired data. The disentanglement of identity and pose in the style space presents opportunities to enhance controllability and realism in synthesized images. Furthermore, making the code and trained models publicly available encourages transparency and facilitates further research.

Potential future directions stemming from this work include:

• Extending the approach to other image generation tasks requiring semantic disentanglement.
• Investigating the adaptability of the method to video data for dynamic face reenactment.
• Exploring the integration of additional pre-trained models for enhanced supervision and accuracy.

In conclusion, through StyleMask, the authors contribute a robust methodology leveraging the StyleGAN2 architecture for effective neural face reenactment, highlighting the potential of unpaired training paradigms and the rich representation capability of latent style spaces in generative adversarial networks.