GANimation: Anatomically-aware Facial Animation from a Single Image (1807.09251v2)

Abstract: Recent advances in Generative Adversarial Networks (GANs) have shown impressive results for task of facial expression synthesis. The most successful architecture is StarGAN, that conditions GANs generation process with images of a specific domain, namely a set of images of persons sharing the same expression. While effective, this approach can only generate a discrete number of expressions, determined by the content of the dataset. To address this limitation, in this paper, we introduce a novel GAN conditioning scheme based on Action Units (AU) annotations, which describes in a continuous manifold the anatomical facial movements defining a human expression. Our approach allows controlling the magnitude of activation of each AU and combine several of them. Additionally, we propose a fully unsupervised strategy to train the model, that only requires images annotated with their activated AUs, and exploit attention mechanisms that make our network robust to changing backgrounds and lighting conditions. Extensive evaluation show that our approach goes beyond competing conditional generators both in the capability to synthesize a much wider range of expressions ruled by anatomically feasible muscle movements, as in the capacity of dealing with images in the wild.

Overview of GANimation: Anatomically-aware Facial Animation from a Single Image

The paper "GANimation: Anatomically-aware Facial Animation from a Single Image" by Pumarola et al. presents a novel approach to facial expression synthesis using Generative Adversarial Networks (GANs). This work introduces an innovative conditioning mechanism that leverages Action Units (AUs), enabling continuous and anatomically-feasible animations. The proposed methodology addresses limitations found in existing models such as StarGAN, which support only a discrete set of facial expressions.

Contributions

This research stands out by utilizing AUs, which are based on the Facial Action Coding System (FACS), to model facial expressions more flexibly and naturally. By moving beyond the discrete emotion categories, GANimation generates a broad spectrum of expressions via continuous AU adjustment, mimicking realistic muscle movements. This capability is complemented by an unsupervised training strategy, eliminating the need for paired datasets of the same individual under different expressions.

Furthermore, the integration of attention mechanisms improves the robustness of the model when handling variable backgrounds and lighting conditions. This approach ensures that the generation remains focused on pertinent facial regions, thus preserving background integrity and addressing the challenges of non-uniform environments.

Methodology

The model architecture comprises two components: a generator and a conditional critic. The generator outputs two masks—a color mask and an attention mask—that facilitate precise control over the modification of facial features in the image. This dual-mask system enhances image quality and realism. Meanwhile, the critic employs WGAN-GP optimizations to ensure photorealism and adherence to the target expression, verified through the estimation of action units.

Training involves several loss functions including adversarial, attention, conditional expression, and identity losses. These functions collectively guide the generator to produce visually coherent outputs while maintaining identity fidelity across transformations.

Evaluation and Results

GANimation's effectiveness is validated through extensive experimentation. Compared to leading alternatives, such as StarGAN, GANimation demonstrates superior expressiveness and quality, especially in managing continuous expressions and complex backgrounds. The system’s ability to animate expressions smoothly and adaptively in videos is emphasized as a future area of development.

The paper reports favorable performance on challenging datasets like EmotioNet, showcasing GANimation's capability in generating a diverse array of expressions with minimal artifacts. The model's attention mechanism significantly contributes to handling images "in the wild," enabling applications in real-world and varied contexts.

Implications and Future Work

Practically, the advancements in GANimation have potential applications in industries such as film, virtual reality, and human-computer interaction, where realistic facial animations enhance user experience and engagement. Theoretically, this work opens new research avenues in continuous expression modeling and unsupervised facial representation learning.

Future developments could explore the adaptation of this methodology to video sequences, enhancing temporal coherence and reducing flicker effects. Furthermore, integrating reinforcement learning could refine action unit mappings by rewarding model outputs that adhere more closely to desired expressions.

In summary, GANimation sets a pivotal milestone in facial animation research, demonstrating innovative uses of action units within GANs to produce rich, realistic facial expressions. Its strength lies in anatomical awareness, robustness in varied conditions, and its broad implications for both academia and industry.