Dual In-painting Model for Unsupervised Gaze Correction and Animation in the Wild (2008.03834v1)

Abstract: In this paper we address the problem of unsupervised gaze correction in the wild, presenting a solution that works without the need for precise annotations of the gaze angle and the head pose. We have created a new dataset called CelebAGaze, which consists of two domains X, Y, where the eyes are either staring at the camera or somewhere else. Our method consists of three novel modules: the Gaze Correction module (GCM), the Gaze Animation module (GAM), and the Pretrained Autoencoder module (PAM). Specifically, GCM and GAM separately train a dual in-painting network using data from the domain $X$ for gaze correction and data from the domain $Y$ for gaze animation. Additionally, a Synthesis-As-Training method is proposed when training GAM to encourage the features encoded from the eye region to be correlated with the angle information, resulting in a gaze animation which can be achieved by interpolation in the latent space. To further preserve the identity information~(e.g., eye shape, iris color), we propose the PAM with an Autoencoder, which is based on Self-Supervised mirror learning where the bottleneck features are angle-invariant and which works as an extra input to the dual in-painting models. Extensive experiments validate the effectiveness of the proposed method for gaze correction and gaze animation in the wild and demonstrate the superiority of our approach in producing more compelling results than state-of-the-art baselines. Our code, the pretrained models and the supplementary material are available at: https://github.com/zhangqianhui/GazeAnimation.

• The paper introduces a dual in-painting framework that achieves unsupervised gaze correction and animation without requiring labeled data.
• It employs a novel Synthesis-As-Training technique to integrate eye region features with gaze angles for realistic transformations.
• The model preserves key identity features using a self-supervised mirror learning method that outperforms existing state-of-the-art baselines.

Summary of "Dual In-painting Model for Unsupervised Gaze Correction and Animation in the Wild"

The paper presents a novel approach for unsupervised gaze correction and animation using a dual in-painting model. The authors introduce a dataset called CelebAGaze, which features two domains: one where individuals are gazing at the camera (domain X) and another where they are looking elsewhere (domain Y). The method involves three main modules: the Gaze Correction module (GCM), Gaze Animation module (GAM), and Pretrained Autoencoder module (PAM). Each module contributes uniquely to the task of in-painting and transforming eye gaze in images taken in unconstrained environments.

The approach leverages a new Synthesis-As-Training technique that ties the eye region features with gaze angles during gaze animation. A self-supervised mirror learning technique is used to maintain identity features, such as eye shape and iris color, during transformation via pre-trained autoencoders. GCM and GAM operate by training a dual in-painting network utilizing domain data to achieve accurate gaze correction and animation.

Key outcomes from this research are the effectiveness of these modules in adjusting eye gaze without requiring labeled training data, as well as their robustness in preserving facial identity features. The model outperforms state-of-the-art baselines in producing more realistic images for gaze correction tasks in the wild.

This paper holds significant implications for applications in video conferencing and photography, where direct gaze and eye contact are vital. It pushes forward the theoretical framework for unsupervised eye-gaze correction and opens up potential future developments in AI-based image manipulation, especially as regards fine-grained in-painting tasks. The proposed in-painting models could be adapted to other domains where correction and animation of localized features are necessary, setting a novel direction for unsupervised learning methodologies in computer vision.