DiffusionRig: Learning Personalized Priors for Facial Appearance Editing (2304.06711v1)

Abstract: We address the problem of learning person-specific facial priors from a small number (e.g., 20) of portrait photos of the same person. This enables us to edit this specific person's facial appearance, such as expression and lighting, while preserving their identity and high-frequency facial details. Key to our approach, which we dub DiffusionRig, is a diffusion model conditioned on, or "rigged by," crude 3D face models estimated from single in-the-wild images by an off-the-shelf estimator. On a high level, DiffusionRig learns to map simplistic renderings of 3D face models to realistic photos of a given person. Specifically, DiffusionRig is trained in two stages: It first learns generic facial priors from a large-scale face dataset and then person-specific priors from a small portrait photo collection of the person of interest. By learning the CGI-to-photo mapping with such personalized priors, DiffusionRig can "rig" the lighting, facial expression, head pose, etc. of a portrait photo, conditioned only on coarse 3D models while preserving this person's identity and other high-frequency characteristics. Qualitative and quantitative experiments show that DiffusionRig outperforms existing approaches in both identity preservation and photorealism. Please see the project website: https://diffusionrig.github.io for the supplemental material, video, code, and data.

• The paper introduces DiffusionRig which learns personalized facial priors from a limited image set to preserve identity while editing features.
• It employs a two-stage training process that uses large-scale datasets to establish generic facial priors before fine-tuning with individual images.
• Results demonstrate that DiffusionRig outperforms existing methods in identity preservation and photorealism, supported by lower RMSE and positive user studies.

Overview of DiffusionRig: Personalized Priors for Facial Appearance Editing

The paper "DiffusionRig: Learning Personalized Priors for Facial Appearance Editing" introduces a novel approach for facial appearance manipulation that emphasizes personalized priors. The method combines the capabilities of both diffusion models and 3D Morphable Models (3DMMs) to address the constraints in altering facial expressions, lighting, and pose in images while preserving key identity characteristics. This research is particularly relevant within the fields of computer vision and graphics, where detailed and personalized facial editing remains a complex challenge.

The proposed framework, DiffusionRig, seeks to learn personalized facial priors from a limited set of images (approximately 20 photos) specific to an individual. This model is distinctive in its ability to retain the individual's identity and high-frequency facial details, which is often a significant limitation in other generative models reliant on large-scale database training without individual-specific finetuning.

Methodology

DiffusionRig operates in two principal stages. Initially, it learns generic facial priors by training on a large-scale facial dataset, such as the CelebA dataset. In this phase, the model establishes a baseline understanding of facial features that is not specific to any individual. Following this, the second stage consists of finetuning the model with a small set of personal images to develop personalized priors. This two-step training process facilitates identity preservation even with substantial modifications in facial attributes.

The innovation leverages a diffusion model conditioned on inputs derived from 3D face models. Such models utilize parameters like lighting, expression, and pose to provide a coarse representation of facial features. By mapping these simplified inputs to photorealistic outputs, DiffusionRig exhibits a balance between controllable editing and fidelity to the individual's appearance.

Results and Analysis

Experiments conducted within the paper demonstrate that DiffusionRig outperforms existing approaches in maintaining identity and generating photorealistic results. The paper highlights quantitative results such as the lower root mean square error (RMSE) in DECA re-inference compared to other models like GIF (GAN-based), indicating superior performance in adhering to modified physical attributes (lighting, shape, expression, and pose).

Additionally, qualitative evaluations were supported by user studies that assessed both photorealism and identity preservation. The studies revealed that the images generated using DiffusionRig were preferred over alternative methods, confirming the system's efficacy.

Implications and Future Directions

From a practical perspective, this work has notable implications for applications requiring personalized image manipulation including entertainment, virtual reality, and augmented reality domains. By allowing precise edits with minimal identity distortion, DiffusionRig provides a platform for more interactive and personalized user experiences.

Theoretically, this approach reinforces the viability of diffusion models as robust generative tools capable of detailed, condition-based edits when combined with preexisting model architectures like 3DMMs. It opens further research avenues, particularly in investigating the scalability of personalized prior learning and optimizing such processes for real-time applications. The current reliance on a small personal dataset for finetuning suggests opportunities to refine the approach further, potentially through more sophisticated initial model training or semi-supervised methods to reduce the reliance on user-specific data.

In conclusion, DiffusionRig represents a significant advancement in facial appearance editing by integrating diffusion models and 3DMMs to achieve both high-quality and personalized results. This work lays a foundation for future research aimed at expanding the scope and efficiency of personalized generative models in computer vision.