MyStyle++: A Controllable Personalized Generative Prior (2306.04865v3)

Abstract: In this paper, we propose an approach to obtain a personalized generative prior with explicit control over a set of attributes. We build upon MyStyle, a recently introduced method, that tunes the weights of a pre-trained StyleGAN face generator on a few images of an individual. This system allows synthesizing, editing, and enhancing images of the target individual with high fidelity to their facial features. However, MyStyle does not demonstrate precise control over the attributes of the generated images. We propose to address this problem through a novel optimization system that organizes the latent space in addition to tuning the generator. Our key contribution is to formulate a loss that arranges the latent codes, corresponding to the input images, along a set of specific directions according to their attributes. We demonstrate that our approach, dubbed MyStyle++, is able to synthesize, edit, and enhance images of an individual with great control over the attributes, while preserving the unique facial characteristics of that individual.

• The paper introduces a novel optimization framework that reorganizes StyleGAN’s latent space using PCA for precise attribute control.
• It refines both latent codes and generator weights with a new loss formulation, achieving superior identity preservation and attribute consistency.
• Its method enables controlled synthesis and semantic editing, offering practical advancements in personalized image generation.

Overview of MyStyle++: A Controllable Personalized Generative Prior

The paper "MyStyle++: A Controllable Personalized Generative Prior" presents a novel method for synthesizing images of an individual with high fidelity while allowing precise control over specific attributes. This work is based on the refinement of MyStyle, a previously proposed approach that personalizes a pre-trained StyleGAN generator to reconstruct images faithfully for a given individual. While MyStyle offers the advantage of personalized synthesis, it falls short in allowing explicit control over generated image attributes, an area in which MyStyle++ shows significant improvements.

MyStyle++ introduces an optimization framework that rearranges anchors, i.e., latent codes obtained by projecting input images into the latent space, to organize this space methodically. By using a set of pre-defined directions, MyStyle++ allows controlled synthesis, semantic editing, and enhancement of personalized images. This capability is achieved without sacrificing the unique facial characteristics of the individual of interest.

Key Contributions

1. Attribute Control through Latent Space Organization: MyStyle++ employs a principal component analysis (PCA) technique to organize the anchor points within the latent space of StyleGAN. Unlike MyStyle, which adapts the generator without explicit control over attributes, MyStyle++ refines not only the generator but also arranges the latent space enabling precise attribute adjustments.
2. Optimization System: A crucial element of the work is its optimization system, which incorporates a novel loss formulation. This system tunes both the latent codes (anchors) and generator weights, allowing for structured latent space to represent attributes more effectively than existing methods such as MyStyle using InterFaceGAN directions.
3. Applications and Practical Implications: The paper demonstrates the applicability of MyStyle++ in various tasks like controlled synthesis, editing, and image enhancement. It can personalize attributes like expression, yaw, pitch, and age with high attributional precision. This makes it beneficial for applications in fields requiring accurate and personalized image generation and editing.
4. Comparative Evaluation: MyStyle++ is shown to outperform previous versions in its ability to retain identity while controlling specific attributes. Experimental results indicate its superiority over MyStyle variants using both InterFaceGAN and PCA directions, with improved consistency in synthesized attributes and better disentanglement between attributes.

Numerical Results

The paper presents a thorough comparative analysis, reporting strong numerical results on several metrics. Specifically, MyStyle++ consistently shows lower standard deviation in controlled attribute synthesis when compared to MyStyle adaptations, confirming its improved ability to maintain consistent attributes across generated samples. Its identity preservation capability is further confirmed using ID metrics, a testament to its refined generator tuning and latent space organization.

Future Developments and Limitations

Despite its advancements, MyStyle++ requires a higher number of images for effective personalization, especially as the attribute dimensionality increases. Additionally, the lack of physical accuracy in some edited attributes points towards future integrations with models incorporating the image formation process.

The framework's potential applicability is theoretically extensible beyond singular subject cases, suggesting possible future work to organize the entire latent space of StyleGAN. However, the challenge of computational demand and the prevention of identity mixing during latent space organization on a broader scope remains an open area for research.

Conclusion

MyStyle++ provides a significant leap in the controllability of personalized generative priors. It enables a more structured and precise way to manipulate a person's image attributes, maintaining high fidelity to original identity features. This paper not only enhances existing methodologies but also opens avenues for further exploration in controlled and personalized image synthesis using generative networks like StyleGAN.