HairFastGAN: Realistic and Robust Hair Transfer with a Fast Encoder-Based Approach (2404.01094v3)

Abstract: Our paper addresses the complex task of transferring a hairstyle from a reference image to an input photo for virtual hair try-on. This task is challenging due to the need to adapt to various photo poses, the sensitivity of hairstyles, and the lack of objective metrics. The current state of the art hairstyle transfer methods use an optimization process for different parts of the approach, making them inexcusably slow. At the same time, faster encoder-based models are of very low quality because they either operate in StyleGAN's W+ space or use other low-dimensional image generators. Additionally, both approaches have a problem with hairstyle transfer when the source pose is very different from the target pose, because they either don't consider the pose at all or deal with it inefficiently. In our paper, we present the HairFast model, which uniquely solves these problems and achieves high resolution, near real-time performance, and superior reconstruction compared to optimization problem-based methods. Our solution includes a new architecture operating in the FS latent space of StyleGAN, an enhanced inpainting approach, and improved encoders for better alignment, color transfer, and a new encoder for post-processing. The effectiveness of our approach is demonstrated on realism metrics after random hairstyle transfer and reconstruction when the original hairstyle is transferred. In the most difficult scenario of transferring both shape and color of a hairstyle from different images, our method performs in less than a second on the Nvidia V100. Our code is available at https://github.com/AIRI-Institute/HairFastGAN.

• The paper introduces a novel fast encoder-based architecture that achieves high-quality hair transfer in real time.
• It designs four dedicated modules—embedding, alignment, blending, and post-processing—to overcome challenges in pose and color discrepancies.
• Experimental results demonstrate that HairFastGAN outperforms benchmarks in FID and CLIP metrics, balancing efficiency with fidelity.

Analyzing HairFastGAN: Advancements in Encoder-Based Hair Transfer

The paper "HairFastGAN: Realistic and Robust Hair Transfer with a Fast Encoder-Based Approach" presents a novel method in the domain of neural style transfer, focusing specifically on transferring hair attributes between images. This task is notoriously challenging due to the complexities of hairstyle sensitivity, variations in photo poses, and the lack of objective performance metrics.

Methodology Overview

HairFastGAN addresses these challenges by developing an encoder-based solution that achieves high-quality hair transfers while maintaining real-time processing speeds. Traditional optimization-based methods, although effective in preserving detail, suffer from inefficiencies. Conversely, prior encoder-based approaches sacrifice quality for speed. HairFastGAN integrates the advantages of both by leveraging a new architecture that operates within the FS latent space of StyleGAN, enhancing inpainting techniques, and introducing improved encoders for alignment and color transfer tasks.

The framework consists of four primary modules:

1. Embedding Module: This module generates representations in multiple latent spaces, crucially avoiding the shortcomings of low-dimensional generators. It employs mixing strategies to balance reconstruction quality with editability.
2. Alignment Module: Using specialized encoders, this module adapts hair shapes to target poses, addressing the common issue of pose mismatch. It employs a Shape Encoder and Shape Adaptor to refine hair segmentation masks.
3. Blending Module: Featuring a Blending Encoder, this module accurately transfers hair color by adjusting StyleGAN's latent space. It enhances detail retention via CLIP embeddings, improving over prior optimization approaches.
4. Post-Processing Module: This module refines image details, correcting deviations introduced in prior stages, and ensures high fidelity to the original identity of the input face.

Experimental Results

HairFastGAN demonstrates its effectiveness across various scenarios, including scenarios with significant pose differences. It achieves impressively low runtime, comparable to state-of-the-art encoder-based methods, such as HairCLIP, yet maintains a quality level rivaling optimization-based techniques like Barbershop and StyleYourHair.

Quantitative evaluation was undertaken using realism metrics like FID and $\text{FID}_{\text{CLIP}$, where HairFastGAN consistently outperformed or matched leading methods. Notably, it excelled in maintaining image fidelity across pose differences and in reconstruction tasks, suggesting robust internal representations.

Implications and Future Directions

The practical implications of HairFastGAN are substantial, particularly for applications requiring both speed and high-quality image editing, such as virtual try-ons and gaming. Theoretically, its architecture opens pathways for further research into encoder-based GAN applications, showcasing the potential for StyleGAN's latent spaces to be leveraged more efficiently.

Future avenues could explore enhancement in texture and detail handling, possibly incorporating text-based style manipulation through incremental adaptations of the Blending Module. Moreover, extending the framework for broader attribute editing beyond hair could position HairFastGAN as a versatile tool within the image editing landscape.

In summary, HairFastGAN presents a significant advancement in the field of hair transfer. By refining and integrating encoder strategies within GANs, this work paves the way for efficient solutions capable of tackling complex image editing tasks without the traditional trade-off between quality and speed.