GAN Prior Embedded Network for Blind Face Restoration in the Wild (2105.06070v1)

Abstract: Blind face restoration (BFR) from severely degraded face images in the wild is a very challenging problem. Due to the high illness of the problem and the complex unknown degradation, directly training a deep neural network (DNN) usually cannot lead to acceptable results. Existing generative adversarial network (GAN) based methods can produce better results but tend to generate over-smoothed restorations. In this work, we propose a new method by first learning a GAN for high-quality face image generation and embedding it into a U-shaped DNN as a prior decoder, then fine-tuning the GAN prior embedded DNN with a set of synthesized low-quality face images. The GAN blocks are designed to ensure that the latent code and noise input to the GAN can be respectively generated from the deep and shallow features of the DNN, controlling the global face structure, local face details and background of the reconstructed image. The proposed GAN prior embedded network (GPEN) is easy-to-implement, and it can generate visually photo-realistic results. Our experiments demonstrated that the proposed GPEN achieves significantly superior results to state-of-the-art BFR methods both quantitatively and qualitatively, especially for the restoration of severely degraded face images in the wild. The source code and models can be found at https://github.com/yangxy/GPEN.

• The paper introduces a novel method that embeds a pre-trained GAN within a U-shaped DNN to effectively restore severely degraded face images.
• It demonstrates superior performance with higher PSNR, lower FID, and improved LPIPS compared to traditional blind face restoration methods.
• Ablation studies confirm that fine-tuning the GAN prior is crucial for accurately reconstructing both global facial structures and fine local details.

GAN Prior Embedded Network for Blind Face Restoration in the Wild

The paper "GAN Prior Embedded Network for Blind Face Restoration in the Wild" presents a novel approach to restoring severely degraded face images without prior knowledge of image degradation specifics. This task, known as Blind Face Restoration (BFR), is particularly challenging due to the unpredictable nature of real-world image degradation. Traditional approaches relying on inverse problem-solving methods have often fallen short in producing high-quality restored images. This paper proposes an integration of Generative Adversarial Networks (GANs) with Deep Neural Networks (DNNs) to overcome these limitations, resulting in the GAN Prior Embedded Network (GPEN).

Methodology

The proposed GPEN framework exploits the generative capabilities of GANs by embedding a pre-trained GAN within a U-shaped DNN architecture. The key steps include:

• GAN Embedding: A GAN is first pre-trained to generate high-quality face images. This GAN is then incorporated into the DNN as a prior decoder, ensuring that the latent code and noise input are informed by both deep and shallow network features.
• Network Architecture: The U-shaped network structure employs GAN blocks akin to those in StyleGAN v2 for hierarchical feature reconstruction, thus allowing the model to preserve global structures while refining local details.
• Fine-Tuning: After embedding, the entire network undergoes fine-tuning using synthesized low- and high-quality face image pairs. This process refines the encoder's ability to map degraded images to the GAN's latent space, aligning the generation of detailed facial features with the target high-quality outputs.

Experimental Results

The GPEN model was evaluated extensively against state-of-the-art BFR methods using both synthetic datasets and real-world low-quality face images. Key findings include:

• Quantitative Assessment: GPEN demonstrated superior performance across multiple metrics such as Peak Signal-to-Noise Ratio (PSNR), Fréchet Inception Distance (FID), and Learned Perceptual Image Patch Similarity (LPIPS), indicating stronger ability to produce perceptually coherent and high-fidelity outputs.
• Qualitative Analysis: Visual comparisons revealed GPEN's effectiveness in restoring facial details while managing complex backgrounds, outperforming competing methods that often resulted in over-smoothed or artifact-laden outputs.
• Ablation Studies: Different network configurations were evaluated to identify the impact of each component. The findings highlighted the importance of fine-tuning the GAN for improved performance and fine-grained details.

Implications and Future Work

The integration of GAN priors within a U-shaped DNN architecture as demonstrated in GPEN offers significant improvements for BFR tasks, showing potential applicability beyond face restoration to other ill-posed tasks like face inpainting and colorization. However, GPEN in its current configuration produces a single high-quality output per low-quality input. Future work could explore extensions allowing multiple plausible restorations, potentially by incorporating style transfer mechanisms while maintaining background consistency.

Overall, GPEN sets a new benchmark for BFR by effectively leveraging the representational power of GANs and the structural learning capabilities of DNNs, marking a promising direction for future research in high-fidelity image restoration applications.