High-Quality Face Image SR Using Conditional Generative Adversarial Networks

Abstract: We propose a novel single face image super-resolution method, which named Face Conditional Generative Adversarial Network(FCGAN), based on boundary equilibrium generative adversarial networks. Without taking any facial prior information, our method can generate a high-resolution face image from a low-resolution one. Compared with existing studies, both our training and testing phases are end-to-end pipeline with little pre/post-processing. To enhance the convergence speed and strengthen feature propagation, skip-layer connection is further employed in the generative and discriminative networks. Extensive experiments demonstrate that our model achieves competitive performance compared with state-of-the-art models.

• The paper introduces FCGAN, a conditional GAN architecture that achieves state-of-the-art face super-resolution without relying on explicit facial priors.
• It employs a U-Net-like encoder-decoder with skip connections and a pixelwise L1 loss to effectively preserve high-frequency facial details.
• Empirical results on the CelebA dataset show a superior PSNR of 32.42 and robust performance across variations in pose, occlusion, and illumination.

High-Quality Face Image Super-Resolution with Conditional GANs: The FCGAN Architecture

Introduction

Face image super-resolution (SR), or face hallucination, is pivotal for applications in surveillance, facial recognition, and biometric authentication, where the high-frequency details in face images are critical. Traditional methods often incorporated elaborate facial priors—landmarks, shape models, or structure constraints—but this imposes complex pre/post-processing and may limit robustness to pose and illumination variation. This work introduces the Face Conditional Generative Adversarial Network (FCGAN), an end-to-end conditional GAN for face image SR, architected around BEGAN principles, with several methodological innovations and strong empirical results.

Network Architecture and Methodology

FCGAN extends the BEGAN framework, conditioning directly on the low-resolution (LR) face image instead of noise. Both generator and discriminator are implemented as encoder-decoders with skip-layer connections, following a U-Net-like topology. This architecture is designed to strengthen local feature propagation, prevent vanishing of fine details across network depth, and accelerate convergence, circumventing the need for facial priors or explicit alignment steps.

Figure 1: FCGAN pipeline and generator/discriminator architecture with stride, filter size, and channel count per layer.

The generator transforms an input $128 \times 128$ (after upsampling from true $32 \times 32$ LR images during preprocessing) through six downsampling and six upsampling residual blocks. Each convolution employs a $4 \times 4$ kernel with stride 2, enabling hierarchical feature extraction without any pooling layers. Discriminator architecture mirrors this, with input being the concatenation of a real or generated HR image with its conditional LR counterpart.

Pixelwise $L_1$ loss is used for both the generator and discriminator. The equilibrium algorithm from BEGAN dynamically balances the discriminator-generator game, leveraging global loss feedback to maintain stable adversarial training.

Loss Formulation and Training Criteria

The generator is optimized via a pixelwise $L_1$ loss between the generated HR image and target HR image:

$$\mathcal{L}_{G} = \left\| I^{HR} - G(I^{LR}) \right\|_1$$

The discriminator loss computes the difference in $L_1$ reconstruction error on real and generated pairs, integrated in an equilibrium-controlled objective:

$$\mathcal{L}_D = \mathcal{L}_{D_r} - k_t \mathcal{L}_{D_f}$$

where $k_t$ is updated as a feedback controller. This dynamic enhances convergence and mitigates mode collapse, supporting resilient adversarial dynamics.

Experimental Setup

The framework is evaluated on the CelebA dataset, comprising over 200,000 labeled face images with considerable diversity in pose, expression, and occlusion. Images are center-cropped and resized to $128 \times 128$ for HR ground truth. LR images are produced via bicubic downsampling to $32 \times 32$0, and then the network operates on upsampled $32 \times 32$1 LR approximations, as is customary for patch-based DCNNs.

Empirical Results

Empirical evaluation demonstrates FCGAN’s superiority over both interpolation baselines and leading CNN/GAN-based SR models, including FSRCNN [dong2016accelerating], LapSRN [LapSRN_CVPR2017], and pix2pix [pix2pix]. Visual inspection shows FCGAN reconstructions more faithfully preserve high-frequency texture and facial details, outperforming rivals particularly in challenging instances involving occlusion, pose, or illumination variations.

Figure 2: Comparison of face SR results across methods on CelebA. FCGAN output exhibits sharper and more realistic restoration compared to bicubic, pix2pix, FSRCNN, and LapSRN.

Detailed inspection of local regions further highlights FCGAN's ability to synthesize plausible fine details such as hair, glasses, and subtle facial features.

Figure 3: Local detail comparison: FCGAN generates finer, more artifact-free facial structures than alternative architectures.

Quantitatively, FCGAN achieves a PSNR of 32.42, surpassing FSRCNN (31.92), LapSRN (32.13), and pix2pix (30.27). These gains are accomplished without facial priors, alignment, or pre-training restrictions.

Numerous qualitative outputs illustrate the robustness of FCGAN to shifts in facial expression, pose, occlusion (e.g., glasses or hats), and lighting conditions.

Figure 4: Sample HR face images (4× upscaling) generated by FCGAN, demonstrating preservation of diverse facial attributes and resilience to occlusions.

Discussion and Implications

FCGAN’s architectural choices—U-Net style skip connections, pixelwise $32 \times 32$2 loss, and equilibrium-stabilized adversarial training—set a foundation for robust and efficient face SR. By explicitly conditioning on LR inputs and forgoing facial priors, FCGAN operates independently of explicit landmark localization or pre/post-processing, facilitating deployment in unconstrained real-world settings.

The model’s design opens avenues for extending SR to broader, non-facial classes, since the architecture does not exploit face-specific domain knowledge. Furthermore, the strong performance on qualitative and quantitative metrics indicates the viability of equilibrium GANs for controlled image-to-image translation tasks beyond SR.

One limitation is that, in the current implementation, training and inference operate on images upsampled to network input size ($32 \times 32$3), with future work poised to extend the receptive field to directly process and generate larger images from smaller LR inputs. Generalization to generic image classes also remains a promising direction.

Conclusion

FCGAN demonstrates a robust approach to face image SR, combining conditional adversarial training, skip-layer feature integration, and equilibrium objectives in an end-to-end, prior-free pipeline. The framework yields state-of-the-art PSNR and substantial perceptual improvements, without the crutch of facial landmarks or alignment procedures. With further scaling and adaptation, this approach is likely to impact a broader range of single image SR and conditional generative modeling applications.