DeepFaceDrawing: Photorealistic Face Synthesis

• The paper introduces a framework that projects freehand sketches onto learned local shape manifolds to achieve robust, photorealistic face synthesis.
• It decomposes sketches into facial components, uses autoencoder embeddings for each part, and leverages a multi-scale adversarial network for high-resolution output.
• Experimental evaluations demonstrate improved fidelity metrics and user ratings, enabling flexible, interactive applications in digital art and forensic analysis.

DeepFaceDrawing is a deep generative framework for synthesizing photorealistic face images from freehand or rough sketches by leveraging local shape manifolds of face components, soft constraints, and a global image synthesis network. Unlike prior methods that require precise or professional sketches, DeepFaceDrawing is robust to incomplete, imprecise, or amateur input and provides fine-grained, interpretable user control over facial features. The system models the local shape space of key face components through learned autoencoder embeddings, projects input strokes onto these manifolds, and subsequently generates full-resolution images via a multi-component, multi-scale adversarial network architecture (Chen et al., 2020).

1. Shape Manifold Formulation and Component Decomposition

DeepFaceDrawing explicitly addresses the challenge of overfitting to hand-drawn freehand or non-canonical sketches by modeling the manifold of plausible face component shapes. The input face sketch $s$ is decomposed into five physically meaningful, spatially overlapping components: left eye ($c=1$), right eye ($c=2$), nose ($c=3$), mouth ($c=4$), and a remainder region covering hair, cheekbones, etc. ($c=5$). Each subimage is cropped to a fixed region, such as $128\times128$ for facial features or $512\times512$ for the overall mask. For each $c$, a component autoencoder $(E_c,D_c)$ with a latent dimension of 512 is trained by minimizing mean squared error reconstruction loss, thus implicitly learning the empirical shape manifold $c=1$0 of component $c=1$1.

Given a component sketch $c=1$2, its embedding $c=1$3 is projected onto $c=1$4 by locally reconstructing it as a convex combination of its $c=1$5 nearest neighbors in the latent space. This projection operation ensures synthesized features remain realistic and plausible even when the input strokes are highly noisy or nonstandard.

To enable further control, a blending parameter $c=1$6 per component allows the user to interpolate between faithfulness to the original (possibly out-of-manifold) sketch and its nearest-manifold projection.

2. Network Architectures

The system's architecture is modular and multi-tiered. Each component autoencoder ($c=1$7) follows a conv–res–deconv design. The encoder consists of five convolutional layers with stride 2 and increasing filter sizes, each followed by batch normalization and leaky ReLU activations, with intermediary residual blocks to increase representational robustness. The decoder mirrors this structure with transposed convolutions (deconvs), batch normalization, ReLU activations, and a $c=1$8 output layer.

The feature-mapping module $c=1$9 transforms each blended embedding $c=2$0 into a 32-channel spatial feature map $c=2$1, using a structure identical to the decoder except for output dimensions.

These five feature maps are spatially arranged and concatenated into a global tensor $c=2$2 used as input for the generator $c=2$3. $c=2$4 uses multiple convolutional layers and residual blocks, followed by symmetric deconvolution pathways culminating in RGB output. Synthesis is adversarially trained using three PatchGAN-style discriminators $c=2$5, each operating at a different spatial scale and taking both the generated image and its corresponding $c=2$6 as input.

3. Training Methodology

Training proceeds in two stages. First, each component autoencoder is trained individually on the sketch dataset by minimizing the componentwise reconstruction loss: $c=2$7 where optimization uses Adam with a learning rate of $c=2$8, batch size 16, and 50 epochs.

Second, with encoder weights fixed, a joint end-to-end training phase optimizes the feature mapping modules, the generator, and the discriminators. The overall generator loss combines multi-scale adversarial loss, an $c=2$9 pixel loss, and perceptual loss using VGG-19 features: $c=3$0 with perceptual loss defined as

$c=3$1

and hyperparameters $c=3$2, $c=3$3. The adversarial components use standard real-vs-fake binary cross-entropy. Training uses 17,000 paired samples from CelebAMask-HQ, processed to sketch–face pairs via digital filters.

4. Inference, User Control, and Practical Features

At test time, the user provides a $c=3$4 freehand sketch. Automated cropping parses the drawing into component subimages for each $c=3$5. Each is fed through its $c=3$6, projected onto its manifold, and blended with adjustable $c=3$7 to determine the final embedding. Feature maps are decoded by $c=3$8, spatially composed into $c=3$9, and the complete face is synthesized as $c=4$0.

A distinctive feature is the soft-constraint scheme: incomplete or imprecise strokes are pulled toward the plausible face manifold, preserving interpretability while avoiding implausible artifacts. The “shadow-guided sketching” aid overlays average training-set features below the user sketch in semi-transparent gray for each component, assisting non-expert users in correct feature placement based on the dataset distribution.

5. Experimental Results and Quantitative Analysis

Empirical validation demonstrates improved fidelity to sketches and realism of output compared to pix2pixHD and alternative image-to-image baselines. Quantitative measures include Fréchet Inception Distance (FID) and Inception Score (IS):

Method	FID	IS
pix2pixHD (baseline)	12.3	2.59
DeepFaceDrawing	10.2	1.82

User studies (60 participants, 22 sketches each) yielded higher faithfulness (Likert scale 4.85) and visual quality (5.50) ratings relative to global and local retrieval schemes (statistical significance $c=4$1). The architecture supports face morphing and copy-paste applications via direct manipulation of component embeddings.

6. Comparisons and Extensions

DeepFaceDrawing's local-to-global manifold construction contrasts with attribute-disentangled models such as the AD-VAE framework, which separates attribute-specific and attribute-agnostic latent factors (e.g., hair color, background) and uses sketch-driven UNet conditioning within a VAE framework (Guo et al., 2017). While AD-VAE emphasizes attribute disentanglement and style transfer via variational inference and KL regularization weighted towards attribute separation, DeepFaceDrawing's primary novelty is in elastic projection of rough, non-professional sketches onto the local shape manifold for robust, high-quality photorealistic generation.

7. Applications and Broader Impact

DeepFaceDrawing facilitates flexible, interactive face photo synthesis for applications in digital artwork, forensic sketch interpretation, entertainment, and data augmentation. The system's explicit manifold projection and soft user control enable both professional and amateur users to synthesize realistic faces from incomplete input, expand creative workflows, and serve as a testbed for research on shape-constrained generative models (Chen et al., 2020).