SketchFaceGS: Sketch-Driven 3D Gaussian Face Generation

• SketchFaceGS is a real-time framework that converts single-view hand-drawn face sketches into fully parameterized 3D Gaussian head models.
• It uses a coarse-to-fine pipeline with Transformer-based UV feature prediction and StyleGAN2-based enhancement for high-frequency detail.
• The system supports interactive editing by localizing modifications in canonical UV space and fusing multi-scale generator features for photorealistic output.

Searching arXiv for the target paper and closely related sketch-to-face / geometry-aware work to ground the article in current arXiv records. tool call: arxiv_search.query({"^^^^^^^^3search_query3 OR ti:\3^^^^^^^^"Real-Time Sketch-Driven Face Editing and Generation with Gaussian Splatting\"","max_results":5,"sort_by":"relevance"}) tool call failed: unknown tool arxiv_search.query tool call: arxiv_search({"query":"SketchFaceGS Real-Time Sketch-Driven Face Editing and Generation with Gaussian Splatting","max_results":5}) SketchFaceGS denotes a sketch-driven framework for real-time generation and editing of photorealistic 3D Gaussian head models from ^^^^3 OR ti:\3^^^^D face sketches. It takes a single-view hand-drawn face sketch as structural input, optionally conditions on a reference portrait image for appearance, and outputs a fully parameterized 3D Gaussian head that supports arbitrary-view rendering and iterative sketch-based modification. The system is organized as a feed-forward coarse-to-fine pipeline: a Transformer-based UV feature-prediction stage reconstructs a coarse UV feature map from sketch and reference cues, a 3D UV feature enhancement stage injects high-frequency detail through a pre-trained GGHead prior, and an editing pathway performs localized, real-time modifications by combining UV Mask Fusion with layer-by-layer feature fusion in generator feature space (&&&^^^^3search_query3^^^^&&&).

^^^^3all:\3^^^^. Problem formulation and representational basis

SketchFaceGS addresses sketch-driven generation and editing of 3D heads represented with 3D Gaussian Splatting. The interaction model is explicit: a user provides a single-view hand-drawn face sketch, optionally supplies a reference portrait image, and receives a 3D Gaussian head that can be rendered in real time and edited by iteratively modifying the sketch. The central difficulty is that sketches are sparse, depth-ambiguous, texture-poor, stylized, and variable, while the target output is a dense, photorealistic, geometrically consistent 3D head under sub-second response constraints (&&&^^^^3search_query3^^^^&&&).

The framework adopts the standard 3DGS head representation. Each Gaussian carries position PRESERVED_PLACEHOLDER_^^^^3search_query3^^^^, scale PRESERVED_PLACEHOLDER_^^^^3all:\3^^^^, rotation quaternion PRESERVED_PLACEHOLDER_^^^^3 OR ti:\3^^^^, opacity $\alpha \in \mathbb{R}$, and color $\mathbf{c} \in \mathbb{R}^3$. The Gaussian ellipsoid is written as

$$G(\mathbf{x}) = e^{-\frac{1}{2}(\mathbf{x} - \boldsymbol{\mu})^\top \Sigma^{-1}(\mathbf{x} - \boldsymbol{\mu})}, \quad \Sigma = R S S^\top R^\top,$$

where $S$ is constructed from $\mathbf{s}$ and $R$ is derived from $\mathbf{q}$. Rendering proceeds by projection, depth ordering, and alpha compositing, with pixel color

PRESERVED_PLACEHOLDER_^^^^3all:\3search_query3^^^^

This explicit representation is operationally important because it supports real-time, high-resolution rendering, which is a prerequisite for interactive editing rather than merely offline synthesis (&&&^^^^3search_query3^^^^&&&).

A useful way to interpret the system is as a structured inversion problem: the sketch supplies underdetermined geometric evidence, the optional reference image supplies appearance evidence, and the generative prior constrains both to a plausible 3D Gaussian head manifold. This suggests that SketchFaceGS is less a direct sketch-to-geometry regressor than a prior-guided reconstruction-and-editing framework in UV space.

^^^^3 OR ti:\3^^^^. Coarse UV prediction from sketch and reference conditions

The generation pipeline begins with a coarse stage whose purpose is to predict a UV feature map that is geometrically consistent with the sketch and chromatically aligned with the reference image. The architecture is dual-branch and Transformer-based. One branch extracts geometry from the sketch; the other extracts appearance from the reference image. In each branch, image features act as keys and values in cross-attention, while the queries are per-vertex queries linked to the template head and global identity tokens encoding overall identity (&&&^^^^3search_query3^^^^&&&).

The template correspondence is mediated by FLAME. Transformer outputs are produced as per-vertex features on FLAME-like vertices and then projected to UV space through barycentric interpolation. This yields dense geometric UV features and dense appearance UV features in a common canonical parameterization. The use of a fixed template is not a minor implementation detail: it is the mechanism that makes sketch conditioning, Gaussian decoding, and later edit localization commensurate in a single canonical domain (&&&^^^^3search_query3^^^^&&&).

Geometry and appearance are then aligned by an AdaIN-based alignment network PRESERVED_PLACEHOLDER_^^^^3all:\3all:\3^^^^. In conceptual form, the alignment normalizes the geometric feature map and re-scales and shifts it using statistics derived from the appearance feature map:

PRESERVED_PLACEHOLDER_^^^^3all:\3 OR ti:\3^^^^

The aligned appearance map is concatenated with the geometric UV map to form the coarse UV feature map. The reported ablation that removes this translation network yields severe artifacts when sketch and reference identities differ, indicating that coarse fusion is not a simple concatenation problem but a conflict-resolution problem between structural and appearance cues (&&&^^^^3search_query3^^^^&&&).

This coarse stage is therefore responsible for establishing the global solution: face shape, major component placement, and approximate color/style. High-frequency realism is intentionally deferred.

3. UV feature enhancement and GGHead-based 3D Gaussian decoding

The fine stage addresses the main limitation of the coarse UV map: correct low-frequency structure without sufficient photorealistic detail. SketchFaceGS introduces a 3D UV feature enhancement module consisting of a U-Net and a pre-trained StyleGAN^^^^3 OR ti:\3^^^^-based GGHead prior. The U-Net takes the coarse UV feature map as input and outputs a global latent feature together with a pyramid of multi-resolution spatial features (&&&^^^^3search_query3^^^^&&&).

The global latent feature is concatenated with the global identity features from the geometry and appearance Transformer branches and mapped by an MLP into StyleGAN’s PRESERVED_PLACEHOLDER_^^^^3all:\33^^^^ space:

PRESERVED_PLACEHOLDER_^^^^3all:\3^^^^4

This latent code supplies global identity-aware control, while the U-Net’s spatial features provide local modulation. GGHead then decodes the combined conditioning into a final, detail-rich UV map encoding the Gaussian attributes. In effect, the model separates global identity/style control from spatially varying UV modulation, with both routed through a pre-trained 3D Gaussian generative prior (&&&^^^^3search_query3^^^^&&&).

The generator does not directly predict a rendered image; it predicts UV-encoded Gaussian parameters which are then assembled into 3D Gaussians and rendered by the 3DGS renderer. This design is what makes single-forward-pass synthesis compatible with real-time free-viewpoint rendering. The ablation without the enhancement module produces geometry that remains acceptable but textures that are oversmoothed and non-photorealistic, which clarifies the role of the fine stage: it is the source of high-frequency surface detail rather than coarse structural correctness (&&&^^^^3search_query3^^^^&&&).

A plausible implication is that the enhancement module plays a role analogous to latent-prior refinement in ^^^^3 OR ti:\3^^^^D sketch-to-face systems, but here the refinement target is a UV-coded 3D Gaussian head rather than a ^^^^3 OR ti:\3^^^^D RGB image.

4. Editing in canonical UV space

SketchFaceGS treats editing as a localized re-synthesis problem in generator feature space rather than as direct image editing or direct Gaussian replacement. The editing pipeline begins by comparing the original and edited sketches to compute a pixel-level difference mask in image space. This ^^^^3 OR ti:\3^^^^D mask is then back-projected into the 3D Gaussian scene. For each masked pixel, Gaussian contributions are accumulated using opacity and transmittance:

PRESERVED_PLACEHOLDER_^^^^3all:\3^^^^5

These weights quantify which Gaussians are responsible for the edited image region. After filtering negligible and back-facing contributions, the selected Gaussians are mapped through the GGHead-plus-FLAME correspondence into canonical UV coordinates, producing a binary UV mask PRESERVED_PLACEHOLDER_^^^^3all:\3^^^^6 (&&&^^^^3search_query3^^^^&&&).

The UV mask is resampled to the spatial resolution of each StyleGAN layer and used in layer-by-layer feature fusion. If PRESERVED_PLACEHOLDER_^^^^3all:\3^^^^7 denotes the intermediate feature map of the original head, PRESERVED_PLACEHOLDER_^^^^3all:\3^^^^8 the feature map generated from the edited sketch, and PRESERVED_PLACEHOLDER_^^^^3all:\3^^^^9 the mask at layer PRESERVED_PLACEHOLDER_^^^^3 OR ti:\3search_query3^^^^, then the fused feature is

PRESERVED_PLACEHOLDER_^^^^3 OR ti:\3all:\3^^^^

and the next-layer feature is updated as

PRESERVED_PLACEHOLDER_^^^^3 OR ti:\3 OR ti:\3^^^^

Because fusion is performed across multiple layers, unedited regions inherit the original multi-scale representation while edited regions are progressively resynthesized according to the new sketch. The stated motivation is to avoid the seams and shading inconsistencies associated with naive 3D Gaussian compositing (&&&^^^^3search_query3^^^^&&&).

This mechanism also makes the editing process view-independent. The mask is synthesized in canonical UV space rather than tied to the current camera view, so the same local edit remains coherent under free-viewpoint rendering. The ablation results indicate that this is not merely an elegant formulation: direct re-generation loses original identity and background, while direct 3D Gaussian compositing introduces visible seams and mismatched shading (&&&^^^^3search_query3^^^^&&&).

5. Training protocol, supervision, and empirical performance

Training is staged. The coarse UV prediction stage is trained on a multi-view dataset synthesized from GGHead. For each GGHead-generated 3D Gaussian head, the system renders multi-view images, derives corresponding sketches, and learns to reconstruct UV features from sketch-plus-reference input. The fine stage is trained on single-view FFHQ images and learns to map coarse UV features and identity codes to a high-quality StyleGAN latent and modulation parameters. The editing pipeline reuses the same networks. Across stages, the losses include pixelwise PRESERVED_PLACEHOLDER_^^^^3 OR ti:\33^^^^, perceptual loss, LPIPS, color-consistency losses, and adversarial loss for photorealism; Adam is used for optimization (&&&^^^^3search_query3^^^^&&&).

On the held-out sketch-based generation test set of ^^^^3all:\3search_query3search_query3^^^^ hand-drawn sketches, SketchFaceGS reports the best FID and KID among the listed baselines.

Method	FID PRESERVED_PLACEHOLDER_^^^^3 OR ti:\3^^^^4	KID PRESERVED_PLACEHOLDER_^^^^3 OR ti:\3^^^^5
S3D	96.^^^^3search_query33^^^^	4.5^^^^3search_query3^^^^ ± ^^^^3all:\3^^^^.^^^^3search_query3
Nano-LAM	^^^^3all:\333^^^^.7^^^^3 OR ti:\3^^^^	7.6^^^^3all:\3^^^^ ± ^^^^3search_query3^^^^.9
SketchFaceNeRF	94.94	4.53 ± ^^^^3search_query3^^^^.6
SketchFaceGS	9^^^^3 OR ti:\3^^^^.65	4.^^^^3search_query3search_query3^^^^ ± ^^^^3search_query3^^^^.4

For editing, the method reports both quality and speed advantages. The end-to-end latency is approximately PRESERVED_PLACEHOLDER_^^^^3 OR ti:\3^^^^6 seconds per edit, and rendering reaches up to PRESERVED_PLACEHOLDER_^^^^3 OR ti:\3^^^^7 FPS.

Method	FID / KID PRESERVED_PLACEHOLDER_^^^^3 OR ti:\3^^^^8	Time per edit / FPS
MagicQuill	46.48 / ^^^^3search_query3^^^^.78 ± ^^^^3search_query3^^^^.^^^^3 OR ti:\3^^^^	~6.^^^^3search_query3^^^^ / —
Nano-LAM	74.^^^^3 OR ti:\3^^^^6 / 3.^^^^3search_query3all:\3^^^^ ± ^^^^3search_query3^^^^.3	~^^^^3all:\3^^^^5.^^^^3search_query3 / ^^^^3 OR ti:\3SketchFaceGS Real-Time Sketch-Driven Face Editing and Generation with Gaussian Splatting3all:\3^^^^
SketchFaceNeRF	6^^^^3 OR ti:\3^^^^.49 / ^^^^3 OR ti:\3^^^^.65 ± ^^^^3search_query3^^^^.3	~^^^^3all:\3search_query3^^^^.^^^^3search_query3 / 4^^^^3 OR ti:\3^^^^
SketchFaceGS	44.6^^^^3search_query3^^^^ / ^^^^3search_query3^^^^.69 ± ^^^^3search_query3^^^^. OR ti:\3^^^^	~^^^^3search_query3^^^^.3 / ^^^^3 OR ti:\3^^^^43

The reported identity preservation outside the editing mask is likewise strongest for SketchFaceGS, with PSNR PRESERVED_PLACEHOLDER_^^^^3 OR ti:\3^^^^9 and SSIM $\alpha \in \mathbb{R}$^^^^3search_query3^^^^, compared with SketchFaceNeRF (no opt.) at PSNR $\alpha \in \mathbb{R}$^^^^3all:\3^^^^, SSIM $\alpha \in \mathbb{R}$^^^^3 OR ti:\3^^^^, and SketchFaceNeRF (opt.) at PSNR $\alpha \in \mathbb{R}$3, SSIM $\alpha \in \mathbb{R}$4 (&&&^^^^3search_query3^^^^&&&).

Ablations further specify which components are load-bearing. Removing the enhancement module degrades generation to FID $\alpha \in \mathbb{R}$5 and KID $\alpha \in \mathbb{R}$6; removing the translation network gives FID $\alpha \in \mathbb{R}$7 and KID $\alpha \in \mathbb{R}$8; replacing the Transformer appearance encoder with a convolutional appearance encoder gives FID $\alpha \in \mathbb{R}$9 and KID $\mathbf{c} \in \mathbb{R}^3$^^^^3search_query3^^^^; removing the global ID feature gives FID $\mathbf{c} \in \mathbb{R}^3$^^^^3all:\3^^^^ and KID $\mathbf{c} \in \mathbb{R}^3$^^^^3 OR ti:\3^^^^. In editing, re-generation yields FID $\mathbf{c} \in \mathbb{R}^3$3 and KID $\mathbf{c} \in \mathbb{R}^3$4, while 3D Gaussian compositing yields FID $\mathbf{c} \in \mathbb{R}^3$5 and KID $\mathbf{c} \in \mathbb{R}^3$6, both inferior to the full layer-wise fusion design (&&&^^^^3search_query3^^^^&&&).

6. Position within sketch-to-face research, limitations, and prospective extensions

SketchFaceGS occupies a distinct position relative to earlier sketch-to-face work. Older recognition-oriented methods such as the geometric-feature pipeline of "Geometric Feature Based Face-Sketch Recognition" focus on cross-modal matching through hand-crafted geometric ratios and K-NN, under frontal pose, neutral expression, and no-occlusion assumptions, rather than synthesis or editable 3D generation (&&&^^^^3all:\3^^^^5&&&). Later synthesis methods moved toward geometry-aware photo generation from sketches: "Unsupervised Facial Geometry Learning for Sketch to Photo Synthesis" introduces perceptual cycle-consistency and a geometry-aware discriminator in a CycleGAN-style unpaired setting, explicitly targeting correction of sketch geometric artifacts (&&&^^^^3all:\3^^^^6&&&). "Semantics-Preserving Sketch Embedding for Face Generation" instead uses a vector-sketch semantic interpretation module and a hybrid $\mathbf{c} \in \mathbb{R}^3$7-$\mathbf{c} \in \mathbb{R}^3$8 StyleGAN encoder to preserve semantic meaning and coarse geometry, especially for minority attributes such as hats, glasses, and earrings (&&&^^^^3all:\3^^^^7&&&).

Against that background, SketchFaceGS extends the design space in three directions at once: it moves from ^^^^3 OR ti:\3^^^^D RGB output to a 3D Gaussian head representation; it integrates a pre-trained GGHead prior rather than a conventional image-only generator; and it turns sketch control into an interactive editing interface through UV-space masking and generator-feature fusion (&&&^^^^3search_query3^^^^&&&). This suggests a broader shift in the field from sketch-conditioned image synthesis toward sketch-conditioned editable 3D head modeling.

The paper states several limitations. If sketch geometry and reference-face identity differ strongly, the system can exhibit slight identity shifts despite good color matching. Its generative capacity is bounded by the GGHead training distribution, so rare accessories, extreme occlusions, and out-of-distribution stylization may be handled poorly. The framework is limited to static heads and does not yet address facial animation. Very rough or ambiguous sketches remain inherently underconstrained. Proposed future directions include stronger identity-consistency losses, improved encoders for sketch-reference conflict resolution, better handling of rare accessories and occlusions, and extension to dynamic or animatable heads, potentially through animatable 3DGS-GAN backbones (&&&^^^^3search_query3^^^^&&&).

In technical summary, SketchFaceGS can be understood as a UV-centric, prior-driven 3D head synthesis system in which sketches specify structure, optional references specify appearance, GGHead supplies a high-capacity 3D Gaussian prior, and edits are localized by projecting ^^^^3 OR ti:\3^^^^D sketch changes into canonical UV space and compositing them in multi-scale generator feature space. The reported results indicate that this combination yields a practical compromise among geometric control, photorealism, 3D consistency, and interactive latency (&&&^^^^3search_query3^^^^&&&).