UV Mask Fusion in SketchFaceGS

Updated 5 July 2026

UV Mask Fusion is a UV-space, mask-driven mechanism in SketchFaceGS that fuses 2D sketch edits with multi-scale StyleGAN features for real-time 3D Gaussian head models.
It converts a user’s sketch into a canonical UV mask and performs layer-by-layer feature fusion to localize edits and preserve unedited regions with seamless blending and multi-view consistency.
Empirical evaluations show improved FID, KID, and identity preservation compared to re-generation and 3D Gaussian compositing strategies, highlighting its practical benefits.

Searching arXiv for the cited papers to ground the article and confirm metadata. I’ll look up the primary paper and closely related mask/UV papers on arXiv. UV Mask Fusion is a UV-space, mask-driven, layer-wise feature fusion mechanism introduced in SketchFaceGS for real-time generation and editing of photorealistic 3D Gaussian head models from 2D sketches (Li et al., 21 Apr 2026). In that formulation, a user’s sketch edit is converted into a canonical UV mask, and the mask is then used to fuse original and edited features inside a StyleGAN generator rather than by directly compositing 3D Gaussians. The resulting operation is designed to localize edits, preserve unedited regions, and maintain seamless blending and multi-view consistency. Related work uses UV-space masks and mask-guided fusion in other ways—such as UV-based de-occlusion and segmentation augmentation, UV-domain generative conditioning, or UV-domain pressure reconstruction—but the explicit term “UV Mask Fusion” is most precisely defined by the SketchFaceGS editing pipeline (Li et al., 21 Apr 2026).

1. Definition and operational scope

In SketchFaceGS, UV Mask Fusion is explicitly defined as a two-part mechanism: UV Mask Synthesis, which converts a user’s 2D sketch edit into a canonical UV mask $\mathbf{M}_{\text{UV}}$ , and Layer-by-Layer Feature Fusion, which uses per-layer resampled masks $\mathbf{M}_{\text{UV}}^{(k)}$ to fuse original and edited StyleGAN feature maps throughout the generator (Li et al., 21 Apr 2026).

The method operates within a sketch-driven pipeline whose inputs are a single-view hand-drawn facial sketch $I_{\text{sketch}}$ and an optional appearance reference image $I_{\text{ref}}$ . The intermediate representation is a UV feature map encoding geometry and appearance in canonical UV space, and the final output is a photorealistic, editable 3D Gaussian head rendered in real time via 3D Gaussian Splatting (3DGS) (Li et al., 21 Apr 2026). SketchFaceGS does not directly regress Gaussians from the sketch; instead, it predicts a UV map from which a pre-trained StyleGAN-based generator, GGHead, outputs Gaussian attributes. This UV parametrization is the basis of the editing method.

A common misconception is that UV Mask Fusion is a direct replacement of Gaussians in 3DGS space. In SketchFaceGS, it is realized as a UV-mask–driven, layer-by-layer feature fusion inside the StyleGAN generator rather than direct compositing in 3DGS space (Li et al., 21 Apr 2026). The distinction is central: the editable object is the UV feature stream, while the 3D Gaussian head is decoded from the fused UV representation.

2. UV-space representation and generator interface

The UV-space representation in SketchFaceGS follows GGHead and binds 3D Gaussians to a canonical template head, described as a FLAME-style mesh, with Gaussian attributes parameterized as UV maps (Li et al., 21 Apr 2026). Conceptually, the template mesh has vertices $v_i \in \mathbb{R}^3$ with associated UV coordinates $(u_i, v_i)$ , and per-vertex geometry and appearance features are projected to dense UV feature maps by barycentric interpolation.

The 3D Gaussian head is represented as a set of Gaussians with attributes

$\mu \in \mathbb{R}^3,\quad s \in \mathbb{R}^3,\quad q \in \mathbb{R}^4,\quad \alpha \in \mathbb{R},\quad c \in \mathbb{R}^3,$

and a rendered pixel color is written as

$C = \sum_{i=1}^{N} c_i \alpha_i \prod_{j=1}^{i-1} (1 - \alpha_j).$

The coarse stage reconstructs a geometrically consistent UV feature map from the sketch and optional reference, while the fine stage refines it with high-frequency, photorealistic detail (Li et al., 21 Apr 2026).

The fine stage uses a U-Net to produce a global latent vector, a pyramid of multi-resolution spatial modulation features, and a StyleGAN latent code

$\mathcal{W} = \mathrm{MLP}\bigl(\mathrm{concat}(F_{\text{latent}}, F_{\text{ID-G}}, F_{\text{ID-A}})\bigr),$

which modulates all StyleGAN layers. Feeding this into the pre-trained GGHead StyleGAN2 backbone yields a final UV map whose channels encode all 3DGS attributes—position, scale, rotation, opacity, and color—per UV texel (Li et al., 21 Apr 2026).

This architectural choice makes UV space both canonical and semantically organized. A plausible implication is that edit locality is easier to express in UV coordinates than in directly rendered image space or in raw Gaussian attribute space, because the generator can use a stable atlas during feature synthesis.

3. UV mask synthesis from 2D edits

In SketchFaceGS, a UV mask is a binary map in UV space,

$\mathbf{M}_{\text{UV}} \in \{0,1\}^{H_{\text{UV}} \times W_{\text{UV}}},$

where 1 denotes an edited region and 0 an unedited region (Li et al., 21 Apr 2026). The mask is derived from image-space edits but is ultimately defined over the canonical UV domain.

The synthesis procedure begins with 2D edit detection. Users draw or erase strokes on the sketch, and differences between the current sketch and the previous one, followed by dilation, yield a 2D pixel mask $\mathbf{M}_{\text{UV}}^{(k)}$ 0 in image coordinates (Li et al., 21 Apr 2026). The method then performs back-projection to 3D Gaussians. For pixels $\mathbf{M}_{\text{UV}}^{(k)}$ 1, it accumulates an influence weight for each Gaussian $\mathbf{M}_{\text{UV}}^{(k)}$ 2,

$\mathbf{M}_{\text{UV}}^{(k)}$ 3

where $\mathbf{M}_{\text{UV}}^{(k)}$ 4 is Gaussian $\mathbf{M}_{\text{UV}}^{(k)}$ 5’s opacity at pixel $\mathbf{M}_{\text{UV}}^{(k)}$ 6 and $\mathbf{M}_{\text{UV}}^{(k)}$ 7 is the transmittance up to Gaussian $\mathbf{M}_{\text{UV}}^{(k)}$ 8 on the ray for $\mathbf{M}_{\text{UV}}^{(k)}$ 9 (Li et al., 21 Apr 2026). Only Gaussians with sufficiently large $I_{\text{sketch}}$ 0 are considered to belong to the edited region, and back-face leakage is reduced by discarding Gaussians that contribute little in the current view.

Selected Gaussians are then mapped to UV space through the FLAME template embedding used by GGHead. Marking the corresponding UV texels yields the binary UV mask. For each StyleGAN layer $I_{\text{sketch}}$ 1, the canonical mask is downsampled to the layer resolution, producing

$I_{\text{sketch}}$ 2

so interpolation introduces soft boundaries even though the canonical mask itself is binary (Li et al., 21 Apr 2026).

This synthesis stage provides the localization signal on which the entire editing method depends. It identifies which UV regions, and therefore which Gaussians, are subject to modification while leaving the rest of the canonical atlas untouched.

4. Layer-by-layer feature fusion inside StyleGAN

The core fusion rule in SketchFaceGS is applied at every StyleGAN layer. Let $I_{\text{sketch}}$ 3 denote the intermediate feature map for the original head at layer $I_{\text{sketch}}$ 4, $I_{\text{sketch}}$ 5 the corresponding feature map for the newly edited head, and $I_{\text{sketch}}$ 6 the UV mask resampled to that layer’s resolution. The fused feature map is defined as

$I_{\text{sketch}}$ 7

followed by

$I_{\text{sketch}}$ 8

This procedure is repeated from coarse to fine layers (Li et al., 21 Apr 2026).

The layer-wise structure is inseparable from UV Mask Fusion itself. At low-resolution layers, the masks enforce geometry and coarse appearance consistency; at high-resolution layers, they correct fine texture details and suppress seams (Li et al., 21 Apr 2026). The generator can therefore be interpreted as synthesizing a new UV map conditioned on a hybrid feature stream: original features outside the mask and edited features inside, merged at all scales.

The motivation is explicitly contrastive. According to the SketchFaceGS description, direct editing in 3D Gaussian space leads to visible seams at edit boundaries, inconsistent shading or texture across views, identity drift, error accumulation over multiple edits, and expensive optimization in prior tri-plane or NeRF-based work (Li et al., 21 Apr 2026). UV Mask Fusion is intended to solve these issues by operating in UV space and by fusing features inside the StyleGAN hierarchy, allowing the generator to heal boundaries and maintain coherence.

A second misconception is that the fusion is a single-level blend. In fact, SketchFaceGS emphasizes semantic-scale-aware fusion: coarse layers affect large-context geometry and appearance, while fine layers affect local texture and boundary quality (Li et al., 21 Apr 2026).

5. Interaction with 3D Gaussian attributes, editing behavior, and empirical results

After layer-wise fusion, the generator outputs a final fused UV map encoding, per UV texel, the Gaussian attributes $I_{\text{sketch}}$ 9 (Li et al., 21 Apr 2026). Through the GGHead UV-to-Gaussian mapping, each UV location corresponds to one or multiple Gaussians on the head. Where $I_{\text{ref}}$ 0, the fused UV map uses edited features at all levels, enabling changes in Gaussian positions, scales, and rotations as well as color and opacity. The examples given include facial contour or hair volume for geometry edits, and hair color, makeup, glasses, or hats for appearance edits (Li et al., 21 Apr 2026).

Because the mechanism operates through the generator, the resulting Gaussians are described as globally coherent, with plausible shading, reflectance, and occlusion across views, and multi-view consistency follows from the underlying 3DGS rendering (Li et al., 21 Apr 2026). The paper also highlights continuous, multi-step edits: Fig. 8 shows five sequential editing steps—hair, contour, eyebrows, mouth, and eyes—and UV Mask Fusion is said to prevent error accumulation and identity drift because unmasked regions always inherit original features.

The operational efficiency is tied to the simplicity of the mask operations. Computation of $I_{\text{ref}}$ 1 is a sum over masked pixels, mask rasterization and resizing are standard GPU operations, and fusion itself is one or two per-layer element-wise multiply-adds (Li et al., 21 Apr 2026). Combined with the feed-forward architecture, this yields an end-to-end edit latency of approximately $I_{\text{ref}}$ 2 and 3DGS rendering at up to 243 FPS (Li et al., 21 Apr 2026).

Quantitatively, the editing pipeline is reported to achieve the best FID and KID among baselines in Table 5, with FID 44.60 and KID $I_{\text{ref}}$ 3, while identity preservation in unedited regions, measured with PSNR and SSIM, surpasses SketchFaceNeRF (Li et al., 21 Apr 2026). This suggests that the principal empirical benefit of UV Mask Fusion is not only edit locality but also stability of unedited content under repeated edits.

The ablation study in SketchFaceGS compares three strategies: Re-generation, 3D Gaussian compositing, and the Full Model (UV Mask Fusion + layer-wise feature fusion) (Li et al., 21 Apr 2026). The comparison is qualitative and quantitative: re-generation fails to preserve identity and unedited context, 3D Gaussian compositing introduces visible seams and mismatches around edit boundaries, and the full model yields coherent shape and appearance with seamless transitions.

Strategy	FID	KID
Re-generation	88.15	$I_{\text{ref}}$ 4
3DGS compositing	68.42	$I_{\text{ref}}$ 5
Full model	44.60	$I_{\text{ref}}$ 6

These results support the paper’s explicit conclusion that layer-wise UV feature fusion significantly outperforms both naive strategies (Li et al., 21 Apr 2026).

The limitations discussed for SketchFaceGS also delimit UV Mask Fusion itself. The mechanism depends on the generative prior provided by GGHead, so rare accessories, extreme occlusions, and out-of-distribution sketches may not be faithfully represented even with accurate masks. Large sketch–reference discrepancies may cause identity shifts in heavily reshaped regions, and the current method targets static heads rather than animated ones (Li et al., 21 Apr 2026). Proposed future directions include identity-preservation losses on unedited UV regions, more semantic UV masks such as per-part segmentation, and enhanced UV mask synthesis for extreme deformations and very fine structures.

Related work uses UV-space masks and mask-guided fusion in other technical senses. Mask-FPAN uses a 3D morphable face model combined with a UV GAN to improve robustness of 2D face parsing under occlusions and large pose variation, and its UVM synthesizes images and corresponding parsing masks by completing occluded UV maps and warping completed UV masks back to image space (Li et al., 2022). UVMap-ID fuses a reference portrait and text prompt into a canonical SMPL UV layout through decoupled cross-attention inside a diffusion model, treating identity and text as separate conditioning streams for personalized UV texture generation (Wang et al., 2024). EgoPressDiff defines pressure on the UV domain of the MANO hand surface and uses a binary UV mask $I_{\text{ref}}$ 7 to upweight reconstruction loss on the hand region during multimodal video diffusion (Zeng et al., 5 Jun 2026). MaSaFusion, although formulated in image space rather than UV space, offers a closely related mask-guided fusion principle by routing self-attention differently inside and outside a user-provided mask, and its description notes that the same conceptual structure can be transplanted to UV-space almost directly (Li et al., 2024).

Taken together, these formulations do not define a single uniform method called UV Mask Fusion. Rather, they indicate a family of UV-domain strategies in which canonical UV parameterization, binary or soft masks, and feature-level fusion are used to preserve locality, enforce structural correspondence, or guide multimodal interaction. Within that broader family, the most specific and technically complete definition of UV Mask Fusion remains the SketchFaceGS mechanism for mask synthesis and layer-by-layer feature fusion in UV space (Li et al., 21 Apr 2026).