FaceMat: Face Material & Matting Techniques

• FaceMat is a collection of face-specific decomposition methods that separate images into intrinsic material layers and soft alpha mattes.
• It employs inverse rendering and UV alignment to extract reusable layers such as bare skin, makeup, and occluders for realistic face transformation.
• Key applications include robust makeup transfer, occlusion-aware filters, and relightable avatar reconstruction using single unconstrained images.

Searching arXiv for FaceMat and closely related papers to ground the encyclopedia entry. FaceMat denotes a cluster of face-specific matting and material-estimation formulations rather than a single universally fixed definition. In one line of work, the term refers to face material/matting: the decomposition of facial appearance into physically meaningful layers such as bare skin, makeup, and a matte in canonical UV space, enabling reusable 3D facial materials from a single portrait (Yang et al., 2023). In another, it names a trimap-free, uncertainty-guided framework for occlusion-aware face matting, where facial skin is treated as foreground and occluders as background in order to support robust face transformation under hands, hair, accessories, and motion blur (Cho et al., 5 Aug 2025). Related work on relightable monocular face reconstruction further aligns with FaceMat-style goals by directly estimating diffuse albedo, specular albedo, normals, and geometry as rendering-ready assets from a single unconstrained image (Galanakis et al., 2023). Taken together, these usages place FaceMat at the intersection of image matting, inverse rendering, 3D morphable modeling, UV-space facial materials, and occlusion-aware compositing.

1. Conceptual scope and mathematical formulations

FaceMat appears in the literature with two principal meanings. The first is face material/matting, in which the objective is to separate facial appearance into intrinsic material layers so that cosmetics can be modeled as a reusable overlay on bare skin. The second is face matting in the narrower compositing sense, where the task is to estimate a soft alpha matte that isolates facial skin from occluders and background for downstream transformation and recompositing (Yang et al., 2023, Cho et al., 5 Aug 2025).

The compositing formulation used by the occlusion-aware matting literature follows the standard equation

$I_i = \alpha_i F_i + (1 - \alpha_i) B_i,$

where $\alpha \in [0,1]$ is a per-pixel alpha matte. In the face-specific reformulation introduced by FaceMat, facial skin, including the eyes, nose, and mouth region, is treated as foreground, while occlusions are treated as background. This makes $\alpha$ directly usable for occlusion-preserving face transformation, with output compositing written as

$I_{\text{out}} = \alpha T + (1 - \alpha) O,$

where $T$ is the transformed face layer and $O$ is the original occluder/background layer (Cho et al., 5 Aug 2025).

The material/matting formulation used for makeup extraction is different. There, the central equation is the decomposition of diffuse albedo in UV space:

$$A_d(u) = (1 - \alpha(u)) \cdot A_{\text{skin}}(u) + \alpha(u) \cdot A_{\text{makeup}}(u).$$

Here, $\alpha(u)$ is not a foreground-background matte but a material matte that controls blending between two intrinsic facial materials, bare skin and makeup, at each UV coordinate. The paper explicitly distinguishes this from foreground-background matting (Yang et al., 2023).

A related but broader formulation appears in monocular relightable avatar reconstruction. FitDiff estimates 3D facial shape $S$, diffuse albedo $A_d(u,v)$, specular albedo $\alpha \in [0,1]$0, normal map $\alpha \in [0,1]$1, and illumination parameters $\alpha \in [0,1]$2 from a single unconstrained image. This does not define FaceMat directly, but it is explicitly described as aligned with “FaceMat-style facial materials estimation,” because it recovers the core facial material suite and compatible geometry required for relighting in standard rendering engines (Galanakis et al., 2023).

2. Face material/matting for makeup-layer extraction

In the work “Makeup Extraction of 3D Representation via Illumination-Aware Image Decomposition,” FaceMat refers to a layered facial material representation derived from a single makeup portrait (Yang et al., 2023). The goal is to extract a reusable, illumination-free makeup layer for a 3D facial model despite unknown lighting and occlusions. The pipeline consists of three stages: regression-based facial inverse rendering with a 3DMM prior to obtain coarse materials in UV space; refinement by inpainting and optimization to address missing pixels caused by occlusions; and decomposition of diffuse albedo into bare skin, makeup, and an alpha matte (Yang et al., 2023).

The inverse-rendering model separates illumination from intrinsic materials through the appearance equation

$\alpha \in [0,1]$3

where $\alpha \in [0,1]$4 is diffuse shading, $\alpha \in [0,1]$5 is diffuse albedo, and $\alpha \in [0,1]$6 is specular reflectance. Diffuse shading is modeled with low-order spherical harmonics,

$\alpha \in [0,1]$7

and specular reflectance uses a Blinn–Phong approximation,

$\alpha \in [0,1]$8

The inverse problem infers geometry, diffuse albedo, specular parameters, and lighting so that a differentiable renderer reproduces the input portrait while keeping materials physically plausible and disentangled from lighting (Yang et al., 2023).

The FaceMat step proper is the UV-space decomposition of diffuse albedo:

$\alpha \in [0,1]$9

Bare skin is regularized by a morphable skin albedo model or a de-makeup/de-lighting prior such as BareSkinNet; the alpha matte is constrained to be sparse and localized to typical cosmetic regions; and the makeup albedo is regularized to remain consistent with cosmetic pigment statistics and bounded saturation/value shifts relative to skin (Yang et al., 2023).

Because the decomposition is performed on illumination-free diffuse albedo rather than on the raw portrait, shading and specular effects are prevented from leaking into $\alpha$0 or $\alpha$1. This illumination-aware disentanglement is central to the paper’s definition of FaceMat, since it makes the matte physically meaningful and portable across faces and lighting conditions (Yang et al., 2023).

UV alignment is a structural feature of the method. All extracted makeup layers and mattes are defined in a canonical UV coordinate system, which enables aggregation across subjects and supports the construction of a large-scale makeup dataset and a parametric makeup model,

$\alpha$2

A plausible implication is that this UV-canonicalization converts makeup from a view-dependent image attribute into a reusable face material asset, suitable for both 2D and 3D editing workflows.

3. FaceMat as trimap-free occlusion-aware face matting

In “Uncertainty-Guided Face Matting for Occlusion-Aware Face Transformation,” FaceMat is a specific framework for predicting alpha mattes under complex facial occlusions without trimaps or auxiliary segmentation at inference time (Cho et al., 5 Aug 2025). The motivation is that face filters, stylization systems, and face swapping degrade under occlusions because hard masks cannot represent partial transparency, soft boundaries, or fine structures such as hair strands and motion blur.

The framework introduces a two-stage teacher–student pipeline. In Stage 1, a teacher model based on RVM with a MobileNetV3 encoder jointly predicts an alpha matte and per-pixel uncertainty. The teacher uses a Gaussian likelihood formulation with per-pixel negative log-likelihood. For a target scalar field with mean $\alpha$3 and variance $\alpha$4, the per-pixel NLL is

$\alpha$5

FaceMat applies this to both the alpha and uncertainty predictions with an additional factor $\alpha$6, yielding $\alpha$7 and $\alpha$8. The teacher’s full objective is

$\alpha$9

with $I_{\text{out}} = \alpha T + (1 - \alpha) O,$0, $I_{\text{out}} = \alpha T + (1 - \alpha) O,$1, and $I_{\text{out}} = \alpha T + (1 - \alpha) O,$2 masked to the trimap unknown region (Cho et al., 5 Aug 2025).

In Stage 2, a trimap-free student predicts a deterministic alpha matte and is trained with uncertainty-guided knowledge distillation. The uncertainty-weighted regression loss is

$I_{\text{out}} = \alpha T + (1 - \alpha) O,$3

with spatial weights

$I_{\text{out}} = \alpha T + (1 - \alpha) O,$4

where $I_{\text{out}} = \alpha T + (1 - \alpha) O,$5 and $I_{\text{out}} = \alpha T + (1 - \alpha) O,$6. The stage-2 objective is

$I_{\text{out}} = \alpha T + (1 - \alpha) O,$7

The teacher is updated as an exponential moving average of the student parameters (Cho et al., 5 Aug 2025).

A distinctive conceptual move is the “skin-as-foreground, occlusions-as-background” definition. This simplifies downstream blending: facial effects are applied only on the skin layer, while original occlusions remain on top after recomposition. The paper positions this as a face-specific alternative to both trimap-based matting and hard segmentation, with the explicit claim that trimap-free inference makes the method suited to real-time short-form video applications (Cho et al., 5 Aug 2025).

The framework is accompanied by CelebAMat, a synthetic dataset constructed from CelebAMask-HQ faces and occluders sourced from SIMD, AM2k, HIU, and DTD. Occluders are composited with random size, orientation, and position, and soft alpha ground truth is produced by blending face and occlusion masks with boundary blurring. Motion-aware adjacent frames are synthesized by affine transforms, resizing, flips, color jitter, and random pauses (Cho et al., 5 Aug 2025).

4. Relation to relightable facial material estimation

FitDiff does not use the name FaceMat as its primary method label, but it is explicitly described as aligned with “FaceMat-style facial materials estimation” because it estimates the core outputs associated with that agenda from a single in-the-wild image (Galanakis et al., 2023). Given an unconstrained face image $I_{\text{out}} = \alpha T + (1 - \alpha) O,$8, the method estimates 3D facial shape $I_{\text{out}} = \alpha T + (1 - \alpha) O,$9, diffuse albedo $T$0, specular albedo $T$1, normal map $T$2, illumination parameters $T$3, and expression parameters. The outputs are relightable “as-is” because the mesh uses the LSFM topology and fixed UV layout, and the maps align to common PBR or Blinn–Phong shaders (Galanakis et al., 2023).

The model is a multi-modal latent diffusion model. A single latent vector

$T$4

concatenates material, shape, and illumination codes, with $T$5, $T$6, and $T$7. Geometry is produced by the LSFM 3DMM:

$T$8

while a branched VQGAN jointly encodes and decodes the three UV maps (Galanakis et al., 2023).

Identity preservation is built into the diffusion model through ArcFace conditioning. A global embedding $T$9 and intermediate activation maps $O$0, $O$1, and $O$2 are injected into the UNet by SPADE layers. The denoising network predicts injected noise in a latent-space DDPM with training loss

$O$3

Auxiliary identity cosine, identity perceptual, and vertex losses are computed after decoding an estimate of $O$4 and rendering it differentiably. The total loss is

$O$5

At sampling time, FitDiff turns reverse diffusion into identity-aware fitting by adding gradient guidance from rendered frames, with guidance energy

$O$6

and update

$O$7

The reported settings are $O$8, $O$9, $$A_d(u) = (1 - \alpha(u)) \cdot A_{\text{skin}}(u) + \alpha(u) \cdot A_{\text{makeup}}(u).$$0, and $$A_d(u) = (1 - \alpha(u)) \cdot A_{\text{skin}}(u) + \alpha(u) \cdot A_{\text{makeup}}(u).$$1, with DDIM sampling using $$A_d(u) = (1 - \alpha(u)) \cdot A_{\text{skin}}(u) + \alpha(u) \cdot A_{\text{makeup}}(u).$$2 and runtime of approximately $$A_d(u) = (1 - \alpha(u)) \cdot A_{\text{skin}}(u) + \alpha(u) \cdot A_{\text{makeup}}(u).$$3 on an NVIDIA Tesla V100-32GB (Galanakis et al., 2023).

Its rendering model uses a standard Blinn–Phong shader,

$$A_d(u) = (1 - \alpha(u)) \cdot A_{\text{skin}}(u) + \alpha(u) \cdot A_{\text{makeup}}(u).$$4

Only albedo maps and normals are estimated; there is no per-pixel roughness. This keeps the material parameterization simple and compatible with widely used engines (Galanakis et al., 2023). A plausible implication is that FitDiff occupies the adjacent space between FaceMat as layered intrinsic decomposition and FaceMat as practical facial material asset generation.

5. Historical antecedents and methodological context

Before FaceMat became associated with uncertainty-guided facial alpha prediction or UV-layered makeup materials, portrait matting research had already established the practical importance of fast alpha estimation for face-centric editing. “Fast Deep Matting for Portrait Animation on Mobile Phone” proposes an automatic deep matting approach for portrait images that runs on mobile devices in real time, using a light dense network to predict a coarse binary mask and a feathering block to convert that mask into an alpha matte (Zhu et al., 2017).

The method starts from the standard matting equation

$$A_d(u) = (1 - \alpha(u)) \cdot A_{\text{skin}}(u) + \alpha(u) \cdot A_{\text{makeup}}(u).$$5

and addresses its ill-posedness by leveraging portrait priors through a lightweight semantic segmentation network. The segmentation block contains an initial downsampling stage, a dilated dense block, a final classifier, and bilinear upsampling. The paper states that the light dense network has 6 convolutional layers and 1 max-pooling layer (Zhu et al., 2017).

Its main technical novelty is the feathering block, which predicts local linear coefficients to transform foreground and background score maps into an alpha matte. Within a sliding window $$A_d(u) = (1 - \alpha(u)) \cdot A_{\text{skin}}(u) + \alpha(u) \cdot A_{\text{makeup}}(u).$$6,

$$A_d(u) = (1 - \alpha(u)) \cdot A_{\text{skin}}(u) + \alpha(u) \cdot A_{\text{makeup}}(u).$$7

The corresponding color-domain form is

$$A_d(u) = (1 - \alpha(u)) \cdot A_{\text{skin}}(u) + \alpha(u) \cdot A_{\text{makeup}}(u).$$8

and the gradient relationship

$$A_d(u) = (1 - \alpha(u)) \cdot A_{\text{skin}}(u) + \alpha(u) \cdot A_{\text{makeup}}(u).$$9

is used to motivate the edge-preserving behavior of the filter. The full loss is

$\alpha(u)$0

The paper reports mobile runtime of $\alpha(u)$1 on CPU and $\alpha(u)$2 on GPU with Android RenderScript optimization, enabling real-time portrait animation at approximately 15 fps (Zhu et al., 2017).

Although this work is not called FaceMat in the 2025 sense, it provides a clear methodological precursor: automatic portrait alpha estimation without user interaction, optimized for mobile deployment and downstream visual effects. This suggests that later face-specific matting frameworks inherit a long-standing tension between fine-boundary quality, semantic correctness, and real-time constraints.

6. Datasets, supervision, evaluation, and limitations

The FaceMat-related literature uses markedly different forms of supervision. The makeup-oriented material decomposition pipeline relies on regression-based inverse rendering, UV-domain inpainting, and optimization, with portrait datasets such as FFHQ providing in-the-wild inputs; BiSeNet is used for face segmentation masks, VGG features for perceptual loss, and differentiable rasterization such as nvdiffrefrast for rendering (Yang et al., 2023). The uncertainty-guided FaceMat framework instead uses synthetic supervision from CelebAMat, which contains 24,602 training images and 716 test images derived from CelebAMask-HQ, together with occluders from SIMD, AM2k, HIU, and DTD (Cho et al., 5 Aug 2025). FitDiff uses 9k manually selected CelebA-HQ images paired with pseudo ground-truth materials and geometry obtained by fitting FitMe and LSFM 3DMM, and explicitly notes that no FaceMat dataset is used (Galanakis et al., 2023).

Evaluation protocols also differ. The uncertainty-guided FaceMat framework is evaluated with matting metrics in the unknown region—MSE, SAD, Grad, and Conn—and segmentation metrics including IoU and Accuracy. For the Stage 2 uncertainty-guided student on CelebAMat, representative numbers are reported as MSE $\alpha(u)$3, SAD $\alpha(u)$4, Conn $\alpha(u)$5, IoU $\alpha(u)$6, and Accuracy $\alpha(u)$7. On RealOcc, FaceMat reports IoU $\alpha(u)$8 versus $\alpha(u)$9 for RVM, Accuracy $S$0 versus $S$1, and Recall $S$2 versus $S$3 (Cho et al., 5 Aug 2025).

FitDiff is evaluated on identity preservation, reflectance accuracy, and shape reconstruction. On the Light Stage dataset, the reported reflectance metrics are diffuse MSE $S$4, PSNR $S$5, SSIM $S$6; specular MSE $S$7, PSNR $S$8, SSIM $S$9; and normals MSE $A_d(u,v)$0, PSNR $A_d(u,v)$1, SSIM $A_d(u,v)$2. The method is also reported to outperform prior monocular 3D face reconstruction methods on identity similarity and separability under diverse in-the-wild conditions (Galanakis et al., 2023). The earlier mobile portrait matting work evaluates Gradient Error and MSE, with LDN+FB reported at CPU $A_d(u,v)$3, GPU $A_d(u,v)$4, Gradient Error $A_d(u,v)$5, and MSE $A_d(u,v)$6 on the Head Matting Dataset (Zhu et al., 2017).

Several limitations recur across the literature. In material decomposition, extreme or highly complex lighting can challenge spherical harmonics estimation and specular separation, while heavy occlusions reduce evidence for inpainting and can smooth away fine cosmetic details (Yang et al., 2023). In uncertainty-guided face matting, extreme occlusions, severe motion blur, and transparent or reflective accessories remain difficult, and the synthetic-to-real gap persists (Cho et al., 5 Aug 2025). In FitDiff, ambiguity between illumination and skin tone can leak into diffuse albedo or illumination parameters; no per-pixel roughness or microfacet parameters are estimated; and the assets remain face-centric rather than explicitly modeling hair, beards, or accessories (Galanakis et al., 2023). The mobile portrait matting precursor reports failure to distinguish tiny hair details because the input image is downsampled (Zhu et al., 2017).

A recurring misconception is to treat all FaceMat work as foreground-background alpha estimation. The makeup-oriented literature explicitly defines FaceMat as material matting between intrinsic facial layers rather than subject-background separation (Yang et al., 2023). Conversely, the 2025 FaceMat framework is not a 3D facial material estimator but a trimap-free alpha matting system for occlusion-aware transformation (Cho et al., 5 Aug 2025). The relation between them is therefore conceptual rather than terminological identity: both seek decompositions that make facial editing more controllable, but they differ in representation, supervision, and downstream use.

7. Applications and research significance

Across its variants, FaceMat targets controllable facial editing under constraints that standard segmentation or direct image translation handle poorly. In the material/matting setting, applications include robust makeup transfer, illumination-aware makeup interpolation and removal without a reference image, and rendering on posed or animated 3D faces with UV-aligned materials and explicit specular behavior (Yang et al., 2023). In the occlusion-aware alpha-matting setting, applications include robust face filters, stylization, beautification, AR overlays, face swapping, and facial reconstruction workflows that must preserve occluders in front of the transformed face (Cho et al., 5 Aug 2025). In relightable monocular reconstruction, the output is a relightable avatar compatible with standard rendering engines such as PyTorch3D and Marmoset Toolbag, enabling environment-map relighting from a single in-the-wild image (Galanakis et al., 2023).

The broader significance of FaceMat lies in the convergence of three research programs. One program seeks physically interpretable facial materials, with UV-aligned diffuse, specular, and geometric representations suitable for rendering and editing. Another seeks soft, semantically correct alpha mattes that remain reliable under fine occlusions and temporal variation. A third seeks single-image robustness, reducing dependence on capture rigs, multi-view setups, or user-supplied trimaps. This suggests that FaceMat is best understood not as a single architecture but as a family of face-specific decomposition strategies in which the target representation—alpha matte, makeup matte, or relightable material stack—is chosen according to the downstream editing or rendering task.