---
title: 'CelebAMat: Synthetic Face Matting'
url: https://www.emergentmind.com/topics/celebamat
type: topic
---

# CelebAMat: Synthetic Face Matting

Searching arXiv for the specified paper to ground the article and citation.
CelebAMat is a synthetic face matting dataset introduced to make occlusion-aware face matting trainable and benchmarkable in the context of face filters, face swapping, stylization, and related face transformation pipelines. It is constructed by compositing clean, occlusion-free faces with diverse occluding objects and materials, together with corresponding soft alpha mattes, and it is explicitly designed for learning face matting and evaluating models under diverse, difficult face occlusions [2508.03055]. Within the associated formulation, face matting is defined as predicting an alpha matte that separates occluding elements from facial regions so that downstream transformations can be applied correctly under hands, hair, accessories, and semi-transparent occlusions.

## 1. Definition and task formulation

CelebAMat is presented as a novel benchmark dataset for evaluating face matting models under diverse and challenging occlusion scenarios, and it also serves as the main training set for the FaceMat framework [2508.03055]. Its stated purpose is to bridge a gap between two pre-existing resource types. Face occlusion datasets such as RealOcc and FaceOcc provide segmentation labels for occluded faces, but not matting-level alpha annotations with soft boundaries or semi-transparency. Generic matting datasets such as SIMD and AM2k provide high-quality alpha mattes, but they are not tailored to faces under complex, realistic occlusions and are not designed around occlusion-aware face filtering [2508.03055].

The underlying task is defined as face matting: given an image of a face with occlusions, predict an alpha matte that separates occluders from facial skin. This framing is central because the target is not merely binary segmentation. The dataset is intended to support alpha estimation with intermediate values in $[0,1]$, including soft transitions at boundaries and semi-transparent mixtures [2508.03055].

The general matting equation used to ground the construction is

$$
I_i = \alpha_i F_i + (1-\alpha_i)B_i, \quad \alpha_i \in [0,1].
$$

In CelebAMat’s use of this formulation, a clean face acts as one layer, occlusions as another, and the stored ground-truth alpha is the face matting alpha that defines how much of each pixel belongs to facial skin versus occlusion or background [2508.03055].

## 2. Dataset construction pipeline

CelebAMat is purely synthetic: all occluded-face images and their ground-truth alpha mattes are generated via compositing [2508.03055]. The clean face source is CelebAMask-HQ, from which samples with visible occlusions are removed using refined annotations and splits from RealOcc/NatOcc. The resulting clean-face pool contains **24,602** training images and **716** test images [2508.03055].

The occlusion sources are drawn from multiple datasets. The paper states that the construction uses **SIMD** and **AM2k** as matting datasets, **HIU-data** as a hand segmentation dataset for realistic hand occlusions, and **DTD** as a texture dataset to increase diversity through textured occlusion or background-like patterns [2508.03055]. For HIU masks, a **Gaussian blur to the boundaries** is explicitly applied to obtain softer transitions and more realistic matting boundaries, so the resulting alpha is not purely binary at edges [2508.03055].

The compositing pipeline is summarized in the paper as follows. It starts from a clean face image $F_{\text{face}}$ from CelebAMask-HQ, samples an occlusion image $I_{\text{occ}}$ and its corresponding mask or alpha from \{SIMD, AM2k, HIU, DTD\}, applies stochastic transformations to the occluder, and composites the occluder onto the face to create an occluded face [2508.03055]. The stochastic transformations include random **size**, **orientation**, and **position**. During video training, although based on still datasets, motion is synthesized by per-frame affine transforms, resizing, horizontal flips, color jitter, and random “pauses” following RVM to emulate motion of the occluding object [2508.03055].

The corresponding soft alpha matte is generated by blending the face mask and occlusion mask, enabling pixel-level soft mask ground truth for training [2508.03055]. The paper does not provide a closed-form blending equation for this step, but it states that the final alpha is derived from the CelebAMask-HQ facial segmentation together with object masks or alphas from the occlusion datasets, including naturally soft or Gaussian-blurred edges [2508.03055]. This suggests that CelebAMat is designed not only to encode semantic separation between face and occluder, but also to preserve physically relevant ambiguity at boundaries.

## 3. Annotations, label semantics, and alpha definition

For each synthetic sample, CelebAMat provides the **occluded face image** $I$ and the ground-truth **alpha matte** $\alpha_{\text{gt}}$ for face matting, defined as skin foreground versus occlusion or background [2508.03055]. The construction pipeline also necessarily uses the original clean face and occluder masks, although the paper does not explicitly state whether those intermediate assets are exposed in the released dataset [2508.03055].

The labeling scheme is more specific than generic portrait matting. The paper’s consistent formulation defines the **skin area** as foreground, encompassing all facial components, while the background consists of the remaining regions outside the foreground [2508.03055]. In operational terms, the alpha is a face-skin matte:

- $\alpha \approx 1$: pixels belonging to skin or tightly attached facial components.
- $\alpha \approx 0$: occluding objects, hair, ears, or background.
- $0 < \alpha < 1$: mixed regions arising from soft boundaries, motion blur, or semi-transparent occlusions [2508.03055].

The conceptual inclusion and exclusion rules are also stated explicitly. The skin region includes skin and the core face area such as eyes, nose, and mouth. Occlusions are defined as elements that should remain in front of the face when recomposing, including hair, hands, ears, accessories, heavy makeup, and some semi-transparent elements such as smoke and fire. An exception is made for “purely transparent lenses without shadows or color,” which are treated as non-occlusions [2508.03055].

A potential source of confusion is addressed in the paper itself. Although one part of the introduction mentions treating facial skin as background and occlusions as foreground, the more precise and consistent formulation in the “Definition of Face Matting” subsection defines skin as the foreground $F$ and occlusions plus everything else as the background $B$ [2508.03055]. In practice, the dataset’s alpha target remains a skin matte even when the downstream compositing objective is to preserve occluders in front of a transformed face.

## 4. Scale, diversity, and benchmark protocol

CelebAMat is designed to maximize **occlusion diversity** while retaining face-domain specificity [2508.03055]. The face diversity is inherited from CelebAMask-HQ, including multiple identities, genders, hair styles, poses, lighting conditions, and high-resolution imagery. The occlusion diversity is obtained from hands in HIU, generic matting objects from SIMD and AM2k, and texture patches from DTD [2508.03055].

The benchmark protocol separates training and evaluation splits at both the face and occluder levels. On the face side, the dataset uses **24,602** clean training faces and **716** clean test faces. On the occluder side, training always draws occlusion instances strictly from the training splits of SIMD, AM2k, HIU, and DTD, whereas evaluation uses only the test splits of those datasets. For the CelebAMat benchmark, the occlusion configurations for test are **fixed** to form a consistent benchmark [2508.03055].

The paper further states that benchmarking uses four test sets, “SIMD, AM2k, HIU, Rand,” generated by different combinations of occlusion types and placements [2508.03055]. The ablation studies indicate that using all four occlusion sets improves performance, and they systematically vary the **occlusion ratio**, defined as the fraction of face area covered. The reported best results occur at ratio **0.25**, described as moderate occlusion [2508.03055].

For evaluation, CelebAMat supports both matting and segmentation metrics. The matting metrics are computed within the unknown region of a trimap derived from $\alpha_{\text{gt}}$ and include **MSE**, **SAD**, **Grad**, and **Conn**. The segmentation metrics, derived from alpha, include **IoU** and **pixel-wise accuracy** [2508.03055]. The dataset therefore functions both as a supervised source of soft alpha labels and as a controlled benchmark with multiple occlusion configurations.

## 5. Role in FaceMat training and uncertainty-guided supervision

CelebAMat is tightly coupled to the FaceMat training design, including teacher-student training, uncertainty modeling, and evaluation [2508.03055]. In Stage 1, the teacher model is trained on synthetic video-like sequences generated from CelebAMat samples. For each base face and occlusion configuration, short sequences are fabricated by transforming the occluding object across frames. The teacher uses the ground-truth alpha and trimaps derived from that alpha to learn both alpha prediction and pixel-wise uncertainty [2508.03055].

Within the unknown trimap region, the Stage 1 loss includes an $L_1$ regression term, a pyramid Laplacian loss for multi-scale edge and detail consistency, and a temporal consistency loss for generated video sequences. The teacher additionally predicts alpha mean and variance $(\mu_\alpha, \sigma_\alpha^2)$ and uncertainty mean and variance $(\mu_u, \sigma_u^2)$, trained via negative log-likelihood losses [2508.03055]:

$$
\mathcal{L}_{\text{NLL}-u}^{\beta} = \frac{1}{2} \left( \frac{(u - \mu_u)^2}{\sigma_u^2} + \log \sigma_u^2 \right) \cdot (\sigma_u^2)^{\beta}
$$

and

$$
\mathcal{L}_{\text{NLL}-\alpha}^{\beta} = \frac{1}{2} \left( \frac{(\alpha - \mu_\alpha)^2}{\sigma_\alpha^2} + \log \sigma_\alpha^2 \right) \cdot (\sigma_\alpha^2)^{\beta}.
$$

The full teacher loss is

$$
\mathcal{L}_{\text{stage1}} = \mathcal{L}_{\text{L1}} + \mathcal{L}_{\text{lap}} + \mathcal{L}_{\text{tc}} + \mathcal{L}_{\text{NLL}-u}^{\beta} + \mathcal{L}_{\text{NLL}-\alpha}^{\beta}.
$$

In Stage 2, the student model is trimap-free and learns directly from CelebAMat alpha targets together with teacher uncertainty. The uncertainty-weighted $L_1$ objective is given as

$$
L_{l1}^{\text{soft}} = \left\| w_{\text{unc}} \odot (\alpha - \alpha_{\text{gt}}) \right\|_1
$$

with

$$
w_{\text{unc}} = w_1 + w_2 \cdot \sigma_u^{\text{teacher}},
$$

where $w_1 = 2$ and $w_2 = 2$ [2508.03055]. Higher teacher uncertainty therefore produces stronger supervision weight in ambiguous regions. The Stage 2 objective is

$$
\mathcal{L}_{\text{stage2}} = L_{l1}^{\text{soft}} + \mathcal{L}_{\text{lap}}.
$$

The paper states that the teacher is updated via EMA of the student parameters using CelebAMat as the common data stream [2508.03055]. In this design, CelebAMat is not merely a passive dataset. Its alpha labels, derived trimaps, and compositional ambiguity directly structure the optimization problem.

## 6. Relation to prior resources, applications, and limitations

Relative to general matting datasets such as SIMD and AM2k, CelebAMat combines a face-specific source domain from CelebAMask-HQ with matting-level occluders, and it is structured to simulate realistic face occlusions such as hands across the face, hair over the forehead, and accessories over the eyes [2508.03055]. Relative to face occlusion datasets such as RealOcc and FaceOcc, it is synthetic but **large-scale**, systematically controllable, and equipped with **per-pixel alpha** rather than only binary or categorical segmentation labels [2508.03055]. The paper also notes that trimaps are used only for training the teacher, not at test time for the student model [2508.03055].

The reported downstream use cases are all tied to the dataset’s alpha definition. In **face filters / AR effects**, stylization, color filters, or makeup filters can be applied to skin while occluders remain in front of the stylized face. In **face swapping**, the alpha matte separates original face content from occlusions so that transformed facial content can be recomposited correctly. In **face completion and editing**, alpha-based isolation of occluders can support inpainting of occluded facial parts prior to recognition, reconstruction, or manipulation [2508.03055].

The empirical claims reported for CelebAMat emphasize its training value. The paper states that training on CelebAMat gives a strong baseline for multiple existing matting models, that RVM trained from scratch on CelebAMat becomes the best baseline on the CelebAMat benchmark, and that the proposed FaceMat improves over RVM in **MSE**, **SAD**, **Grad**, **Conn**, **IoU**, and **pixel accuracy** [2508.03055]. The ablations further report that using all occlusion sets improves performance and that a moderate occlusion ratio of **0.25** yields the best generalization [2508.03055]. A plausible implication is that the benchmark is useful not only for model comparison but also for studying the interaction between occlusion composition and inductive bias in matting systems.

The limitations are stated clearly. CelebAMat is **synthetic**: faces are real, but occlusions are composited, which creates a domain gap relative to fully natural, in-the-wild occlusions. Some real-world artifacts, including lighting inconsistencies and subtle 3D occlusion cues, may not be fully captured. In addition, CelebAMask-HQ annotations are not matting-accurate at very fine detail, so thin structures such as glasses or fine facial hair may be imperfectly delineated, although the compositing and blurring steps partially mitigate this issue [2508.03055]. The paper nevertheless reports good generalization to RealOcc and notes that maintaining temporal inpainting consistency in video remains an open challenge beyond alpha prediction [2508.03055].

The abstract states that the source code and CelebAMat dataset are available at the FaceMat GitHub repository, indicating public release through that channel [2508.03055]. The main text does not spell out exact licensing terms, but it does make clear that the dataset is derived from CelebAMask-HQ, SIMD, AM2k, HIU-data, and DTD, so any downstream use must remain compatible with those upstream resources [2508.03055].

Source: https://www.emergentmind.com/topics/celebamat