---
title: 'FairFaceGen: Balanced Synthetic Face Dataset'
url: https://www.emergentmind.com/topics/fairfacegen
type: topic
---

# FairFaceGen: Balanced Synthetic Face Dataset

FairFaceGen is a balanced synthetic face dataset and generation framework introduced for studying whether face recognition (FR) systems trained on synthetic data can achieve both high accuracy and fairness. It is built by generating demographically controlled seed identities with **Flux.1-dev** and **Stable Diffusion v3.5 (SD35)**, and then expanding those identities with identity-preserving augmentation methods including **Arc2Face** and **four IP-Adapters**. The work is organized around scale-controlled comparison: synthetic and real datasets are matched in identity count so that measured differences in FR performance and bias are more directly attributable to generation choices rather than dataset size [2507.20782].

## 1. Definition and research objective

FairFaceGen is described as a purpose-built balanced synthetic face dataset for evaluating the impact of synthetic data on both FR performance and demographic bias. Its explicit target is the question of whether synthetic data can simultaneously support strong recognition accuracy and bias mitigation. In that framing, FairFaceGen is not merely a collection of generated images; it is a controlled experimental substrate for isolating the effects of the **seed generator**, the **identity augmentation generator**, and the **number of intra-class augmentations** on downstream FR behavior [2507.20782].

The paper identifies three problems in prior synthetic-face studies. First, comparisons between synthetic and real datasets are often unfair because the datasets differ substantially in scale. Second, prior work tends to emphasize identity generation while underexploring the role of intra-class augmentation. Third, bias is often insufficiently evaluated even when synthetic data is nominally balanced. FairFaceGen addresses these points by controlling dataset scale, separating seed identity generation from augmentation generation, and evaluating fairness on **Racial Faces in-the-Wild (RFW)** [2507.20782].

A central methodological choice is that all datasets in the comparison are limited to approximately **10K identities**. Synthetic variants contain about **10–11K identities**, while the real baselines include **CASIA-WebFace** at about **10.5K identities** and **WebFace42M-10K** as a 10K subset. This scale matching is presented as necessary for fair comparison [2507.20782].

## 2. Demographic design and synthetic generation pipeline

FairFaceGen constructs a balanced identity pool over **2 genders**—male and female—**4 races**—Caucasian White, Black, East-Asian, and Latino/Hispanic—and **7 age groups** ranging from children to seniors. The combinatorial design is explicit:

$$
7 \times 4 \times 2 = 56
$$

With **200 identities per combination**, each seed generator yields

$$
56 \times 200 = 11{,}200
$$

seed identities [2507.20782].

The generation process has two stages. In the first stage, seed identities are generated from text prompts using **Flux.1-dev** or **SD35**. The same prompts are used for both models so that output differences are attributable to the generator rather than prompt variation. In addition to the controlled demographic attributes, the prompts vary soft visual attributes such as **hairstyle**, **hair color**, **glasses**, **earrings**, **lighting**, **background**, **photo style**, **image quality**, and **camera type**. These soft attributes are sampled randomly and are therefore intended to be evenly distributed across groups [2507.20782].

In the second stage, each seed identity is expanded through identity-preserving augmentation. The augmentation count per identity is varied systematically across **8**, **16**, **24**, and **32** images, allowing the study to separate the effect of identity balance from the effect of intra-class diversity [2507.20782].

| Stage | Components | Role |
|---|---|---|
| Seed identity generation | Flux.1-dev; SD35 | Create balanced synthetic identities |
| Identity-preserving augmentation | Arc2Face; four IP-Adapters | Create intra-class variation |
| Controlled comparison | Synthetic and real datasets at approximately 10K identities | Isolate generation effects |

This construction makes FairFaceGen a framework of dataset variants rather than a single fixed dataset. A plausible implication is that the paper treats demographic control, identity diversity, and augmentation fidelity as separable design axes rather than as a single monolithic notion of “synthetic data quality.”

## 3. Augmentation mechanisms and identity preservation

The augmentation stage is central to FairFaceGen’s design. **Arc2Face** conditions the generation process using an **ArcFace embedding** and is paired in the paper with **Stable Diffusion v1.5 (SD15)**, using face embeddings from an ArcFace model pretrained on **WebFace42M**. The four **IP-Adapter** variants differ along two dimensions: the diffusion backbone (**SD15** or **SDXL**) and the identity embedding source (**InsightFace** or **CLIP**) [2507.20782].

These methods are intended to generate multiple images of the same identity under variations in pose, expression, and illumination while preserving identity consistency. The paper emphasizes that this stage is not ancillary: it is one of the primary determinants of both accuracy and fairness. In effect, FairFaceGen separates the problem of generating *who* the identity is from the problem of generating *how* that identity varies across samples.

The paper supplements this design with an identity-similarity analysis using an independent **EdgeFace** model. For each identity, intra-class cosine distances among augmentations are computed; distributions close to zero indicate highly consistent augmentations. The reported result is that synthetic data, including FairFaceGen variants, has intra-class similarity distributions comparable to real **WebFace** data. Inter-class analysis is stricter: the minimum cosine distance from each identity to its most similar different identity is used to assess separation. On this measure, synthetic datasets generally have **lower inter-class variability** than real data. The paper notes that **HyperFace** is closest to real **WebFace42M**, while **Flux-based** identities are furthest from real **WebFace42M**, suggesting that Flux may be weaker as a seed generator for FR training [2507.20782].

This diagnostic result clarifies an important distinction. Identity-preserving augmentation can succeed in keeping samples within a class coherent while still producing an overall dataset whose classes are not sufficiently separated from one another. That distinction becomes consequential on more difficult FR benchmarks.

## 4. Experimental protocol and fairness metrics

All FR models in the study use the same backbone and loss: **ResNet-50** trained with **AdaFace loss**. Face detection and landmark extraction use **MTCNN**; faces are aligned and resized to **112 × 112 RGB**. Training proceeds for **30 epochs** with **batch size 256**, an initial learning rate of **0.1**, and learning-rate drops by a factor of 10 at epochs **12**, **20**, and **26**. Standard AdaFace augmentation is applied with probability **0.2**. At inference time, embeddings are compared using **cosine similarity** [2507.20782].

The evaluation is divided across three regimes. Standard FR performance is assessed on **LFW**, **CA-LFW**, **CP-LFW**, **CFP-FP**, and **AgeDB-30**. More difficult generalization is tested on **IJB-B** and **IJB-C**, using **True Accept Rate (TAR)** at

$$
\text{FAR} = 10^{-3}, \, 10^{-4}.
$$

Bias is measured on **RFW**, which contains four racial groups—**African**, **Asian**, **Caucasian**, and **Indian**—with balanced genuine and impostor pairs [2507.20782].

The paper uses both direct group-wise performance and a compact fairness summary. The bias metric is

$$
\text{Bias} = \operatorname{Std}(\text{group accuracies}),
$$

so lower standard deviation indicates lower disparity across demographic groups. A joint accuracy–fairness score is also defined as

$$
\text{TO} = \text{Average Accuracy} - \text{Std}.
$$

This design makes the evaluation explicitly multi-objective: a synthetic dataset is not considered successful purely by maximizing average recognition accuracy if it also increases disparity [2507.20782].

The synthetic baselines are **DCFace**, **Digi2Real**, and **HyperFace**; the real baselines are **CASIA-WebFace** and **WebFace42M-10K**. This comparative set is intended to distinguish the effect of FairFaceGen’s balancing and augmentation strategy from generic effects of synthetic-data usage [2507.20782].

## 5. Empirical findings on accuracy, generalization, and bias

The paper’s most consequential finding is that standard benchmark accuracy does not guarantee strong real-world generalization. Models can score well on **LFW**-style benchmarks yet perform poorly on **IJB-B/C**. **CASIA-WebFace** is highlighted as a striking example: it achieves strong results on standard benchmarks but very poor **IJB-B/C** performance [2507.20782].

Against that background, FairFaceGen reveals a mixed but informative picture of synthetic data. Synthetic datasets are competitive on standard benchmarks, but they still **lag behind real data on the challenging IJB-B/C protocols**. The best synthetic baseline, **DCFace-10K**, remains below **WebFace42M-10K** on **IJB-B/C**. The paper therefore treats synthetic data as promising but not yet a replacement for real-data generalization on the hardest verification settings [2507.20782].

On fairness, the results are more favorable. **SD35-based synthetic data** shows the strongest behavior on **RFW**, and the best reported result is **SD35-All-IPA** with the lowest standard deviation,

$$
\text{Std} = 1.06,
$$

which is lower than the real-data baselines. The paper presents this as evidence that demographically balanced synthetic generation can reduce racial disparity in FR [2507.20782].

The number and quality of intra-class augmentations are also decisive. Increasing images per identity from **8** to **16** generally improves recognition on both standard benchmarks and **IJB-B/C**. Beyond that, the effect becomes mixed: **24** and **32** images per identity do not guarantee better outcomes, and for both **SD35** and **Flux**, increasing augmentation count eventually causes **IJB-B/C** performance to drop even if standard benchmark scores continue to improve. The paper interprets this as possibly arising from excessive intra-class variation, identity leakage between classes, or augmentations becoming less identity-faithful. It also reports that **All-IPA** mixtures can reduce **IJB-B/C** performance, indicating that greater synthetic diversity is not automatically beneficial [2507.20782].

The generator choice matters as well. **Flux-based** datasets show higher bias than **SD35-based** ones, and the paper suggests that Flux images may be more “professionally styled” and less representative of in-the-wild face variation. This suggests that visual plausibility alone is not a sufficient criterion for synthetic FR training data; distributional resemblance to operational imagery is also material [2507.20782].

## 6. Interpretation, limitations, and relation to adjacent research

FairFaceGen’s scope is narrower than generic face generation and broader than a simple dataset release. It is specifically about constructing balanced synthetic data for FR training under controlled comparisons. That places it in a distinct position relative to adjacent face-generation and benchmarking efforts. **DeepFaceGen**, for example, is a large-scale universal benchmark for **face forgery detection** that explicitly considers **fairness across ethnicities** and provides rich prompt labels, but its target task is detection rather than FR training [2406.09181]. **UniF^2ace** addresses **fine-grained face understanding and generation** in a unified multimodal model, emphasizing captioning, VQA, and text-to-face synthesis rather than fairness-aware FR data construction [2503.08120]. **FaceScore** focuses on **face quality** as a weak point of text-to-image diffusion models and proposes a face-specific metric and refinement method, again targeting generation quality rather than FR bias [2406.17100].

FairFaceGen therefore addresses a different research question: whether synthetic generation can be operationalized as a controlled intervention on demographic balance without sacrificing recognition performance. Its main limitation, made explicit by the results, is that balanced synthetic data still underperforms real data on **IJB-B/C**. Another limitation is that fairness gains depend strongly on augmentation design; demographic balance at the seed-identity stage does not by itself guarantee optimal downstream behavior. This suggests that fairness in synthetic FR datasets is mediated jointly by class balance, inter-class separation, and identity-preserving intra-class variability.

The paper’s practical guidance follows from these findings. If fairness matters, demographic attributes should be controlled during synthetic generation; synthetic FR datasets should not be judged only on easy benchmarks; dataset scale should be matched when comparing synthetic and real data; the seed generator should be chosen carefully, with **SD35** generally yielding better fairness properties than **Flux**; and excessive intra-class augmentation should be avoided. The paper further suggests that combining synthetic and real data may offer the best route to improving both accuracy and fairness [2507.20782].

In that sense, FairFaceGen is best understood not as a claim that synthetic data has solved FR bias, but as a controlled empirical framework showing where synthetic balancing helps, where it fails, and which design variables most strongly govern the trade-off between recognition accuracy and demographic disparity.

Source: https://www.emergentmind.com/topics/fairfacegen