FairFaceGPT: Synthetic Facial Q-A Corpus

• FairFaceGPT is a synthetic question–answer corpus designed for fine-grained facial image understanding in multimodal language models.
• The dataset comprises 10,954 images and 87,632 Q-A pairs covering eight detailed forensic and biometric attribute axes with balanced demographic representation.
• It employs a weakly supervised pipeline using ChatGPT-4 to generate high-quality, visual attribute descriptions that enhance model performance on facial analysis tasks.

FairFaceGPT is a synthetic question–answer corpus constructed for the purpose of fine-grained facial image understanding in multimodal LLMs. Built upon the validation split of the FairFace image dataset, FairFaceGPT provides a substantial resource for advancing face-centric vision-LLMs by introducing high-quality, attribute-targeted annotations generated through a weakly supervised LLM pipeline. Its primary use is in fine-tuning multimodal LLMs (MLLMs) to reason about complex facial cues, forensic traits, and demographic diversity, thereby improving state-of-the-art performance on tasks requiring granular face analysis (Shahreza et al., 14 Jul 2025).

1. Dataset Composition and Coverage

FairFaceGPT is derived from 10,954 images, corresponding exactly to the validation split of FairFace. For each image, eight distinct prompts are posed, resulting in a total of 87,632 Q–A pairs. Each question targets one of eight attribute axes relevant to forensic and biometric face analysis:

1. Demographic Attributes: estimated age range, gender, and ethnicity
2. Facial Structure: cheekbones, jawline, chin, and forehead proportions
3. Skin Texture: smoothness, wrinkles, blemishes, scars, freckles
4. Expression & Emotion: neutral, smiling, frowning, surprise, etc.
5. Lighting & Image Quality: brightness, shadows, blur, resolution
6. Face Pose: frontal, profile, tilt, head orientation
7. Forensic Considerations: occlusions, low-light issues, distinct marks, identification suitability
8. General Description: open-ended holistic caption of the face

The demographic distribution mirrors FairFace’s balanced design: approximately equal gender representation (50% male, 50% female), proportional assignment across seven age bins (3–9, 10–19, 20–29, ..., 60+), and nearly uniform coverage over seven ethnicity categories: White, Black, Indian, East Asian, Southeast Asian, Middle Eastern, and Latino.

2. Data Generation Pipeline

The dataset employs a weakly supervised pipeline wherein for each image, attribute-aware prompts are generated by injecting known metadata (age, gender, ethnicity) into templated instructions. For every attribute, the prompt is submitted via API to ChatGPT-4o using a fixed system prompt:

“You are an AI assistant specialized in analysing face images for forensic and biometric applications. … When analyzing an image consider demographic estimation, facial structure, skin texture, expression, lighting, pose, occlusions, and forensic considerations. Provide detailed descriptions based on visible traits only.”

Each answer is collected verbatim. To avoid information leakage, the demographic preamble is stripped from the final public-facing question, leaving a pure visual query (e.g., “Describe the skin texture of the image based on the visual information.”). This prompt engineering framework ensures high contextual consistency while maximizing the richness of attribute-specific description.

An illustrative sample of the pipeline is:

Step	Input/Operation	Output
Image selection	Choose “FairFace_0001234.jpg” (metadata: 3–9, East Asian, male)	“This is a face image of a boy of East Asian ethnicity, approx. 3–9 years old. …”
Prompting	Attribute: Skin Texture, prompt injected with metadata	“Describe only the skin texture …”
LLM Q–A	Query ChatGPT-4o	Verbatim answer returned
Finalization	Remove metadata from prompt, retain answer	Q: “Describe the skin texture…”; Answer: [verbatim]

The pipeline’s use of synthetic supervision via LLMs enables scalable annotation of visual attributes without manual curation, a significant methodological advance for domain-adapted multimodal AI (Shahreza et al., 14 Jul 2025).

3. Dataset Structure and Format

FairFaceGPT is distributed in a straightforward directory arrangement:

• images/: Contains 10,954 face crops (JPEG), each named with a unique <image_id>.jpg.
• annotations/: Contains fairfacegpt.jsonl, where each line is a JSON object with:

{ "image_id": "FairFace_0001234", "question_id": "FairFace_0001234_q3", "question": "Describe the facial structure of the image based on the visual information.", "answer": "The facial structure exhibits rounded cheekbones... [truncated]" }

Alternatively, a CSV representation can be generated for integration with data analysis pipelines:

image_id	question_id	question	answer
FairFace_0001234	FairFace_0001234_q1	Describe the demographic attributes...	The individual appears to be...

This schema supports alignment of image, prompt, and response for automated benchmarking and model fine-tuning tasks.

4. Quality Assurance and Evaluation

The dataset incorporates multi-tiered quality controls. Automated checks include format validation (strict JSON schema adherence, non-emptiness, and answer length requirements) and attribute keyword overlap, verifying the presence of domain-relevant tokens within each answer (e.g., mention of “wrinkles” in skin texture answers).

A random 5% sample (4,382 Q–A pairs) undergoes human spot checking by two annotators. Inter-annotator agreement across attribute categories achieves Cohen’s κ > 0.75, indicating substantial annotation consistency.

Conventional supervised metrics such as precision or recall are not directly applicable due to the absence of human-written ground truth. Nonetheless, the schema enables the definition of a token-matching F1-score for future use:

$$F1 = 2\cdot \frac{\text{precision}\times\text{recall}}{\text{precision}+\text{recall}}$$

This suggests that, while not directly replacing human annotation, FairFaceGPT’s automatically generated annotations attain high representational fidelity for model development (Shahreza et al., 14 Jul 2025).

5. Applications, Limitations, and Licensing

FairFaceGPT is primarily intended for:

• Fine-tuning multimodal LLMs (MLLMs) for face-centric tasks, including visual question answering (VQA), forensic description generation, and nuanced facial expression analysis.
• Benchmarking multimodal models’ ability to ground language in facial visual semantics.

Limitations include the synthetic nature of all answers (potentially introducing LLM biases), lack of suitability for legal or clinical forensic reporting, and exclusive focus on still images without video or extreme pose coverage.

The dataset, including pretrained FaceLLM models, is released under an MIT-style license, constrained to research use and distributed alongside FairFace-derived resources. It is accessible from https://www.idiap.ch/paper/facellm.

6. Integration with FaceLLM and Empirical Impact

FairFaceGPT serves as the principal training resource for FaceLLM, a multimodal LLM tailored to facial image tasks. Fine-tuning leverages the InternVL3 architecture (separate vision encoder, Qwen2.5 LLM decoder) with LoRA (low-rank adaptation, rank $r=8$, $\alpha=16$), over a single epoch ($\text{LR}=1 \times 10^{-5}$), on a single NVIDIA H100.

Empirical integration yields measurable improvements over the baseline InternVL3 across multiple model sizes and task domains:

Model Size	Overall Accuracy	Face Analysis (Attributes, Expression, Pose)	Bias & Fairness	Face Localization
1B parameters	+0.72% (38.14→38.86)	+3.12% (49.88→53.00)	n/a	n/a
8B parameters	+0.58% (55.56→56.14)	n/a	n/a	+4.18% (44.00→48.18)
38B parameters	+0.62% (59.90→60.52)	+1.74% (63.38→65.12)	+0.67% (70.73→71.40)	n/a

A plausible implication is that synthetic, attribute-targeted supervision as embodied by FairFaceGPT can drive statistically significant gains in both the accuracy and fairness of vision-LLMs for face-intensive scenarios.

7. Significance and Prospects

FairFaceGPT exemplifies a shift toward large-scale, domain-specialized, synthetic datasets for multimodal learning, with particular salience in human-centric AI where interpretability, fairness, and reliability of facial analysis are paramount. It establishes an annotation blueprint for task- and attribute-specific multimodal datasets, enabling systematic benchmarking and development of models tailored to sensitive visual domains, such as biometric identification and facial expression inference.

The precedent set by FairFaceGPT highlights the efficacy and practical scalability of weak supervision via LLMs in overcoming bottlenecks endemic to human annotation at scale, while simultaneously foregrounding the need for continuous scrutiny of bias, representational accuracy, and ethical deployment in downstream applications (Shahreza et al., 14 Jul 2025).