FaceInstruct-1M: Face-Centric Instruction-Tuning Dataset

• FaceInstruct-1M is a face-centric dataset comprising approximately 930,000 instruction–response pairs that integrate cropped face inputs with task-specific instructions and detailed natural language explanations.
• It employs robust preprocessing techniques including MediaPipe BlazeFace cropping, landmark verification, and video segmentation to ensure high-quality face data for diverse analysis tasks.
• The dataset enhances multimodal language models' perception and reasoning capabilities in critical areas such as emotion recognition, action unit detection, and deepfake identification.

FaceInstruct-1M is a large-scale, face-centric instruction-tuning corpus aimed at augmenting multimodal LLMs (MLLMs) with both perception and reasoning capabilities in face analysis tasks. The dataset is designed to bridge the gap between traditional, label-only face classifiers and human-level face understanding that requires both structured outputs and natural language justifications. FaceInstruct-1M contains approximately one million instruction–response pairs derived from established public face datasets, supporting both still images and short video clips, and covers a wide array of face-related tasks fundamental to social signal processing and vision–language modeling (Chaubey et al., 9 Apr 2025).

1. Scope and Purpose

FaceInstruct-1M addresses limitations in existing face-analysis corpora, which are typically task-limited, modality-specific, and capped at less than 50,000 samples. The primary objective is to supply MLLMs with instruction-tuning data enabling complex, interpretable face processing workflows. Each sample comprises a cropped face input, a handcrafted task-specific instruction, and a rich, model-generated explanation contextualized by ground-truth labels. This structure permits downstream MLLMs to answer diverse prompts such as emotion identification with reasoning, action unit detection with anatomical explanations, and deepfake classification with artifact-based justification.

2. Construction Methodology

FaceInstruct-1M is assembled from twelve public face-centric datasets, each focusing on one or more of the core targeted tasks:

• Facial expression recognition: DFEW, MAFW, FERV39k, Crema-D, AffectNet, RAF-DB
• Action unit (AU) detection: DISFA, BP4D
• Facial attributes: CelebA
• Age estimation: MORPH II, UTKFace
• Deepfake detection: FaceForensics++, Fake AV-Celeb, and real control clips from DFEW/MAFW/FERV39k

To ensure annotation reliability, only samples with a single, clearly visible face (in each video frame for temporal samples) and robustly extracted facial landmarks were retained. Face images were cropped using MediaPipe BlazeFace. Video samples in which faces exited the frame or multiple faces appeared were discarded, amounting to an exclusion rate of approximately 5–10% per source. Deepfake videos were divided into 3-second segments to enhance diversity.

Each face crop or video segment was paired with (a) a randomly selected instruction from a pool of 100 handcrafted task-specific templates, and (b) a natural language explanation generated using Gemini 1.5 Flash, conditioned on the visual input, ground-truth label, and a negative prompt to ensure the exclusion of extraneous context. Generated responses were filtered by GPT-4o-mini based on label correctness, visual consistency, ground-truth alignment, and textual quality; samples scoring ≤ 6/10 were excluded, accounting for a further 7% reduction.

3. Dataset Composition and Statistics

Following curation, cropping, annotation, and quality filtering, the dataset contains approximately 930,000 instruction–response pairs, split between still images (≈350,000) and video clips (≈580,000), with median video length 2–4 seconds.

The distribution across tasks is summarized in the following table:

Task	Sample Count (N)	Percentage (p)
Expression	350,000	~38%
AU	290,000	~31%
Attribute	200,000	~22%
Age	70,000	~8%
Deepfake	95,000	~10%
Total	930,000	100%

Class distributions within tasks follow the source datasets. For example, within expression recognition, “happy” composes 30–40% of samples while “fear” and “disgust” are each less than 5%. AU data are biased toward commonly activated units (AU 6—cheek raiser, AU 12—lip corner puller), and facial attributes such as “smile” (~60%) and “beard” (~10%) reflect source statistics. Age labels predominantly cluster in the 20–40-year range. Deepfake samples are distributed ~55% fake, ~45% real.

4. Annotation Structure and Examples

Each entry in FaceInstruct-1M is an (image or video, instruction, response) triple. Instructions are selected from a template bank to introduce linguistic and semantic variability, while responses are generated to deliver both task completion and explanatory depth. All reasoning is facial-cue–focused—audio and background details are excluded by design. The constraint reduces hallucination risk and forces the model to rely on local visual evidence.

Typical examples:

• Expression Recognition Instruction: "Based on the facial movements, what emotion is shown and why?" Response: "The subject displays surprise. Their eyebrows suddenly arch, eyes widen with visible sclera above and below the iris, and the jaw drops slightly. These rapid muscle changes around the brow and jaw correspond to a startled expression."
• AU Detection Instruction: "Identify all active Action Units and explain your reasoning." Response: "Active AUs: 12 (lip corner puller) and 6 (cheek raiser). You can see the lip corners are drawn upward into a smile (AU 12), and the cheeks are lifted forming fine webbing around the lower eyelids (AU 6)."
• Deepfake Detection Instruction: "Is this video a deepfake? Provide the label and reasoning." Response: "Fake. The skin texture is unnaturally smooth—lacking normal pore structure—and there is minor misalignment where the eyelid edge flickers between frames. These artifacts indicate GAN‐based synthesis."

5. Comparison with Prior Face Datasets

FaceInstruct-1M differs from established face datasets along several critical axes:

• Scale: Contains ≃1 million samples, surpassing the 20–200k typical of prior corpora.
• Granularity: Each sample combines an input, task-specific instruction, a detailed explanation, and the corresponding label; most prior datasets offer labels or minimal attribute flags without natural language rationales.
• Multimodal Coverage: Supports both static and dynamic imagery, thus encompassing the temporal dimension inherent in facial behavior.
• Reasoning Focus: Explanations are strictly grounded in the presented facial features, minimizing reliance on context or background information. This focus aims to mitigate the risk of models generating extraneous or hallucinatory reasoning.

6. Preprocessing, Licensing, and Intended Use

All underlying media are drawn from publicly released, appropriately licensed datasets (e.g., CelebA, FaceForensics++). FaceInstruct-1M will distribute only its annotation, instruction templates, and filtering scripts; end users must procure original visual data separately. Preprocessing pipelines involved robust face detection, face cropping, landmark verification, and video chunking (approximately 3 seconds per segment).

The corpus is targeted at fine-tuning and evaluating MLLMs in human-centered AI applications requiring both interpretable outputs and rich, language-driven face analysis. Potential deployment domains include healthcare triage, accessibility tools, video forensic analysis, and social robotics, where precise, explainable face understanding is mandated (Chaubey et al., 9 Apr 2025).