Face-LLaVA: Unified Face Analysis MLLM

• Face-LLaVA is a multimodal large language model that fuses facial geometry with visual tokens via region-guided cross-attention for comprehensive face analysis.
• The model leverages the large-scale FaceInstruct-1M dataset to achieve state-of-the-art performance in tasks like facial expression recognition, age estimation, and deepfake detection.
• Innovative architectural design and instruction tuning empower Face-LLaVA to generate natural language explanations grounded in precise facial cues, enhancing interpretability and trust.

Face-LLaVA is a multimodal LLM (MLLM) designed for comprehensive face-centered analysis, including recognition of facial expressions, attributes, action units, age estimation, and deepfake detection, while also supporting natural language reasoning and explanation. Developed through instruction tuning on the large-scale FaceInstruct-1M dataset, Face-LLaVA introduces architectural innovations such as explicit fusion of facial geometry and local visual features via region-guided cross-attention, enabling state-of-the-art performance across a wide variety of face-centric tasks and datasets. The model establishes a new paradigm for unified, explainable, and socially aware AI in face understanding (Chaubey et al., 9 Apr 2025).

1. Model Architecture and Face-Specific Innovations

Face-LLaVA utilizes a multimodal transformer architecture optimized for face-centered understanding. Its pipeline combines the following core modules:

• Patch-based vision encoder ($\mathbf{E}_V$): Processes face images or videos (single or consecutive frames) into dense visual tokens, using a LanguageBind backbone pretrained for image or video input.
• Vision projector ($\mathbf{P}_\mathbf{V}^\theta$): Projects visual feature tokens to the embedding space compatible with the LLM.
• Tokenizer ($\mathbf{E}_T$): Maps user-provided instructions into token sequences for the LLM.
• LLM decoder ($\Phi$): An autoregressive transformer that generates free-form outputs—labels, descriptions, and reasoned explanations.
• Face-expert module ($\mathbf{E}_L$): Detects dense 2D facial landmarks as geometric representations.
• Face-Region Landmark Projector (FRLP): Implements both local and global projection of facial landmarks into token embeddings, grouping landmarks into semantically meaningful facial regions (e.g., left/right eye, brows, nose, mouth).
• Face-Region Guided Cross-Attention (FRGCA): Instead of simple concatenation, FRGCA fuses landmark tokens with visual tokens using cross-attention with spatial proximity masks, prioritizing vision-language alignment near salient face regions.

Region-Guided Cross-Attention Mechanics

For image/frame $x_v$: let $$h_v = \mathbf{P}_\mathbf{V}^\theta(\mathbf{E}_V(x_v))$$ (visual tokens), $z = \mathbf{E}_L(x_v)$ (landmarks), and $h_l = h_l^{global} + h_l^{local}$ (projected landmark tokens).

• Query ($Q$): visual tokens.
• Key, Value ($K, V$): landmark tokens.
• The cross-attention is modulated by a Region-Patch Proximity Mask $$m^{\text{RPP}}_{ji} = -\left\| \text{centroid}(z_i) - \text{centroid}(h_{v,j}) \right\|_2$$, promoting attention to geometrically adjacent features.
• Output: $$h_v^{l} = \text{Linear}\left( \text{Softmax}\left( \frac{Q K^T}{\sqrt{d_{\text{attn}}}} + m^{\text{RPP}} \right)V \right) + h_v$$.

This structure harnesses both global and fine-grained geometric relationships, providing inductive bias essential for subtle facial cue understanding, often missing in generic vision-language encoders.

2. FaceInstruct-1M Dataset: Design and Role

Lack of large-scale, instruction-tuned, face-focused multimodal data is a primary bottleneck for MLLMs in face analysis. Face-LLaVA addresses this by introducing the FaceInstruct-1M dataset, which provides approximately 1 million instruction-description pairs across five face analysis tasks:

• Facial Expression Recognition
• Action Unit Detection
• Facial Attribute Detection
• Age Estimation
• Deepfake Detection

Key properties:

• Source datasets: Integrates images and labels from major face benchmarks (e.g., DFEW, MAFW, FERV39k, DISFA, BP4D, CelebA, UTKFace, MORPH II, FaceForensics++).
• Preprocessing: All samples are face-cropped and verified for single-subject visibility. Landmarks are estimated for each sample.
• Instruction/Description Generation: For each face, Gemini 1.5 Flash LLM produces a label-grounded, visually justified description of the facial category, strictly referencing facial cues only. Each sample is paired with one of 100 task-specific, hand-crafted instructions to simulate realistic user queries.
• Quality Filtering: Outputs are rated by GPT-4o-mini (accuracy, label-consistency, visual evidence, overall sample quality). Approximately 7% of samples are removed for low ratings.
• Coverage: The dataset comprises ~850k images, 120h of video, and balanced representation across class categories, with every task represented with rich, multi-turn, human-like annotations.

3. Training and Evaluation Methodology

Face-LLaVA is instruction-tuned on FaceInstruct-1M to generate both accurate categorical predictions and natural language explanations.

Tasks and Benchmark Datasets

The model is rigorously evaluated, under zero-shot and fine-tuned settings, across:

Task	Datasets
Facial Expression	DFEW, Crema-D, RAF-DB
Action Unit (AU)	DISFA, BP4D
Attribute Detection	CelebA
Age Estimation	MORPH II, UTKFace
Deepfake Detection	FaceForensics++

Predicted descriptors are parsed to extract label predictions for comparative evaluation.

Metrics

• Standard classification metrics: Unweighted/WAR recall, accuracy, F1, MAE (for age estimation).
• Open-ended reasoning: GPT-4o-mini rates generated descriptions on (1) consistency with input, (2) fidelity to ground truth, (3) completeness (1–10 scale).

Baselines

• Open-source MLLMs: LLaVA, Qwen2.5VL, Video-LLaMA 3, LLaVA-OneVision, EmoLA, AU-LLaVA, VL-FAU.
• Commercial LLMs/MLLMs: GPT-4o-mini, Gemini-1.5 Flash.
• Task-specific models: SOTA supervised models for each subtask.

Zero-shot evaluation removes the target dataset from training data for strict generalization tests; fine-tuned comparisons are provided where standard splits exist.

4. Comparative Results and Reasoning Performance

Empirical Results

Face-LLaVA delivers superior or SOTA results across evaluated tasks:

• Facial Expression Recognition: Outperforms all open-source MLLMs in recall, notably on minor classes.
• Action Unit Detection: Achieves the highest F1, surpassing multimodal models such as AU-LLaVA and VL-FAU, approaching task-dedicated supervised methods.
• Attribute Detection and Age Estimation: Exceeds open-source and rivals commercial models, with robust zero-shot transfer.
• Deepfake Detection: Demonstrates strong discrimination, rare among generalist models.

Explanation and Reasoning

• Descriptions generated by Face-LLaVA are rated 33% higher for reasoning completeness (mean ≈ 7.6/10) than the best prior baselines by GPT-4o-mini.
• Qualitative review shows explanations reference precise facial regions and movement (e.g., "raised inner brow indicates surprise"), demonstrate minimal hallucination, and align closely with ground-truth labels.
• Outputs are consistent and grounded, a contrast to models lacking explicit geometric integration.

5. Technical Innovations and Integration with Existing Paradigms

Face-LLaVA advances the state of face-centered MLLMs through:

• Explicit face geometry fusion: Cross-attention between region-grouped facial landmarks and patch visual tokens enhances spatial and semantic grounding.
• Instruction-based learning: Diverse, natural-language prompts and descriptions foster robust in-context and open-ended reasoning.
• Efficient context use: Token context is optimized by fusing rather than concatenating facial geometry, preserving LLM window capacity.
• Extensible instruction design: 100+ hand-crafted prompt types per task ensure coverage of user query styles and increase real-world applicability.

In the context of preceding research:

• Architecturally, Face-LLaVA builds on the transformer-based LLaVA lineage while introducing face-specific enhancements unavailable in generic MLLMs or earlier models like AU-LLaVA or VL-FAU.
• It leverages recent advances in instruction tuning and large-scale, multi-source dataset assembly, coupling visual and geometric representations in a way that supports both high-accuracy recognition and explainable natural language reasoning.

6. Significance for Multi-Task, Explainable, and Social AI

Face-LLaVA establishes a new standard for unified, modular, and explainable face-centered AI:

• Unified face processing: Supports multiple face tasks with a single model and instruction interface, increasing efficiency and simplifying deployment.
• Evidence-based output: Produces justifications citing explicit facial cues, essential for trustworthiness in domains such as healthcare, surveillance, forensic analysis, and social robotics.
• Social AI foundation: Enables development of assistants that can perceive, interpret, and communicate about human facial signals in natural domains.
• Public resource: FaceInstruct-1M dataset and model weights are released to enable reproducible research and benchmarking in the broader community.

---

In summary, Face-LLaVA demonstrates that explicitly integrating facial geometry via cross-attention with visual features, coupled with large-scale, instruction-tuned training across multiple face-related tasks, empowers MLLMs to achieve both high perceptual accuracy and explainable reasoning in face understanding. This approach represents a significant step toward human-aligned, socially aware multimodal assistants for face analysis and communication (Chaubey et al., 9 Apr 2025).