FaceBench: Multi-View Facial VQA Benchmark

• FaceBench is a comprehensive, hierarchically structured facial VQA dataset featuring multi-view attributes (appearance, accessories, surrounding, psychology, identity) for nuanced face perception.
• The dataset integrates over 15,000 images with 211 attributes and 701 attribute values through rigorous manual annotation and systematic QA generation.
• Benchmark evaluations show that fine-tuned, face-specialized MLLMs significantly improve performance, yet a gap remains compared to human-level facial analysis.

FaceBench, as introduced in "FaceBench: A Multi-View Multi-Level Facial Attribute VQA Dataset for Benchmarking Face Perception MLLMs" (Wang et al., 27 Mar 2025), is a large-scale, hierarchically structured dataset specifically designed for evaluating multimodal LLMs (MLLMs) on the task of fine-grained face perception. Unlike prior benchmarks with limited attribute coverage and shallow annotation, FaceBench encompasses multi-view and multi-level attributes, supporting extensive visual question answering (VQA) for nuanced facial analysis. It enables both evaluation and fine-tuning of MLLMs, revealing current limitations and facilitating targeted progress in face-centric AI.

1. Hierarchical Multi-View Attribute Construction

FaceBench is built around a rigorous, hierarchical taxonomy, delineating five primary facial "views" (Editor's term) and up to three levels of attribute granularity. The five views are:

• Appearance: Biological fundamentals—hair, eyes, skin, mouth, nose, face shape, etc.
• Accessories: Worn items including glasses, hats, earrings, face masks, and jewelry.
• Surrounding: Environmental context and background features.
• Psychology: Expressions, emotions, and action units indicating affective states.
• Identity: Social markers such as race, age, and gender.

For each view, attributes are organized at up to three granularity levels:

• Level 1: Coarse grouping (e.g., "nose").
• Level 2: Sub-component (e.g., "nostrils").
• Level 3: Fine-grained descriptors (e.g., "nose tip size", "hair color", "skin texture"), each associated with categorical values (e.g., colors, shapes).

The dataset comprises:

• 211 attributes and 701 attribute values, meticulously catalogued and cross-referenced.

Attribute Level	Appearance	Accessories	Surrounding	Psychology	Identity	Overall
Level 1	18	13	4	1	3	39
Level 2	29	8	1	1	0	39
Level 3	85	45	3	0	0	133
Values	430	200	19	34	18	701
Templates	121	61	7	2	3	194

This multi-view, multi-level design allows for targeted probing of MLLM capability across attribute classes and abstraction levels.

2. Dataset Collection, Annotation, and QA Generation

FaceBench leverages both public datasets and manual annotation to ensure high-quality, diverse coverage:

• Source images: 15,842 from FairFace (identity), RAF-DB/RAF-AU (psychology), CelebA-HQ, and FFHQ (appearance, accessories, surrounding).
• Manual annotation: 200 trained annotators labeled 300 high-resolution images (CelebA-HQ, FFHQ), with 5 independent ratings per Level 1 attribute.
• Quality control: Majority voting for TFQ/SCQ, thresholding for MCQ, ChatGPT synthesis with timing control for OEQ, senior review.
• VQA templates: 194 question templates crafted for systematic coverage, resulting in:
  • 49,919 QA pairs for evaluation.
  • 23,841 QA pairs for fine-tuning.

QA Type	Appearance	Accessories	Surrounding	Psychology	Identity	Overall
TFQ	2,905	1,401	107	/	/	4,413
SCQ	6,799	643	428	3,068	32,862	43,800
MCQ	166	/	/	920	/	1,086
OEQ	297	207	116	/	/	620

The full test set consists of 49,919 VQA pairs, supporting granular analysis of attribute classification, reasoning, and contextual judgement.

3. Evaluation Protocols and Benchmarking Procedures

FaceBench is used to benchmark MLLMs on both evaluation and fine-tuning splits. Its multi-type VQA pairs test various model capabilities, including:

• True/False (TFQ): Binary attribute identification.
• Single Choice (SCQ): Classification into one attribute value.
• Multiple Choice (MCQ): Multi-label, often for composite or accessory features.
• Open-Ended (OEQ): Free-form attribute description.

Evaluation metrics include accuracy for TFQ/SCQ, macro F1 for MCQ, and ROUGE-L for OEQ.

All models are assessed with these protocols, providing a robust framework for comparative and ablation studies. Extensive template variety and rigorous QA generation mitigate superficial dataset memorization, instead requiring models to perform true visual-semantic reasoning across hierarchies.

4. Benchmarking MLLM Performance: Empirical Results and Baseline Models

FaceBench experiments include broad-based evaluation of both open-source and commercial MLLMs, as well as the introduction of Face-LLaVA, a face-specialized MLLM baseline:

• Face-LLaVA-13B (trained on the FaceBench fine-tuning split) achieves superior results among open-source models:
  • Appearance: 60.29%
  • Psychology: 61.85%
  • Identity: 71.64%
  • Overall: 61.16%
• Commercial SOTA models:
  • GPT-4o: 63.21% overall
  • Gemini-1.5-Pro: 62.72% overall
• Human-level: 67.38% (Appearance, Accessories, Surrounding views)

Performance remains consistently 7–15% below human accuracy, even for leading closed-source MLLMs. Accessories and Surrounding view tasks are especially challenging, reflecting unresolved limitations in current models.

On specific attributes ("face mask", "gender"), both Face-LLaVA and commercial models approach or attain high accuracy (>90%), while nuanced features ("emotion", "lips", "background") commonly yield large inter-model variance and lower scores.

5. Implications for MLLM Design, Data Construction, and Open Problems

FaceBench empirically demonstrates that:

• Fine-tuning on structured, hierarchical, high-quality facial VQA data offers marked improvement in multimodal model attribute perception, moving open-source performance toward commercial levels.
• Manual annotation and systematic template/question construction are critical for revealing weaknesses and avoiding superficial learning.
• Multi-view, multi-level evaluation exposes domain-specific gaps, especially for composite and context-dependent attributes.

A plausible implication is that further progress toward human-level face perception in AI will require not only increased data scale but also enhanced modeling of facial attribute hierarchies, contextual dependencies, and semantic generalization.

6. Future Directions, Open Resources, and Community Impact

FaceBench, as the most comprehensive face perception VQA dataset to date, sets a precedent for future face-centric multimodal benchmarks. Its publicly available dataset (https://github.com/CVI-SZU/FaceBench), annotation protocols, and fine-tuning splits provide an authoritative foundation for:

• Continued MLLM benchmarking,
• Data-driven architectural improvements,
• Analysis of human–AI gaps in face perception,
• Research in fairness, robustness, and context-aware vision-language understanding.

The dataset structure and findings encourage investigation into advanced attribute hierarchies, flexible reasoning architectures, and improved annotation protocols, with the aim of narrowing the gap between artificial and human face perception capabilities.

FaceBench's systematic evaluation approach thus actively directs the research community toward more representative, robust, and generalizable solutions for face-oriented multimodal language modeling.