FaceXBench: Face Understanding Benchmark

• FaceXBench is a systematic benchmark that evaluates multimodal LLMs on nuanced face understanding using 5,000 curated VQA questions and data from 25 public face datasets.
• It organizes 14 face-centric subtasks across six principal domains—including bias, recognition, authentication, analysis, localization, and tool retrieval—to rigorously test model capabilities.
• Empirical results reveal that current models struggle with fine-grained visual discriminations, while preliminary fine-tuning and agentic tool integration show promising performance improvements.

FaceXBench is a comprehensive and systematic benchmark designed to evaluate the capacity of multimodal LLMs (MLLMs) to perform nuanced face understanding. It addresses a critical gap in the assessment of MLLMs by assembling a rigorously curated set of 5,000 multimodal, multiple-choice questions that probe models across a diverse landscape of face-centric tasks, organized into 14 subtasks spanning six principal domains. By leveraging 25 public face datasets and introducing a novel tool-retrieval corpus (FaceXAPI), FaceXBench provides an empirical framework for measuring the strengths and limitations of state-of-the-art MLLMs, including both open-source and proprietary systems (Narayan et al., 17 Jan 2025).

1. Benchmark Composition and Structure

FaceXBench interweaves 10,441 unique images into its 5,000 VQA-style questions, which include 2,750 multi-image, 2,150 single-image, and 100 text-only items. The benchmark pools data from 25 established datasets (e.g., LFW, FairFace, IMDB, CelebA, AffectNet, WMCA, CelebDF) and supplements these with the custom FaceXAPI tool-retrieval set for agentic reasoning evaluation. Question construction employs 757 handcrafted templates, with phrasings diversified via GPT-4o-powered in-context prompts.

Each question is presented with four answer choices (A–D), with positional balancing to mitigate location-based selection biases. Distractors are algorithmically sampled: for numerical queries (e.g., age brackets), distractors reflect adjacent values; for categorical tasks (e.g., gender, race), options are randomly drawn from available classes.

2. Task Taxonomy and Coverage

FaceXBench organizes its evaluation space into six broad categories, listed with their respective subtasks:

Category	Subtasks (14 total)	Example Datasets
Bias & Fairness	Age Estimation, Gender Prediction, Race Estimation	FairFace, UTKFace
Face Recognition	High/Low-Resolution Recognition, Celebrity Identification	LFW, TinyFace, IMDB
Face Authentication	Anti-Spoofing, Deepfake Detection	WMCA, CelebDF, FF++
Face Analysis	Static Attribute Prediction, Facial Expression Recognition	CelebA, AffectNet, RAF-DB
Face Localization	Head Pose Estimation, Face Parsing, Crowd Counting	BIWI, CelebAMask-HQ, JHUCrowd++
Tool Retrieval	FaceXAPI tool selection sequences (text-only)	FaceXAPI

Each category operationalizes distinct visual reasoning: demographic attribute extraction, cross-identity matching, liveness verification, dynamic and static attribute inference, geometric localization, and automated agentic tool orchestration.

3. Evaluation Protocols and Metric

FaceXBench evaluates models in three prompt regimes:

• Zero-Shot: Base instruction plus the candidate question and answer options.
• In-Context Task Description: A concise subtask statement precedes the main prompt (e.g., describing the face analysis duty).
• Chain-of-Thought (CoT): An explicit request for stepwise reasoning is appended.

Answer extraction follows a robust three-pronged regex-based parsing pipeline: (1) leading-letter match, (2) in-string letter match, (3) full-text string comparison. The principal quantitative metric is accuracy: $$\text{Accuracy} = \frac{\#\{\text{correct predictions}\}}{\#\{\text{total questions}\}}$$ No F₁-score or fairness-specific statistical assessment is conducted beyond measuring demographic attribute task performance. This indirect bias probing is limited to age, gender, and race accuracy profiles.

4. Model Suite and Inference Pipeline

A total of 28 MLLMs are benchmarked, including 26 open-source systems and 2 proprietary models (GPT-4o, GeminiPro 1.5). Open-source systems are stratified by LLM parameter count:

• < 4B: PaliGemma, LLaVA-OneVision-0.5b-OV, VILA 1.5-3b
• 4B–13B: Chameleon-7b, Eagle-X4-8B-Plus, Idefics-9b/2-8b, LLaVA-1.5-7b, Monkey-Chat, MiniCPM-Llama3-v2.5, Mantis-SIGLIP-8b, Phi-3.5-Vision, Qwen2-VL-7b, InternVL2-8b
• > 13B: Idefics-80b, LLaVA-1.5-13b, VILA 1.5-13b/40b, CogVLM2-19b, InternVL-Chat-v1.5, LLaVA-OV-72b, Qwen2-VL-72b, InternVL2-76b

All architectures follow a two-stage pathway: a visual encoder (SigLIP-So, CLIP-ViT, InternViT) encodes images into tokens consumed by a generative LLM (Vicuna, Qwen, Llama3, Mistral, Phi). Inputs are formatted per prompt regime, with instructive or self-reflective pre/suffixes as dictated by evaluation mode.

5. Empirical Performance and Error Patterns

Aggregate analysis indicates current MLLMs are sub-human on face understanding: no model exceeds 60% accuracy overall.

Key outcomes:

• Best overall (open-source): Qwen2-VL-72b (57.86%), InternVL2-76b (57.80%)
• Best (proprietary): GeminiPro 1.5 (56.96%), GPT-4o (50.50%)
• Bias & Fairness: ≈70% (proprietary underpenetrate due to safety-alignment)
• Face Recognition: 69–70% (HR), ≈42% (LR)
• Face Authentication: anti-spoofing ≈40%, deepfake ≈35%
• Face Analysis: attribute prediction ≈60%, facial expression ≈57%
• Localization: parsing ≈53%, head pose ≈31%, crowd ≈27%
• Tool Retrieval: top ≈57% (GeminiPro)

Notable error patterns:

• Multi-image queries are systematically more difficult than single-image questions.
• Fine-grained visual discriminations (deepfake, crowd counting, head pose) yield the poorest results, implying limited sub-pixel/density reasoning.
• In-Context prompting offers modest gains in tools/authentication subcategories (+5%) but a net 1–4% overall performance drop.
• CoT prompting consistently degrades performance (–5% to –12%), indicating contemporary MLLMs fail to transfer stepwise textual reasoning routines to visual face tasks (Narayan et al., 17 Jan 2025).

6. Implications, Limitations, and Future Directions

FaceXBench highlights the marked shortfall of current MLLMs in robust, human-level face understanding. While coarse demographic extraction and simple celebrity identification are moderately mastered, capabilities deteriorate on more complex or visually nuanced dimensions. Prompting schemas designed for textual reasoning—particularly CoT—do not generalize to the visual-linguistic face domain.

Two research trajectories are emphasized:

1. Face-Specific Supervised Fine-Tuning: Initial LoRA fine-tuning on LLaVA1.5 with a mix of generic and 70k face-centric samples raises performance modestly (≈+2%), signaling the need for extensive, heterogeneous face-centric instruction tuning resources.
2. Agentic Tool Integration: Incorporating outputs from specialist models (e.g., top-tier deepfake detectors) into MLLMs boosts task performance by 6–10%. This suggests that tool-enabled orchestration architecture—where the MLLM acts as the integrative agent for visual/external modules—is most promising for complex face understanding.

FaceXBench sets a high standard: any future model claiming robust face-competence must demonstrate superior accuracy across all subtasks and prompting conditions. The benchmark, open-source question sets, and evaluation toolkit provide the community with a reproducible and challenging testbed for the next generation of MLLMs in vision-language face understanding (Narayan et al., 17 Jan 2025).