FaceXBench: Evaluating Multimodal LLMs on Face Understanding (2501.10360v2)

Abstract: Multimodal LLMs (MLLMs) demonstrate impressive problem-solving abilities across a wide range of tasks and domains. However, their capacity for face understanding has not been systematically studied. To address this gap, we introduce FaceXBench, a comprehensive benchmark designed to evaluate MLLMs on complex face understanding tasks. FaceXBench includes 5,000 multimodal multiple-choice questions derived from 25 public datasets and a newly created dataset, FaceXAPI. These questions cover 14 tasks across 6 broad categories, assessing MLLMs' face understanding abilities in bias and fairness, face authentication, recognition, analysis, localization and tool retrieval. Using FaceXBench, we conduct an extensive evaluation of 26 open-source MLLMs alongside 2 proprietary models, revealing the unique challenges in complex face understanding tasks. We analyze the models across three evaluation settings: zero-shot, in-context task description, and chain-of-thought prompting. Our detailed analysis reveals that current MLLMs, including advanced models like GPT-4o, and GeminiPro 1.5, show significant room for improvement. We believe FaceXBench will be a crucial resource for developing MLLMs equipped to perform sophisticated face understanding. Code: https://github.com/Kartik-3004/facexbench

• The paper presents FaceXBench, a benchmark of 5,000 questions from diverse datasets categorizing face understanding into six distinct tasks.
• It evaluates 28 multimodal LLMs, revealing significant gaps particularly in low-resolution face recognition, head pose estimation, and deepfake detection.
• The benchmark underscores the need for improved models and specialized tuning to address biases and enhance overall facial analysis capabilities.

An Analytical Overview of "FaceXBench: Evaluating Multimodal LLMs on Face Understanding"

The paper "FaceXBench: Evaluating Multimodal LLMs on Face Understanding" presents the creation and application of a benchmark specifically designed to assess the capabilities of Multimodal LLMs (MLLMs) in understanding facial data. As the use of MLLMs becomes more prevalent in various sectors, such as virtual reality, authentication, and human-computer interaction, understanding their limitations in face-related tasks is of paramount importance.

Key Contributions

The core contribution of this work is the development of FaceXBench, which includes a diverse set of 5,000 questions derived from 25 public datasets and one newly created dataset called FaceXAPI. This benchmark is specifically designed to evaluate different aspects of face understanding by categorizing tasks into six broad segments: Bias and Fairness, Face Recognition, Face Authentication, Face Analysis, Face Localization, and Face Tools Use.

Detailed Task Categories:

1. Bias and Fairness: This category assesses models on tasks like age estimation, gender prediction, and race estimation, thereby uncovering potential biases and ensuring fair outcomes across diverse demographic groups.
2. Face Recognition: Here, both high-resolution and low-resolution face recognition tasks are included along with celebrity identification to test proficiency in feature extraction and identification.
3. Face Authentication: Tasks in this category, such as face anti-spoofing and deepfake detection, are crucial, given the security implications in detecting genuine faces versus spoofing attempts.
4. Face Analysis: The models are evaluated for their ability to predict attributes and recognize facial expressions.
5. Face Localization: This involves tasks like head pose estimation, face parsing, and crowd counting, testing the model's spatial awareness and segmentation capabilities.
6. Face Tools Use: Introduction of the FaceXAPI dataset assesses the competency in tool usage for complex facial understanding scenarios.

Results and Observations

The authors evaluate a set of 28 models, spanning both proprietary and open-source MLLMs. Performance on FaceXBench revealed that the current MLLMs have significant room for improvement, particularly in facial tasks that require fine-grained analysis such as crowd counting and deepfake detection. An analysis of the task categories demonstrated a consistent struggle across models with low-resolution face recognition and nuanced tasks like head pose estimation.

A particularly noteworthy observation is the performance gap between proprietary models like GPT-4o and GeminiPro 1.5 and their open-source counterparts, with certain open-source models surprisingly outperforming these closed models in several categories. This performance is attributed to the safety alignments in proprietary models, limiting their applicability in sensitive contexts like Bias and Fairness assessments.

Implications and Future Directions

FaceXBench provides a significant step towards establishing a standardized metric for face understanding tasks within MLLMs. By exposing the limitations of existing models, it propels forward research into more sophisticated, fair, and unbiased MLLMs. The paper also outlines potential future directions, notably the development of instruction-tune datasets targeting face-related tasks and the incorporation of specialized tools that expand on model capabilities. These could include more intricate datasets for fine-tuning and enhanced computational frameworks for tool-based task resolution.

The work opens avenues for enhancing Domain Specific MLLMs, emphasizing the inadequacy of existing models in real-world applications requiring robust face-processing capabilities. The authors encourage further experiments integrating FaceXBench results to develop the next generation of multimodal models that are comprehensive, adaptable, and more aligned with the complex nature of human faces in digital interactions.

The meticulous approach to benchmarking by the authors—covering the intricacies of face-related tasks—positions FaceXBench as a cornerstone resource for the advancement of AI models capable of sophisticated face understanding. Future iterations and expansions on this benchmark could contribute significantly to improved deployments of MLLMs in sensitive and critical applications.