ChEF: A Comprehensive Evaluation Framework for Standardized Assessment of Multimodal Large Language Models

Abstract: Multimodal LLMs (MLLMs) have shown impressive abilities in interacting with visual content with myriad potential downstream tasks. However, even though a list of benchmarks has been proposed, the capabilities and limitations of MLLMs are still not comprehensively understood, due to a lack of a standardized and holistic evaluation framework. To this end, we present the first Comprehensive Evaluation Framework (ChEF) that can holistically profile each MLLM and fairly compare different MLLMs. First, we structure ChEF as four modular components, i.e., Scenario as scalable multimodal datasets, Instruction as flexible instruction retrieving formulae, Inferencer as reliable question answering strategies, and Metric as indicative task-specific score functions. Based on them, ChEF facilitates versatile evaluations in a standardized framework, and new evaluations can be built by designing new Recipes (systematic selection of these four components). Notably, current MLLM benchmarks can be readily summarized as recipes of ChEF. Second, we introduce 6 new recipes to quantify competent MLLMs' desired capabilities (or called desiderata, i.e., calibration, in-context learning, instruction following, language performance, hallucination, and robustness) as reliable agents that can perform real-world multimodal interactions. Third, we conduct a large-scale evaluation of 9 prominent MLLMs on 9 scenarios and 6 desiderata. Our evaluation summarized over 20 valuable observations concerning the generalizability of MLLMs across various scenarios and the composite capability of MLLMs required for multimodal interactions. We will publicly release all the detailed implementations for further analysis, as well as an easy-to-use modular toolkit for the integration of new recipes and models, so that ChEF can be a growing evaluation framework for the MLLM community.

• The paper introduces ChEF, a modular and adaptable evaluation framework that standardizes assessment of MLLMs across multiple performance dimensions.
• The evaluation methodology leverages diverse metrics including calibration, in-context learning, and instruction following to highlight performance gaps among nine benchmark models.
• Results reveal that while some MLLMs excel in instruction adherence, significant challenges persist in multi-task generalization and model robustness.

"ChEF: A Comprehensive Evaluation Framework for Standardized Assessment of Multimodal LLMs" (2311.02692)

Introduction

The paper introduces a Comprehensive Evaluation Framework (ChEF) designed to holistically analyze Multimodal LLMs (MLLMs). Currently, there is a significant gap in standardized assessment methodologies that accommodate the complex, multimodal nature of these models. ChEF seeks to address this void by implementing a modular and scalable evaluation strategy that can adapt to various models and tasks. It aims to facilitate fair comparisons among MLLMs by establishing consistency in evaluation processes across multiple dimensions of model capabilities.

Framework Architecture

The ChEF framework is structured around four core components:

1. Scenario: This involves scalable multimodal datasets, which are a set of datasets concerning representative tasks suitable for MLLMs. It ensures that scenarios can adjust flexibly to include new task datasets as they emerge.
2. Instruction: This outlines the flexible instruction retrieval strategies suitable for the tasks. ChEF integrates in-context examples (ICE) and queries to assist MLLMs in generating contextually relevant responses.
3. Inferencer: This component dictates the models' approach to answering questions, incorporating various strategies such as Chain-of-Thought (CoT) prompting and Perplexity (PPL), to enhance prediction reliability.
4. Metric: These are task-specific score functions that provide indicative evaluations of model performance, facilitating a nuanced understanding of task accomplishment and reliability.

   Figure 1: (a) ChEF Overview. (b) Current MLLM benchmarks can be readily absorbed into ChEF.

Evaluation and Desiderata

ChEF proposes six desiderata that encompass the qualities a robust MLLM should exhibit:

• Calibration: Evaluates how well the model expresses uncertainty and correlates with prediction accuracy.
• In-context Learning: Assesses the ability to utilize ICE effectively to give contextually appropriate responses.
• Instruction Following: Measures adherence to given instructions, ensuring the model does not deviate from expected behavioral patterns.
• Language Performance: Focuses on the generation quality of responses, emphasizing grammatical and contextual quality.
• Hallucination: Tests the model’s capability to avoid generating content inconsistent with the provided inputs.
• Robustness: Evaluates the model's resilience to input alterations, such as corrupted text or images.

  Figure 2: Recipes for evaluating six dimensions of desiderata.

Methodological Evaluation

The study performed extensive evaluations on nine prominent MLLMs across nine scenarios and six desiderata. The findings suggest that while recent MLLMs demonstrate potential, they often struggle with specific tasks, most notably in in-context learning, instruction following, and exhibiting robustness.

Results and Discussion

1. Performance Metrics: MLLMs exhibit considerable variability in performance across different scenarios, emphasizing the need for comprehensive benchmarks like ChEF.
2. Stability and Reliability: LLaVA and Shikra, among other models, display significant variance in performance, albeit benefiting from robust instruction-following capabilities.
3. Impact of Multi-Task Learning: The assessment on both single and multi-task datasets highlighted gaps in MLLMs' abilities to generalize across diverse tasks. Models like InstructBLIP outperformed others in multi-task datasets, underscoring its superior adaptability.

   Figure 3: Results of desiderata.

Conclusion

The ChEF framework successfully provides a systemic approach to evaluating MLLMs, bridging the gap between current benchmarks and the need for a standardized evaluation framework. The framework’s adaptability to include new datasets and its comprehensive evaluation of multiple desirable traits positions it as a critical tool for future research in MLLMs. While certain limitations exist, such as its initial support scope limited to certain scenarios, the inherent scalability of ChEF holds promise for its evolution alongside advancements in AI models. Future developments could include broadening the response analysis and incorporating safety and bias assessment features to further augment its applicability and effectiveness.