Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark (2501.05444v1)

Abstract: The ability to organically reason over and with both text and images is a pillar of human intelligence, yet the ability of Multimodal LLMs (MLLMs) to perform such multimodal reasoning remains under-explored. Existing benchmarks often emphasize text-dominant reasoning or rely on shallow visual cues, failing to adequately assess integrated visual and textual reasoning. We introduce EMMA (Enhanced MultiModal reAsoning), a benchmark targeting organic multimodal reasoning across mathematics, physics, chemistry, and coding. EMMA tasks demand advanced cross-modal reasoning that cannot be addressed by reasoning independently in each modality, offering an enhanced test suite for MLLMs' reasoning capabilities. Our evaluation of state-of-the-art MLLMs on EMMA reveals significant limitations in handling complex multimodal and multi-step reasoning tasks, even with advanced techniques like Chain-of-Thought prompting and test-time compute scaling underperforming. These findings underscore the need for improved multimodal architectures and training paradigms to close the gap between human and model reasoning in multimodality.

• The paper introduces the EMMA benchmark to rigorously evaluate multimodal reasoning in MLLMs, exposing significant performance gaps compared to human experts.
• It analyzes state-of-the-art MLLMs using techniques like Chain-of-Thought prompting and test-time scaling, which yield only marginal improvements.
• The study underscores the need for innovative multimodal architectures and training paradigms to effectively integrate visual and textual data.

Multimodal Reasoning Challenges in MLLMs: An Analysis of EMMA Benchmark

This essay provides an overview of the paper titled "Can MLLMs Reason in Multimodality? An Enhanced MultiModal ReAsoning Benchmark". The central focus of this research is the evaluation of Multimodal LLMs (MLLMs) and their reasoning capabilities across different modalities, specifically text and images. The authors introduce a new benchmark called EMMA to rigorously assess MLLM performance and highlight the challenges presented by multimodal reasoning tasks.

EMMA Benchmark Overview

EMMA (Enhanced MultiModal reAsoning) is introduced as a benchmark specifically designed to test the integrative reasoning capabilities of MLLMs. Unlike existing benchmarks that emphasize text-dominant reasoning, EMMA challenges models to solve complex problems that necessitate genuine cross-modal reasoning across subjects such as mathematics, physics, chemistry, and coding. The benchmark comprises 2,788 problems, with a focus on visual and textual information integration, involving advanced cross-modal reasoning that is unattainable by focusing on a single modality.

Evaluation of State-of-the-Art MLLMs

The paper evaluates nine state-of-the-art MLLMs using the EMMA benchmark, revealing that these models currently struggle with multimodal reasoning. Even under advanced prompting techniques like Chain-of-Thought (CoT) and test-time compute scaling, these models fail to achieve accuracy close to human expert levels. For instance, the best-performing model on EMMA achieves only 45.75% accuracy compared to human experts' significantly higher scores, underscoring the limitations of current architectures and the need for improved multimodal reasoning frameworks.

Key Findings and Implications

1. Multimodal Reasoning Limitation: The paper identifies a significant gap in the ability of MLLMs to handle complex reasoning tasks that require simultaneous engagement with both visual and textual data. This suggests that current models are not fully leveraging multimodal inputs effectively.
2. Impact of CoT Prompting: While CoT prompting generally aids closed-source models in achieving higher performance, the efficacy varies across different models and tasks. Notably, CoT reduced performance in open-source models, particularly on tasks requiring intricate visual reasoning.
3. Test-Time Scaling: The paper investigates test-time compute scaling methods such as majority voting and Best-of-N selection, finding only marginal improvements in model performance. The reliability of these enhancements is contingent on the strength of the base and reward models utilized.

Future Directions in AI Development

The findings from the EMMA benchmark point to several important implications for future AI research and development:

• Architectural Innovations: There is a pressing need for new multimodal architectures that can more effectively integrate and reason across diverse data types. MLLMs must evolve to dynamically process and interlink visual and textual information seamlessly.
• Training Paradigm Shifts: As traditional training paradigms fall short in boosting multimodal reasoning capabilities, exploring novel training methods that emphasize real-time visual reasoning and dynamic input handling could be beneficial.
• Benchmark Refinement: Enhancements to benchmarks such as EMMA, including expanding underrepresented areas like physics and adding diversified chemistry topics, could provide a robust, comprehensive evaluation of MLLM capabilities.

Conclusion

The paper presents a comprehensive paper of MLLMs' reasoning abilities using the EMMA benchmark, which highlights the limitations and potential areas for growth in AI's capability to handle multimodal reasoning tasks. The insights garnered from this research are critical to guiding the next steps in AI development, promoting architectural advancements, and refining evaluation benchmarks to ensure alignment with real-world problem-solving requirements. As AI continues to progress, such enhancements will be crucial to closing the performance gap between machine and human reasoning in multimodality.