MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems? (2403.14624v2)

Abstract: The remarkable progress of Multi-modal LLMs (MLLMs) has garnered unparalleled attention, due to their superior performance in visual contexts. However, their capabilities in visual math problem-solving remain insufficiently evaluated and understood. We investigate current benchmarks to incorporate excessive visual content within textual questions, which potentially assist MLLMs in deducing answers without truly interpreting the input diagrams. To this end, we introduce MathVerse, an all-around visual math benchmark designed for an equitable and in-depth evaluation of MLLMs. We meticulously collect 2,612 high-quality, multi-subject math problems with diagrams from publicly available sources. Each problem is then transformed by human annotators into six distinct versions, each offering varying degrees of information content in multi-modality, contributing to 15K test samples in total. This approach allows MathVerse to comprehensively assess whether and how much MLLMs can truly understand the visual diagrams for mathematical reasoning. In addition, we propose a Chain-of-Thought (CoT) evaluation strategy for a fine-grained assessment of the output answers. Rather than naively judging True or False, we employ GPT-4(V) to adaptively extract crucial reasoning steps, and then score each step with detailed error analysis, which can reveal the intermediate CoT reasoning quality by MLLMs. We hope the MathVerse benchmark may provide unique insights to guide the future development of MLLMs. Project page: https://mathverse-cuhk.github.io

Introducing MathVerse: Evaluating Multi-modal LLMs in Visual Math Problem Solving

Overview of MathVerse

MathVerse is an innovative benchmark designed to rigorously assess the capabilities of Multi-modal LLMs (MLLMs) in solving visual math problems. This benchmark distinguishes itself by focusing on the actual interpretation of diagrams by MLLMs, rather than relying predominantly on accompanying textual descriptions. MathVerse comprises 2,612 visual math problems, each meticulously transformed into six versions with varying degrees of information content across modalities, resulting in a comprehensive dataset of 15,672 test samples.

Why MathVerse?

Current benchmarks often do not accurately evaluate an MLLM's ability to interpret visual information within math problems. They typically contain redundant textual information which MLLMs could exploit, bypassing the need for genuine diagram understanding. MathVerse addresses this by offering problems with progressively reduced textual content and enhanced diagram details, compelling models to rely more on visual interpretation for problem solving.

Key Features of MathVerse

• Rich Dataset: MathVerse includes a wide range of visual math problems covering plane geometry, solid geometry, and functions. These problems are further categorized into twelve detailed subfields, facilitating a multi-dimensional evaluation of MLLMs.
• Investigating Diagram Interpretation: By creating six distinct versions of each problem with varying degrees of multimodal content, MathVerse allows for a deep dive into how MLLMs utilize visual information in mathematical reasoning.
• Chain-of-Thought Evaluation: Leveraging a novel Chain-of-Thought (CoT) evaluation method, MathVerse enables a fine-grained assessment of MLLMs' reasoning processes. This approach not only judges the correctness of the final answer but also provides detailed insights into the intermediate reasoning steps.
• Comprehensive Assessment: The inclusion of a variety of problem versions and subjects ensures that MathVerse offers a comprehensive platform for evaluating the visual mathematical reasoning capabilities of MLLMs, from basic diagram understanding to complex mathematical deduction.

Insights and Findings

Through extensive experiments involving leading MLLMs, MathVerse reveals significant insights:

• Dependence on Textual Cues: Contrary to expectations, most MLLMs perform better when visual cues are minimized. This indicates a predominant reliance on textual information, highlighting a gap in true diagram understanding.
• Challenges in Diagram Interpretation: As textual information is reduced, MLLMs' performance decreases, underscoring the difficulty models face in extracting and interpreting mathematical conditions directly from diagrams.
• Superior Performance of Closed-source MLLMs: While closed-source MLLMs like GPT-4V generally outperform their open-source counterparts, there remains a considerable performance gap compared to human solvers, indicating room for improvement in visual reasoning capabilities.

The Path Forward

MathVerse stands as a pivotal step towards truly understanding and enhancing the visual mathematical reasoning abilities of MLLMs. The insights gained from this benchmark pave the way for future development in this area. Possible directions include improving visual encoders within MLLMs, developing richer training datasets encompassing a broader range of mathematical concepts and increasing the diversity of problem types to include multilingual and higher-difficulty problems.