DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Models (2411.00836v1)

Abstract: The rapid advancements in Vision-LLMs (VLMs) have shown great potential in tackling mathematical reasoning tasks that involve visual context. Unlike humans who can reliably apply solution steps to similar problems with minor modifications, we found that SOTA VLMs like GPT-4o can consistently fail in these scenarios, revealing limitations in their mathematical reasoning capabilities. In this paper, we investigate the mathematical reasoning robustness in VLMs and evaluate how well these models perform under different variants of the same question, such as changes in visual numerical values or function graphs. While several vision-based math benchmarks have been developed to assess VLMs' problem-solving capabilities, these benchmarks contain only static sets of problems and cannot easily evaluate mathematical reasoning robustness. To fill this gap, we introduce DynaMath, a dynamic visual math benchmark designed for in-depth assessment of VLMs. DynaMath includes 501 high-quality, multi-topic seed questions, each represented as a Python program. Those programs are carefully designed and annotated to enable the automatic generation of a much larger set of concrete questions, including many different types of visual and textual variations. DynaMath allows us to evaluate the generalization ability of VLMs, by assessing their performance under varying input conditions of a seed question. We evaluated 14 SOTA VLMs with 5,010 generated concrete questions. Our results show that the worst-case model accuracy, defined as the percentage of correctly answered seed questions in all 10 variants, is significantly lower than the average-case accuracy. Our analysis emphasizes the need to study the robustness of VLMs' reasoning abilities, and DynaMath provides valuable insights to guide the development of more reliable models for mathematical reasoning.

Evaluation of Mathematical Reasoning Robustness in Vision-LLMs Using DynaMath

The paper presents DynaMath, a novel dynamic visual benchmark designed to evaluate the robustness of mathematical reasoning capabilities in Vision-LLMs (VLMs). This research addresses the inherent limitations of existing static benchmarks and introduces a dynamic approach that enables a more comprehensive assessment of VLMs by generating multiple variants of each seed question.

Mathematical Reasoning Robustness in VLMs

Traditional benchmarks for evaluating VLMs often present a static set of mathematical problems, which limit their ability to test how well these models can generalize across different problem formulations. The DynaMath benchmark aims to overcome this shortcoming by introducing variability in both visual and textual elements of mathematical questions. It comprises 501 seed questions, each encapsulated within a Python program capable of generating numerous question variants by modifying conditions such as numerical values, geometric transformations, and function types. This approach allows researchers to assess a model's worst-case accuracy and reasoning robustness, providing valuable insights into its capabilities beyond average-case performance.

Key Findings and Contributions

The paper evaluates 14 state-of-the-art VLMs, including both closed-source models like GPT-4o and open-source models such as Qwen2-VL, using the DynaMath benchmark comprising 5,010 diverse problem instances. Key findings reveal a significant discrepancy between the average-case and worst-case accuracies in these models, suggesting a lack of robustness in their mathematical reasoning abilities. For example, closed-source models like GPT-4o show a worst-case accuracy that is significantly lower than average-case performance, indicating their unreliability across varied problem contexts.

Furthermore, the paper highlights several consistent failure cases where models struggled to adapt their reasoning to simple variations that are trivially solvable for humans. This finding underscores the need for improvements in model training, particularly in enhancing the adaptability of their reasoning processes in the face of input variations.

Implications and Future Directions

The introduction of DynaMath has significant implications for the development and evaluation of future VLMs. It challenges the community to consider not just the accuracy of models on a fixed set of problems but also their flexibility and adaptability to variations. The findings suggest that current models could benefit from techniques like adversarial training or reinforcement learning that focus on robustness and adaptability.

Going forward, a promising direction involves automating the design and generation of even more complex and varied mathematical problems to push the limits of what models can handle. Additionally, incorporating human feedback into the training process to fine-tune models' reasoning abilities could provide another avenue for enhancing VLM robustness.

In conclusion, DynaMath represents a significant step forward in evaluating the mathematical reasoning capabilities of VLMs, offering a more dynamic and robust framework that better reflects the challenges these models will face in real-world applications. The insights gained from this work pave the way for developing more versatile and reliable models capable of tackling the intricacies of mathematical problem-solving in diverse contexts.