MultiMath: Bridging Visual and Mathematical Reasoning for Large Language Models (2409.00147v1)

Abstract: The rapid development of LLMs has spurred extensive research into their domain-specific capabilities, particularly mathematical reasoning. However, most open-source LLMs focus solely on mathematical reasoning, neglecting the integration with visual injection, despite the fact that many mathematical tasks rely on visual inputs such as geometric diagrams, charts, and function plots. To fill this gap, we introduce \textbf{MultiMath-7B}, a multimodal LLM that bridges the gap between math and vision. \textbf{MultiMath-7B} is trained through a four-stage process, focusing on vision-language alignment, visual and math instruction-tuning, and process-supervised reinforcement learning. We also construct a novel, diverse and comprehensive multimodal mathematical dataset, \textbf{MultiMath-300K}, which spans K-12 levels with image captions and step-wise solutions. MultiMath-7B achieves state-of-the-art (SOTA) performance among open-source models on existing multimodal mathematical benchmarks and also excels on text-only mathematical benchmarks. Our model and dataset are available at {\textcolor{blue}{\url{https://github.com/pengshuai-rin/MultiMath}}}.

The paper "MultiMath: Bridging Visual and Mathematical Reasoning for LLMs" introduces MultiMath-7B, a domain-specific multimodal LLM (MLLM) designed to integrate visual and mathematical reasoning. This model addresses a significant gap in current open-source MLLMs that typically lack comprehensive capabilities in handling combined visual and mathematical tasks, which are prevalent in many real-world applications.

Key Contributions

1. Model Development: MultiMath-7B is a novel MLLM created to effectively handle multimodal mathematical reasoning tasks by integrating visual inputs into mathematical problem-solving. The model architecture builds on the strengths of vision-language alignment and endeavors to extend these capabilities to the mathematical domain.
2. Training Methodology: The training process for MultiMath-7B is structured in four stages:
   • Vision-Language Alignment: Aligns the vision encoder with the LLM to support visual input processing.
   • Visual Instruction-tuning: Improves the model's ability to comprehend and respond to visual tasks.
   • Mathematical Instruction-tuning: Enhances mathematical reasoning, targeting chain-of-thought (CoT) capabilities with rigorous multistep training.
   • Process-Supervised Reinforcement Learning: Utilizes reinforcement learning to refine step-level reasoning processes, correcting errors through a preference-driven reward model.
3. Dataset Construction: The MultiMath-300K dataset was developed to fuel the training of MultiMath-7B. This dataset is comprehensive, spanning a wide range of K-12 mathematical problems, and includes multimodal content with image captions for vision-language alignment, and detailed chain-of-thought solutions for training the model in stepwise mathematical reasoning.
4. Benchmark Performance: MultiMath-7B demonstrates state-of-the-art performance among open-source models in multimodal mathematical reasoning tasks, outperforming several other models on datasets such as MathVista and MathVerse. It also surpassed existing models in more traditional text-only mathematical benchmarks.

Experimental Results

• Visual Math Benchmarks: MultiMath-7B excelled in tasks requiring both visual and mathematical reasoning, achieving higher accuracy in tasks involving geometry problem-solving and mathematical word problems compared to other models.
• Textual Math Benchmarks: The model maintained strong performance for text-based mathematical reasoning, providing competitive results against specialized mathematical models, particularly in problem-solving from foundational mathematics exams and competitions.

Discussion

The authors highlight the dual advantage of their approach:

• Reasoning Boost: The integration of multimodal training contributes significantly to the enhancement of reasoning capabilities, not only in the visual domain but also improving text-only reasoning tasks.
• Visual Enhancement: Injecting visual reasoning into the mathematical domain aids in forming a more robust problem-solving framework.

Conclusions

The paper concludes with an emphasis on the effectiveness of bridging visual and mathematical reasoning within a singular model framework. Future research directions include expansion to other domains and further fine-tuning techniques to increase the versatility and accuracy of such multimodal LLMs. The researchers provide a comprehensive comparative analysis, suggesting that the novel dataset and training methodologies have tangible impacts on the model's performance across a varied array of reasoning tasks.