Multimodal Chain-of-Thought Reasoning in Language Models (2302.00923v5)

Abstract: LLMs have shown impressive performance on complex reasoning by leveraging chain-of-thought (CoT) prompting to generate intermediate reasoning chains as the rationale to infer the answer. However, existing CoT studies have primarily focused on the language modality. We propose Multimodal-CoT that incorporates language (text) and vision (images) modalities into a two-stage framework that separates rationale generation and answer inference. In this way, answer inference can leverage better generated rationales that are based on multimodal information. Experimental results on ScienceQA and A-OKVQA benchmark datasets show the effectiveness of our proposed approach. With Multimodal-CoT, our model under 1 billion parameters achieves state-of-the-art performance on the ScienceQA benchmark. Our analysis indicates that Multimodal-CoT offers the advantages of mitigating hallucination and enhancing convergence speed. Code is publicly available at https://github.com/amazon-science/mm-cot.

Multimodal Chain-of-Thought Reasoning in LLMs

The paper "Multimodal Chain-of-Thought Reasoning in LLMs" introduces Multimodal-CoT, a novel framework leveraging both language and vision modalities to enhance the reasoning capabilities of LLMs. The proposed framework operates through a two-stage process that separates rationale generation from answer inference, aiming to provide more accurate answers by utilizing informed rationales derived from multimodal data.

Introduction

LLMs, such as GPT-3.5, have demonstrated significant prowess in complex reasoning tasks through techniques like chain-of-thought (CoT) prompting. CoT involves generating intermediate reasoning steps to justify answers. However, existing CoT methodologies predominantly focus on textual information. The integration of multimodal data, particularly the fusion of text and images, has not been extensively explored in the context of CoT prompting. The proposed Multimodal-CoT framework endeavors to address this gap, hypothesizing that incorporating images could provide additional context that bolsters reasoning accuracy.

Methodology

Two-Stage Framework

Multimodal-CoT consists of two distinct stages:

1. Rationale Generation: The model generates rationales based on both text and image inputs. This stage leverages a Transformer-based architecture to encode text and vision features, interact them via attention mechanisms, and then fuse them using a gated fusion approach.
2. Answer Inference: In the subsequent stage, the generated rationale from the first stage is concatenated with the original text input. This combined input, along with the original vision features, guides the model to infer the final answer.

Model Architecture

The architecture is built on a Transformer network and is enhanced to handle multimodal inputs:

• Encoding: Text and image inputs are processed through separate encoders, extracting feature representations for both modalities.
• Interaction: An attention mechanism associates text tokens with image patches.
• Fusion: The text and image representations are combined through a gated mechanism, balancing the contributions of each modality.
• Decoding: The fused representation is fed into the Transformer decoder to generate the output, whether it's a rationale or an answer.

Experiments and Results

The proposed Multimodal-CoT framework was evaluated on the ScienceQA benchmark—a dataset featuring 21k multimodal science questions. The model's performance was benchmarked against various state-of-the-art methods, including UnifiedQA and GPT-3.5.

Key findings include:

1. Performance: Multimodal-CoT significantly outperforms the previous state-of-the-art methods, including GPT-3.5, on the ScienceQA benchmark by 16 percentage points (75.17% to 91.68%). This result is especially notable given the model size of less than 1 billion parameters.
2. Vision Features: The research demonstrates that direct integration of vision features, as opposed to using captions, significantly enhances model performance. Specifically, models incorporating vision features showed superior rationale generation and answer accuracy.
3. Commonsense and Logical Errors: Error analysis revealed that the model still struggles with commonsense and logical mistakes, suggesting areas for future improvement. Incorporating more comprehensive vision features and commonsense reasoning could mitigate these issues.

Implications and Future Directions

The paper provides several contributions:

1. Broadening CoT Methodology: By pioneering the integration of multimodal data in CoT prompting, the paper opens new avenues for developing more sophisticated reasoning models.
2. Practical Deployment: The framework demonstrates that effective multimodal CoT reasoning can be achieved with models under 1 billion parameters, facilitating deployment in resource-constrained environments.
3. Human-Comparable Performance: Achieving and surpassing human-level performance in benchmarks like ScienceQA illustrates the potential for these models to be applied in educational and assistive technologies.

Future research can explore the following dimensions:

1. Enhanced Vision-Language Interaction: Developing more advanced mechanisms for multimodal feature fusion could further elevate reasoning accuracy.
2. Incorporating Diverse Modalities: Beyond text and images, integrating other modalities such as audio could extend the applicability of CoT reasoning models.
3. Commonsense Knowledge Integration: Embedding extensive commonsense knowledge bases within the reasoning frameworks could reduce commonsense and logical errors.

Conclusion

This paper introduces a robust framework for enhancing the reasoning capabilities of LLMs by incorporating both textual and visual data. Multimodal-CoT not only demonstrates substantial performance gains over existing models but also sets a precedent for future research in multimodal reasoning frameworks. By addressing current limitations and exploring new modalities, future advancements can further push the boundaries of what LLMs can achieve in complex reasoning tasks.