Muffin or Chihuahua? Challenging Multimodal Large Language Models with Multipanel VQA (2401.15847v3)

Abstract: Multipanel images, commonly seen as web screenshots, posters, etc., pervade our daily lives. These images, characterized by their composition of multiple subfigures in distinct layouts, effectively convey information to people. Toward building advanced multimodal AI applications, such as agents that understand complex scenes and navigate through webpages, the skill of multipanel visual reasoning is essential, and a comprehensive evaluation of models in this regard is important. Therefore, we introduce Multipanel Visual Question Answering (MultipanelVQA), a novel benchmark comprising 6,600 triplets of questions, answers, and multipanel images that specifically challenge models in comprehending multipanel images. Our evaluation shows that questions in the MultipanelVQA benchmark pose significant challenges to the state-of-the-art Multimodal LLMs (MLLMs) tested, even though humans can attain approximately 99% accuracy on these questions. Distinctively, the MultipanelVQA benchmark features synthetically generated multipanel images specifically crafted to isolate and assess the impact of various factors, such as the layout, on MLLMs' multipanel image comprehension abilities. As a result, in addition to benchmarking the capabilities of MLLMs in understanding multipanel images, we analyze various factors of the multipanel image that affect MLLMs' performance with synthetic data and offer insights for enhancement. Code and data are released at https://sites.google.com/view/multipanelvqa/home.

• The paper presents a novel MultipanelVQA benchmark with 6,600 image-question-answer triplets to rigorously evaluate LVLM capabilities on complex multipanel tasks.
• The analysis reveals that leading LVLMs face significant performance drops due to content interference from multiple subfigures compared to human near-perfect accuracy.
• The study demonstrates that using sequential visual prompts and controlled synthetic data offers actionable insights for improving future LVLM architectures and training methods.

Evaluating the Proficiency of LVLMs in Multipanel Visual Question Answering

The paper "Muffin or Chihuahua? Challenging Large Vision-LLMs with Multipanel VQA" presents a meticulous exploration into the competencies of Large Vision-LLMs (LVLMs) in interpreting complex multipanel images. Authored by Yue Fan, Jing Gu, Kaiwen Zhou, Qianqi Yan, Shan Jiang, Ching-Chen Kuo, Yang Zhao, Xinze Guan, and Xin Eric Wang, this paper introduces a novel benchmark, MultipanelVQA, comprising 6,600 image-question-answer triplets that pose formidable challenges to prevailing LVLMs.

Motivation and Contributions

The primary motivation of this research is anchored in addressing a critical question: can LVLMs effectively decipher multipanel images, a task typically effortless for humans? These multipanel images are ubiquitous, appearing in web screenshots, posters, and other composite visuals, and they possess unique layouts incorporating multiple subfigures. Human performance on understanding these images approaches near-perfect accuracy, approximately 99%, prompting an essential evaluation of current state-of-the-art LVLMs.

To this end, the authors propose the MultipanelVQA benchmark, featuring a combination of real-world multipanel images sourced from web screenshots and posters, and synthetically generated multipanel images explicitly designed to isolate and examine various influencing factors. This benchmark is significant in several ways:

1. Diverse Data Composition: MultipanelVQA comprises both real-world data and synthetic data, with the latter scrupulously generated to control and analyze the impact of subfigure layout, content interference, and visual hints.
2. Holistic Benchmarking: The benchmark tests model performance using three distinct question types per multipanel image, evaluating content reasoning and positional understanding.
3. Error Analysis and Insights: The paper employs synthetic data to conduct a rigorous error analysis, unearthing insights that can guide future enhancements.

Evaluation and Findings

The paper benchmarks several open-source and proprietary LVLMs, such as LLaVA-1.5-13B, MiniGPT4-v2, InstructBLIP, mPLUG-Owl2, GPT-4V, and Gemini Pro Vision, on the MultipanelVQA dataset. The findings are notable:

• Performance Gap: Even top-performing models, such as GPT-4V and Gemini Pro Vision, show substantial gaps when compared to human performance. For instance, Gemini Pro Vision, the best among the tested models, exhibited a notable drop in accuracy from single-panel to multipanel tasks.
• Influence of Subfigure Interference: LVLMs struggle significantly with content interference caused by multiple subfigures. Simplifying multipanel images to contain only the target subfigure markedly improves performance, highlighting susceptibility to neighboring subfigures.
• Impact of Layout and Visual Prompts: Models show varied sensitivity to layout styles and subfigure sizes. Incorporating sequential number captions as visual prompts benefits some models, especially when coupled with explicit reference in the questions.

Practical and Theoretical Implications

From a practical perspective, the findings underscore the necessity of advancing LVLM capabilities to better navigate and interpret multipanel images, given their prevalence in daily digital interactions. This is particularly relevant for applications requiring complex scene understanding and navigation, such as virtual assistants and automated content analysis tools.

Theoretically, the paper contributes rich insights into the attributes impacting LVLM performance. The synthetic data's even distribution of various factors aids in isolating and examining specific elements like layout complexity and background interference, offering valuable directions for refining model architectures and training approaches.

Future Directions

The results suggest several avenues for future work:

1. Enhanced Visual Reasoning: Developing models with more robust visual reasoning capabilities to handle multipanel images effectively.
2. Advanced Visual Prompts: Exploring sophisticated visual prompting methods that integrate cues seamlessly into the model's contextual understanding.
3. Comprehensive Benchmarks: Extending the benchmark to include more diverse and intricate scenarios, further challenging and pushing the boundaries of current LVLMs.

In conclusion, the MultipanelVQA benchmark represents a significant stride in evaluating and understanding the capabilities of LVLMs in handling complex visual tasks. The detailed analysis and insights offered by this paper provide a substantial foundation for future advancements, aiming to bridge the existing performance gap between human-level understanding and current AI models.