Are They the Same? Exploring Visual Correspondence Shortcomings of Multimodal LLMs (2501.04670v2)

Abstract: Recent advancements in multimodal models have shown a strong ability in visual perception, reasoning abilities, and vision-language understanding. However, studies on visual matching ability are missing, where finding the visual correspondence of objects is essential in vision research. Our research reveals that the matching capabilities in recent multimodal LLMs (MLLMs) still exhibit systematic shortcomings, even with current strong MLLMs models, GPT-4o. In particular, we construct a Multimodal Visual Matching (MMVM) benchmark to fairly benchmark over 30 different MLLMs. The MMVM benchmark is built from 15 open-source datasets and Internet videos with manual annotation. We categorize the data samples of MMVM benchmark into eight aspects based on the required cues and capabilities to more comprehensively evaluate and analyze current MLLMs. In addition, we have designed an automatic annotation pipeline to generate the MMVM SFT dataset, including 220K visual matching data with reasoning annotation. Finally, we present CoLVA, a novel contrastive MLLM with two novel technical designs: fine-grained vision expert with object-level contrastive learning and instruction augmentation strategy. CoLVA achieves 51.06\% overall accuracy (OA) on the MMVM benchmark, surpassing GPT-4o and baseline by 8.41\% and 23.58\% OA, respectively. The results show the effectiveness of our MMVM SFT dataset and our novel technical designs. Code, benchmark, dataset, and models are available at https://github.com/zhouyiks/CoLVA.

• The paper evaluates multimodal LLMs' visual matching abilities using the new MMVM benchmark and finds current models perform poorly (below 50% accuracy).
• A novel model, CoLVA, is proposed which uses object-level contrastive learning and instruction augmentation to improve visual matching performance.
• The findings highlight the critical need to enhance visual correspondence in MLLMs for real-world applications and provide resources for future research.

Analyzing Visual Correspondence in Multimodal LLMs

The paper "Are They the Same? Exploring Visual Correspondence Shortcomings of Multimodal LLMs" presents a comprehensive paper that systematically evaluates the visual matching capabilities of multimodal LLMs (MLLMs). This research identifies a critical gap in the visual correspondence abilities of contemporary MLLMs, despite their notable advancements in various vision-related tasks.

Key Contributions

The authors introduce a Multimodal Visual Matching (MMVM) benchmark, designed explicitly to evaluate the visual matching capabilities of MLLMs. The benchmark is constructed from 15 open-source datasets and internet video samples, leading to a total of 1,510 samples, each annotated with multiple-choice questions. These samples are categorized into eight distinct types based on the cues necessary for effective matching, such as color, shape, and relative position, among others.

To further the development of visual matching abilities in MLLMs, the authors propose a novel model named CoLVA (Contrastive Language Vision Architecture). CoLVA leverages two innovative techniques: object-level contrastive learning and instruction augmentation strategy. These methods aim to enhance the models' fine-grained vision understanding and improve their visual reasoning processes.

Experimental Validation

The evaluation on the MMVM benchmark indicates that current MLLMs, including GPT-4o, encounter significant difficulties when handling visual matching tasks, achieving less than 50% in overall accuracy. The proposed CoLVA, however, demonstrates superior performance, achieving an overall accuracy of 51.06% on the MMVM benchmark, thereby outperforming baseline models such as GPT-4o by a noteworthy margin.

Implications and Future Directions

The findings of this paper carry profound implications for both the practical application and theoretical understanding of MLLMs. Practically, the results highlight the necessity for enhancing visual correspondence capabilities in MLLMs to better support applications such as visual tracking, feature matching, and multi-image understanding tasks. The MMVM benchmark and CoLVA model provide valuable resources and approaches for future researchers aiming to address these deficiencies.

From a theoretical perspective, the paper underscores the importance of comprehensive, category-specific evaluation metrics in understanding the limitations and potentials of MLLMs. The paper suggests a path forward for developing more holistic MLLMs that can integrate detailed visual understanding with linguistic processing.

In conclusion, this research lays the groundwork for future efforts in cultivating more adept multimodal systems, equipped with the nuanced capacity for visual correspondence, and ultimately, better equipped for the complex demands of real-world applications.