Emergent Mind

IsoBench: Benchmarking Multimodal Foundation Models on Isomorphic Representations

(2404.01266)

Published Apr 1, 2024 in cs.AI and cs.CL

Abstract

Current foundation models exhibit impressive capabilities when prompted either with text only or with both image and text inputs. But do their capabilities change depending on the input modality? In this work, we propose $\textbf{IsoBench}$, a benchmark dataset containing problems from four major areas: math, science, algorithms, and games. Each example is presented with multiple $\textbf{isomorphic representations}$ of inputs, such as visual, textual, and mathematical presentations. IsoBench provides fine-grained feedback to diagnose performance gaps caused by the form of the representation. Across various foundation models, we observe that on the same problem, models have a consistent preference towards textual representations. Most prominently, when evaluated on all IsoBench problems, Claude-3 Opus performs 28.7 points worse when provided with images instead of text; similarly, GPT-4 Turbo is 18.7 points worse and Gemini Pro is 14.9 points worse. Finally, we present two prompting techniques, $\textit{IsoCombination}$ and $\textit{IsoScratchPad}$, which improve model performance by considering combinations of, and translations between, different input representations.

IsoBench covers Mathematical Functions, Science, Graph Algorithms, Chess with varied tasks and isomorphic image/text representations.

Overview

IsoBench is a benchmark tool designed for evaluating the performance of multimodal foundation models across different tasks requiring text, image, or combined understanding.
It covers four domains: mathematics, games, algorithms, and science, emphasizing isomorphic representations to test models' capabilities in handling equivalent inputs in various formats.
Findings show a general preference for textual over visual representations among the models, highlighting challenges in vision model integration, input format sensitivity, and multimodal fusion techniques.
IsoBench introduces strategies like IsoCombination and IsoScratchPad to address these gaps, with IsoCombination particularly improving performance by combining multiple representations into a single input.

Evaluating Multimodal Foundation Models with IsoBench: Insights and Challenges

Introduction to IsoBench

IsoBench is a benchmark designed to systematically evaluate the capabilities of multimodal foundation models across a diverse range of tasks that require understanding texts, images, or combinations thereof. This benchmark spans four domains: mathematics, science, algorithms, and games. Unique to IsoBench is its emphasis on isomorphic representations, where the same problem is presented in different modalities, including both visual and textual formats. By doing so, IsoBench provides a granular assessment of how well these models handle semantically equivalent inputs in distinct representations, revealing preferences or biases toward specific modalities.

Domains and Tasks

IsoBench comprises four major domains, each testing different aspects of model capabilities:

Mathematics: Focusing on continuous mathematics and plot understanding, tasks include classifying function properties and identifying breakpoints in piecewise functions.
Games: Chess puzzles and winner identification tasks test strategic reasoning and understanding of complex game states.
Algorithms: Graph algorithms such as connectivity, maximum flow, and isomorphism challenge the models' algorithmic reasoning skills.
Science: Chemistry and physics questions assess the models' understanding of scientific concepts and their ability to interpret diagrams and visual information.

Key Observations and Findings

Across the evaluated multimodal foundation models, a consistent preference for textual representations over visual ones was observed, contradicting human tendencies to benefit from visual information processing. This discrepancy raises questions about the current multimodal fusion mechanisms in these models and their ability to leverage visual inputs effectively. The findings from IsoBench highlight several limitations and challenges:

Vision Model Shortcomings: Visual recognition errors and a lack of capability in utilizing low-level visual features for reasoning suggest that the vision components may not be optimally integrated or trained.
Input Format Sensitivity: Models display varying performance across different textual representations, indicating potential biases or overfitting to specific formats encountered during training.
Multimodal Fusion Gaps: The observed performance gaps between visual and textual representations suggest that current fusion techniques may not effectively leverage the complementary strengths of different modalities.

Addressing the Gaps: IsoCombination and IsoScratchPad

To mitigate the performance discrepancies observed between input modalities, two strategies were introduced: IsoCombination (IsoCB) and IsoScratchPad (IsoSP). IsoCB explores the effect of combining multiple isomorphic representations into a single input, aiming to provide models with a richer set of information. IsoSP, on the other hand, employs a two-step process where a model first translates a visual input into text, leveraging the higher performing text representations for downstream tasks. These strategies showed promising improvements, especially IsoCB, which significantly reduced the performance gap in certain tasks.

Implications and Future Directions

The findings from IsoBench underscore the need for advances in the representations and fusion techniques used by multimodal foundation models to more effectively process and integrate information across modalities. The observed preference for textual inputs points to potential biases in current models, possibly stemming from imbalances in pre-training data or limitations in the models' architectural design.

Future research should focus on developing more sophisticated multimodal fusion mechanisms that can capitalize on the unique advantages of each modality. Additionally, expanding the diversity of tasks and representations in benchmarks like IsoBench will be crucial for comprehensively assessing and improving the capabilities of multimodal foundation models.

In summary, IsoBench brings to light critical challenges in current multimodal foundation models and proposes avenues for research to enhance their understanding and reasoning capabilities across diverse input modalities. With continued development and evaluation, we can move closer to models that truly comprehend and reason with the richness of human communication.