OmniBench: Towards The Future of Universal Omni-Language Models (2409.15272v3)

Abstract: Recent advancements in multimodal LLMs (MLLMs) have aimed to integrate and interpret data across diverse modalities. However, the capacity of these models to concurrently process and reason about multiple modalities remains inadequately explored, partly due to the lack of comprehensive modality-wise benchmarks. We introduce OmniBench, a novel benchmark designed to rigorously evaluate models' ability to recognize, interpret, and reason across visual, acoustic, and textual inputs simultaneously. We define models capable of such tri-modal processing as omni-LLMs (OLMs). OmniBench is distinguished by high-quality human annotations, ensuring that accurate responses require integrated understanding and reasoning across all three modalities. Our main findings reveal that: i) most OLMs exhibit critical limitations in instruction-following and reasoning capabilities within tri-modal contexts; and ii) most baselines models perform poorly (below 50\% accuracy) even when provided with alternative textual representations of images or/and audio. These results suggest that the ability to construct a consistent context from text, image, and audio is often overlooked in existing MLLM training paradigms. To address this gap, we curate an instruction tuning dataset of 84.5K training samples, OmniInstruct, for training OLMs to adapt to multimodal contexts. We advocate for future research to focus on developing more robust tri-modal integration techniques and training strategies to enhance OLM performance across diverse modalities. The codes and live leaderboard could be found at https://m-a-p.ai/OmniBench.

An Analysis of "OmniBench: Towards The Future of Universal Omni-LLMs"

The paper "OmniBench: Towards The Future of Universal Omni-LLMs" by Yizhi Li et al. introduces a comprehensive benchmark, OmniBench, designed to evaluate multimodal LLMs' (MLLMs) capability to concurrently recognize, interpret, and reason across visual, acoustic, and textual inputs. The authors define models capable of such tri-modal processing as omni-LLMs (OLMs). This benchmark stands apart due to its high-quality human annotations, ensuring that accurate responses necessitate integrated understanding and reasoning across all three modalities.

Key Contributions

1. Benchmark Design and Rigorous Evaluation: The OmniBench benchmark encompasses a diverse range of task types, progressing from fundamental perception (Object Identification) to complex inference (Contextual and Environmental). These tasks necessitate human-like cognitive abilities, such as temporal and logical order understanding, spatial awareness, entity recognition, symbolic processing, and quantitative reasoning. This taxonomy aims to test a wide spectrum of reasoning and cognitive abilities, providing a holistic assessment of MLLMs.
2. Annotation and Quality Control: The authors employed a stringent annotation protocol, involving three stages: initial annotation, human inspection, and model inspection. This process ensured that all annotated instruction-response pairs required information from both image and audio components to be accurately answered. The annotations included detailed rationales for correct answers, explaining the specific information derived from each modality. This rigor in design and quality control underscores the challenge and depth of the benchmark.
3. Experimental Results: The results demonstrated that existing open-source OLMs, such as the UnifiedIO2 series, show critical limitations in integrating tri-modal information. For instance, even the most well-performing open-source OLMs processed visual and acoustic information separately and struggled to leverage increased model capacity effectively. Moreover, the results revealed a general bias towards speech data, suggesting the need for a more balanced approach in future research and training paradigms.
4. Textual Approximation Experiments: To further expand the evaluation framework, the authors conducted textual approximation experiments where audio and images were replaced with text transcripts and captions, respectively. Vision-LLMs showed superior results compared to audio-LLMs in this approximation setting, indicating a potential direction for developing more robust OLMs. These findings highlight the unique challenges and opportunities in processing and understanding combined multimodal information.

Implications and Future Directions

The findings from the OmniBench benchmark reveal substantial gaps in the current state of multimodal LLMs. Several important implications arise from this research:

1. Need for Advanced Multimodal Integration: The performance of existing models indicates a significant need for developing new architectures and methodologies that can seamlessly integrate and reason across multiple modalities. Future research should focus on designing models that inherently understand and process multimodal inputs holistically rather than as separate entities.
2. Balanced and Diverse Training Data: The observed bias towards speech data suggests that the current training datasets may not be adequately balanced across modalities. There is a need for more diverse and representative datasets that encompass a wide array of real-world scenarios involving visual, acoustic, and textual data.
3. Future Benchmarks and Metrics: The comprehensive nature of OmniBench sets a new standard for evaluating multimodal models. Future benchmarks should continue to build on this foundation, incorporating more complex and nuanced tasks that mirror the real-world demands on AI systems.
4. Toward Human-like Multimodal Understanding: The ultimate goal of OmniBench is to drive progress towards models that approach human-like understanding and reasoning with multimodal data. This paper underscores the importance of challenging benchmarks in accelerating the development of such advanced AI systems.

In conclusion, the introduction of OmniBench represents a significant step forward in the evaluation of multimodal LLMs. While current models exhibit notable limitations, OmniBench provides a critical tool for identifying areas of improvement and guiding future research efforts. The comprehensive and rigorous nature of OmniBench will undoubtedly play a pivotal role in the continued advancement of AI towards achieving true omni-understanding capabilities.