Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videos (2501.13826v1)

Abstract: Humans acquire knowledge through three cognitive stages: perceiving information, comprehending knowledge, and adapting knowledge to solve novel problems. Videos serve as an effective medium for this learning process, facilitating a progression through these cognitive stages. However, existing video benchmarks fail to systematically evaluate the knowledge acquisition capabilities in Large Multimodal Models (LMMs). To address this gap, we introduce Video-MMMU, a multi-modal, multi-disciplinary benchmark designed to assess LMMs' ability to acquire and utilize knowledge from videos. Video-MMMU features a curated collection of 300 expert-level videos and 900 human-annotated questions across six disciplines, evaluating knowledge acquisition through stage-aligned question-answer pairs: Perception, Comprehension, and Adaptation. A proposed knowledge gain metric, {\Delta}knowledge, quantifies improvement in performance after video viewing. Evaluation of LMMs reveals a steep decline in performance as cognitive demands increase and highlights a significant gap between human and model knowledge acquisition, underscoring the need for methods to enhance LMMs' capability to learn and adapt from videos.

An Analytical Overview of "Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videos"

The paper "Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videos" introduces Video-MMMU, a novel benchmark designed to evaluate the capability of Large Multimodal Models (LMMs) in acquiring and applying knowledge through video content. The investigation centers on assessing the extent to which these models can progress through the cognitive stages delineated by Bloom's taxonomy: perception, comprehension, and adaptation. As conventional video benchmarks lack a systematic approach to evaluating such competencies, this work fills a critical gap in multimodal learning evaluation by utilizing a curated dataset of professional videos and corresponding questions.

Core Contributions

1. Benchmark Design: Video-MMMU is a multi-disciplinary platform, encompassing 300 educational videos and 900 annotated questions spanning six fields: Art, Business, Science, Medicine, Humanities, and Engineering. Each discipline is explored through three key cognitive stages:
   • Perception: Focused on the identification of crucial information, assessable through visual and auditory recognition tasks.
   • Comprehension: Evaluates the understanding of complex concepts presented in the videos.
   • Adaptation: Challenges models to apply learned knowledge to novel scenarios, effectively translating theoretical knowledge into practical solutions.
2. Knowledge Gain Metric: A novel metric, $\Delta_{\text{knowledge}$, measures the improvement in performance of LMMs from viewing videos. This metric provides a quantitative perspective on how well these models integrate and employ newly acquired information.

Evaluation and Findings

The paper undertakes an extensive evaluation of both proprietary and open-source LMMs using the Video-MMMU benchmark. Notably, it reveals a pronounced decline in model performance as cognitive demands increase from perception to comprehension and ultimately to adaptation. Furthermore, a comparative analysis indicates a substantial performance gap between humans and models in terms of knowledge acquisition. Human participants consistently outperform their machine counterparts, particularly in the adaptation phase, underscoring the models' current limitations in applying acquired knowledge to new problems.

• Quantitative Insights: Human experts achieved a $\Delta_{\text{knowledge}$ improvement of 33.1%, while the leading model, GPT-4o, attained only 15.6%. This significant gap highlights the intrinsic challenges faced by LMMs in learning and adapting knowledge from multimedia sources.
• Error Analysis: A detailed dissection of error types prevalent in adaptation tasks highlights weaknesses in method adaptation and selection, which limit models' efficacy in novel problem-solving contexts pertinent to real-world applications.

Implications and Future Directions

The findings of this paper hold substantial implications for the future development of multimodal AI systems. The evident gap between human and model learning capabilities elucidates areas for improvement, particularly in refining LMMs' abilities to generalize learned knowledge to unfamiliar scenarios. This involves enhancing models' perception accuracy, deepening their comprehension faculties, and optimizing their complex problem-solving strategies.

Furthermore, the inclusion of audio transcripts was found to enhance comprehension performance, suggesting that a comprehensive multimodal approach that leverages both visual and auditory information could substantially improve learning outcomes.

Conclusion

In summary, "Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videos" provides a vital platform for scrutinizing the knowledge acquisition capabilities of LMMs from video media. The benchmark challenges models to transition from simple recognition tasks to complex, adaptive problem-solving, mimicking human learning processes. While current models exhibit some degree of improvement post-video engagement, substantial advancements are necessary for LMMs to achieve human-like learning and adaptation, particularly in dynamically applying video-acquired knowledge to varied real-world tasks. This work sets the stage for advancing multimodal model education through video, representing a crucial step towards facilitating continuous, autonomous knowledge acquisition in AI.

Future research endeavors should aim at bridging the knowledge acquisition gap by focusing on nuanced adaptive learning techniques, incorporating multi-sensory data, and developing robust evaluation benchmarks that reflect the complexities of real-world problem-solving.