VideoAutoArena: An Automated Arena for Evaluating Large Multimodal Models in Video Analysis through User Simulation (2411.13281v2)

Abstract: Large multimodal models (LMMs) with advanced video analysis capabilities have recently garnered significant attention. However, most evaluations rely on traditional methods like multiple-choice questions in benchmarks such as VideoMME and LongVideoBench, which are prone to lack the depth needed to capture the complex demands of real-world users. To address this limitation-and due to the prohibitive cost and slow pace of human annotation for video tasks-we introduce VideoAutoArena, an arena-style benchmark inspired by LMSYS Chatbot Arena's framework, designed to automatically assess LMMs' video analysis abilities. VideoAutoArena utilizes user simulation to generate open-ended, adaptive questions that rigorously assess model performance in video understanding. The benchmark features an automated, scalable evaluation framework, incorporating a modified ELO Rating System for fair and continuous comparisons across multiple LMMs. To validate our automated judging system, we construct a 'gold standard' using a carefully curated subset of human annotations, demonstrating that our arena strongly aligns with human judgment while maintaining scalability. Additionally, we introduce a fault-driven evolution strategy, progressively increasing question complexity to push models toward handling more challenging video analysis scenarios. Experimental results demonstrate that VideoAutoArena effectively differentiates among state-of-the-art LMMs, providing insights into model strengths and areas for improvement. To further streamline our evaluation, we introduce VideoAutoBench as an auxiliary benchmark, where human annotators label winners in a subset of VideoAutoArena battles. We use GPT-4o as a judge to compare responses against these human-validated answers. Together, VideoAutoArena and VideoAutoBench offer a cost-effective, and scalable framework for evaluating LMMs in user-centric video analysis.

• The paper introduces VideoAutoArena as a novel benchmarking framework that uses user simulations and an automated judging system to evaluate large multimodal video models.
• It employs a modified ELO rating system and fault-driven question evolution to generate realistic queries, achieving 84.20% real-world likeness and 87.29% alignment with human judgments.
• Experiments reveal significant performance gaps between open-source and proprietary models, underscoring the challenges in dynamic video analysis and the need for advanced multimodal AI solutions.

A Structured Evaluation Framework for Multimodal Video Analysis Models

The paper introduces VideoAutoArena, a novel benchmarking framework for evaluating large multimodal models (LMMs) with advanced video analysis capabilities. VideoAutoArena addresses the shortcomings of traditional evaluation methods that rely heavily on multiple-choice questions and human annotations. These traditional benchmarks often fail to capture the complexity of real-world video analysis tasks. The proposed framework aims to provide a scalable, cost-effective alternative by leveraging user simulations and an automated judging system, ultimately offering insights into LMM capabilities in user-centric scenarios.

VideoAutoArena utilizes a modified ELO Rating System within its peer battle mechanism, automatically comparing responses from competing LMMs. By simulating user personas and generating open-ended questions, this framework avoids the high costs and time constraints associated with human annotations, validating model performance through simulated real-world queries. The paper demonstrates that VideoAutoArena’s questions better mimic real-world user interactions 84.20% of the time, outperforming benchmark questions from VideoMME and LongVideoBench. Additionally, 87.29% of the time, the automatic judging results align with human preferences, which underscores the system's reliability and capacity to distinguish performance differences among LMMs.

The framework also introduces a fault-driven question evolution strategy, progressively increasing question complexity based on model responses. This approach probes model weaknesses by iteratively generating and challenging LMMs with increasingly difficult and context-rich questions. The significance of this strategy is highlighted by demonstrating that evolved questions consistently achieve higher difficulty scores across various evaluation parameters.

Experiments reveal notable performance disparities, showing that open-source LMMs significantly trail behind proprietary models like GPT-4o in video analysis tasks. The performance gap, particularly between proprietary and open-source models, grows with longer video lengths and more challenging questions, emphasizing the challenges open-source models face in processing dynamic, context-heavy video inputs.

VideoAutoArena is complemented by VideoAutoBench, an auxiliary benchmark streamlining response evaluation through comparison with human-labeled answers. This auxiliary system reinforces findings from VideoAutoArena, allowing faster and more accessible comparisons without sacrificing the depth of user-centric assessments.

In conclusion, VideoAutoArena expands the scope of LMM evaluation through its automated, user-centric framework, providing practical insights into model capabilities in video analysis. By focusing on real-world applicability and scalability, this framework paves the way for more robust LMM development while highlighting the need for advancements in open-source technologies. Future work might explore multiturn interactions and multilingual capabilities to further enrich the evaluation process, aligning it with the diverse requirements of video analysis applications. The paper firmly establishes a baseline for developing more sophisticated and context-aware LMMs, ultimately encouraging innovations within the multimodal AI research community.