MMWorld Benchmark: Video World Modeling

• MMWorld Benchmark is a comprehensive evaluation suite designed to assess multimodal language models through video-based world modeling tasks across diverse disciplines.
• It rigorously tests causal reasoning, temporal understanding, and domain expertise using structured datasets and multi-faceted question types in over 70 subdisciplines.
• The benchmark leverages both human-annotated and synthetic modalities to identify key performance gaps and guide future advancements in AI’s multi-modal reasoning.

MMWorld Benchmark refers to a rigorous, systematically-constructed evaluation suite targeting the multi-discipline, multi-faceted capabilities of multimodal LLMs (MLLMs) in video-based world modeling. Designed for advanced research in world model assessment, MMWorld establishes significant new requirements for video understanding: rigorous domain coverage, rich reasoning facet taxonomy, and detailed difficulty control. The benchmark is specifically constructed to highlight the gaps in current MLLMs regarding the comprehensive interpretation and causal reasoning about real-world dynamics as embedded in complex video material (He et al., 12 Jun 2024).

1. Motivation and Conceptual Framework

The core motivation for MMWorld is that videos naturally encode rich spatiotemporal context, integrating both visual and auditory signals, thus serving as the most comprehensive medium for evaluating “world models”—systems that must not only perceive what is present but also reason about why events unfold, predict future states, infer counterfactuals, and demonstrate domain expertise across a broad spectrum of tasks. Unlike typical video QA or captioning datasets, MMWorld is structured explicitly to probe both multi-disciplinary and multi-faceted reasoning in real-world scenarios, including causal attribution, procedure understanding, and counterfactual thinking.

Critically, MMWorld addresses two unique challenges:

• Multi-Discipline Coverage: The benchmark encompasses seven major domains—Art & Sports, Business, Science, Health & Medicine, Embodied Tasks, Tech & Engineering, Games—spanning 69 subdisciplines. This broad coverage demands both breadth and depth of reasoning, including fields such as chemistry, robotics, trading, and medicine.
• Multi-Faceted Reasoning: MMWorld explicitly annotates seven types of reasoning that transcend direct perception: Explanation, Counterfactual Thinking, Future Prediction, Domain Expertise, Temporal Understanding, Attribution Understanding, and Procedure Understanding.

2. Dataset Construction and Characteristics

MMWorld contains 1,910 videos in total, divided into a primary human-annotated subset and two synthetic ablation subsets for mono-modal assessment.

• Main human-annotated subset:
  • 417 real-world videos, average length ≈102 seconds, each with Creative Commons licensing.
  • 1,559 multiple-choice question–answer pairs, with an average of 4.05 questions and 3.9 answer options per video.
  • Each question is carefully authored to probe a variety of reasoning types and is supported by a concise caption.
• Synthetic mono-modal subsets:
  • Audio-only subset: 746 videos (2,969 Q-A pairs), generated by extracting transcripts and using GPT-4V for QA/caption synthesis.
  • Visual-only subset: 747 videos (2,099 Q-A pairs), using keyframe extraction and visual summarization.
  • Mono-modal subsets are designed for single-modality ablation: audio- or video-specific reasoning, ensuring models cannot rely on the alternate channel.

Key summary statistics:

Subset	#Videos	#Q-A pairs	Avg Vid Len (s)	Avg Qs/Video	Avg Opts
Human Main	417	1,559	102.3	4.05	3.90
Synthetic I	746	2,969	103.4	3.98	4.00
Synthetic II	747	2,099	115.8	2.81	4.00

Structured pipelines for the synthetic sets prevent information leakage and enable precise analysis of auditory vs. visual comprehension (He et al., 12 Jun 2024).

3. Reasoning Facet Taxonomy and Task Types

MMWorld introduces a precise taxonomy to capture the multi-faceted nature of real-world cognition in video:

1. Explanation: Why a specific action or event occurs, emphasizing causal mechanism inference.
2. Counterfactual Thinking: What would happen if a variable or action changed, probing hypothetical reasoning.
3. Future Prediction: Anticipating subsequent events based on observed dynamics.
4. Domain Expertise: Application of field-specific knowledge (tools, procedures, theory).
5. Temporal Understanding: Measuring event durations, intervals, or rates.
6. Attribution Understanding: Mapping cause-effect pairs across a sequence, often multi-step.
7. Procedure Understanding: Predicting next actions in procedural tasks.

Each question type is anchored by a concrete example and demands not only semantic, but also causal and temporal inference capabilities from MLLMs.

4. Evaluation Metrics and Protocols

Performance is measured primarily in terms of top-choice accuracy:

$$\mathrm{Accuracy} = \frac{\text{\# correct answers}}{\text{Total \# questions}} \times 100\%$$

Metrics are computed not only at the aggregate level but also decomposed per discipline and per reasoning facet, yielding granular analysis of MLLM strengths and deficiencies. No composite or weighted metrics are introduced beyond these averages.

Additionally, facet- and discipline-specific accuracies enable identification of systematic weaknesses, such as lower model performance on Attribution or Procedure reasoning compared to Explanation or Domain Expertise.

5. Baseline Models and Empirical Results

MMWorld baseline evaluation includes 12 advanced MLLMs:

• 2 proprietary (GPT-4V, Gemini Pro),
• 10 open-source (e.g., Video-LLaVA-7B, Video-Chat-7B, ChatUniVi-7B, mPLUG-Owl-7B).

Key performance findings:

Model	Overall Acc.	Science	Embodied	Games
Random	26.3%	26.4%	26.5%	25.2%
GPT-4V	52.3%	66.5%	55.5%	73.5%
Gemini Pro	51.0%	62.8%	43.6%	66.3%
Video-LLaVA-7B	44.6%	56.3%	63.2%	49.0%

• GPT-4V and Gemini Pro top overall accuracy, especially in fields like Business, Science, Health, and Games.
• Video-LLaVA-7B outperforms both proprietary models in Embodied Tasks, underscoring the impact of extensive video-language pretraining.
• Sub-facet analysis: GPT-4V excels in Future Prediction (78.6%), Counterfactual (64.9%), and Domain Expertise (61.1%). Temporal Understanding is highest among open-source models (Video-LLaVA-7B at 34.5%).

A notable finding is that all models perform poorly (typically <40%) on Attribution and Procedure questions, demonstrating the current limits of causal and sequential reasoning in MLLMs (He et al., 12 Jun 2024).

6. Ablation Studies, Insights, and Error Analysis

MMWorld includes several targeted ablations:

• Single-modality ablations: Audio-only and visual-only subsets show that models like Gemini Pro are heavily vision-biased, while Video-Chat achieves >39% in audio-only tasks, reflecting different ASR backbones and audio-handling capacities.
• Difficulty annotation: Human annotators label each question’s difficulty. Both humans and MLLMs demonstrate reduced accuracy as question difficulty increases, but error patterns are only partially correlated. GPT-4V sometimes answers "Expert" questions beyond human reach, but misses some "Easy" items, suggesting MLLM knowledge is orthogonal and complementary to human skills.

Error taxonomy highlights six primary categories: Question Understanding Error, Audio Understanding Error, Visual Perception Error, Hallucination, Reasoning Error, Lack of Domain Knowledge, and explicit Rejection-to-Answer.

7. Limitations, Impact, and Future Directions

MMWorld exposes significant deficiencies in the current generation of MLLMs, including:

• Weakness in domain-specific and multi-step reasoning, particularly in scientific, medical, and technical fields.
• Over-specialization, where certain video models underperform random baselines outside their training discipline.
• Error patterns suggesting a lack of unified, cross-modal world modeling and insufficient causal/temporal inference.

Planned future improvements include integration of structured domain knowledge (e.g., knowledge graphs), explicit temporal modeling (event segmentation, clock reasoning), improvement of cross-modal alignment via unified audio–visual backbones, and advanced instruction-tuning with balanced disciplinary coverage. Development of chain-of-thought or planning-style reasoning strategies is cited as particularly promising for counterfactual and procedural tasks.

In summary, MMWorld advances world model evaluation by requiring comprehensive multi-modal, multi-discipline, and multi-faceted reasoning in video understanding, offering a demanding performance standard for both current and future multimodal systems (He et al., 12 Jun 2024).