MMBench2: Multimodal Benchmark Dataset

• MMBench2 is a large-scale multimodal dataset designed for training and evaluating visual world models, featuring 427 hours of video data across 210 tasks.
• It encompasses diverse domains such as control, manipulation, and game-like environments, with 23 million RGB frames and standardized train/val/test splits.
• The dataset provides rigorous annotations, state-action coverage metrics, and validated hallucination predictors to improve model generalization and data efficiency.

MMBench2 is a large-scale, multimodal benchmark dataset designed for training and evaluating visual world models with a focus on action-controllable, future-predictive generation. It serves as a standardized resource for measuring model performance, diagnosing hallucination phenomena, and evaluating generalization and data efficiency across a diverse set of continuous control, manipulation, and synthetic environments. MMBench2 provides 427 hours of video data spanning 210 tasks, with rich, high-fidelity annotation and a live simulator suite, establishing a comprehensive infrastructure for systematic study and advancement of world modeling research (Hansen et al., 25 Jun 2026).

1. Dataset Composition and Domains

MMBench2 comprises 65,600 expert and behaviorally diverse trajectories, totaling approximately 23 million RGB frames (224×224 resolution, 15 fps), collected across 210 tasks drawn from 10 domains. Of these, 20 million frames (about 370 hours) are allocated for pretraining and 3 million frames (about 57 hours) are held out for testing. Each task includes RGB observations, ground-truth continuous actions (padded to 16 dimensions with per-dimension validity mask), scalar per-step rewards, OpenAI-Gym-style Python API access to a live simulator, and tokenized language instructions (CLIP-ViT/B format) for the task.

The data covers 10 domains:

Domain	Example Tasks	Modality/Focus
DMControl	Hopper-Stand, Acrobot Swing-Up	Classic control
DMControl-Extended	Giraffe-Run, Spinner-Spin	Expanded control
Meta-World	Hammer, Basket-Place	Manipulation
ManiSkill3	Dexterous locomotion/manipulation	Locomotion/manipulation
MuJoCo	Standard locomotion	Locomotion
Box2D	Walker/Lander variants	Locomotion
RoboDesk	Tabletop push/slide	Manipulation
OGBench	Goal-conditioned maze	Navigation
MiniArcade	Whirlpool, Dungeon Explorer	Bespoke arcade games
Continuous Atari	Atari with continuous actions	Game-like and control

MMBench2's task taxonomy includes 200 pretraining tasks and 10 completely held-out transfer tasks for evaluating out-of-distribution generalization. Per-task frame counts are heavy-tailed, with the top 20 tasks accounting for 26% of the dataset, and per-episode lengths vary from 25 up to 1000 frames. This reflects diverse horizon and difficulty regimes (Hansen et al., 25 Jun 2026).

2. Data Collection Protocols and Annotations

The data collection protocol leverages MMBench single-task expert policies for 200 tasks and augments the dataset with a broad spectrum of behaviors:

• Random policies: Uniform random actions sampled over $[-1,1]$.
• No-op (all-zeros) actions
• Expert policy ($\pi^\star$) demonstrations
• Transformed expert: Action scaling, axis flips, and dropout.
• Structured noise: Gaussian and Ornstein–Uhlenbeck processes around expert actions.
• Curiosity-driven rollouts: World-model planning that maximizes hallucination signal.
• Human play: Trajectories collected via a keyboard interface.

Behaviors are mixed within episodes to maximize state-action coverage. All episodes log synchronized RGB frames (224×224, 15 fps), actions, rewards, and per-step simulator metadata (such as ground-truth state, if needed). Annotation adheres to exact action vectors (no pseudo-labels), precise reward labeling, and explicit “policy_type” fields for expert/human identification in metadata.

RGB frames are provided as uint8 PNG or Torch tensors. Actions are stored as $N \times 16$ float32 with corresponding boolean masks, and trajectories are encoded as fixed-length buffers or ragged lists with explicit length headers (Hansen et al., 25 Jun 2026).

3. Structure, Splits, and Formats

MMBench2 follows a hierarchical filesystem-based organization. Each task folder contains standardized subfolders for train, val, and test splits, with individual trajectories represented as directories containing frame sequences, action arrays, rewards, and metadata. Simulators are provided as Python packages or bindings, and task-specific CLIP-tokenized instructions are stored in a central JSON file.

Pretraining tasks feature 260 episodes partitioned 90% train, 5% validation, and 5% test by trajectory. The 10 held-out transfer tasks are split into 50 episodes each for train, validation, and test. Explicit lists of “seen” and “unseen” tasks are provided. All modality formats and metadata schemas are standardized across domains, supporting direct comparability and batching (Hansen et al., 25 Jun 2026).

4. Coverage Metrics and Hallucination Predictors

A central theme in MMBench2 is quantitative assessment of state-action space coverage and prediction of hallucination risk in generative world models.

• State-action coverage: Defined as $$C(D) = \sum_{(s,a) \in D} w(s,a)\,I[(s,a) \in {\rm Support}(D)]$$, for dataset $D$ and weighting $w(s,a)$ (uniform or importance-weighted).
• Density estimation: $$\hat p(s,a) = \frac{1}{N} \sum_{i=1}^N K((s,a),(s_i,a_i))$$ via kernel smoothing or histogram binning.
• Coverage-aware sampling: Sampling task or state-action bins uniformly, i.e., $q(t) = 1/T$ for $T=210$, in contrast to frame-proportional sampling.

Three empirically validated hallucination predictors (each normalized by scene motion $m = {\rm RMS}(z_t-z_{t-1})$):

• Tokenizer residual: $\pi^\star$0.
• Flow instability: $\pi^\star$1.
• Inter-seed variance: $\pi^\star$2.

These signals yield high calibration ($\pi^\star$3 Spearman rank with rollout $\pi^\star$4PSNR) and are used both for detection and active mitigation of hallucination (Hansen et al., 25 Jun 2026).

5. Benchmarking Protocols and Model Baselines

A typical MMBench2 benchmarking pipeline uses a two-stage world modeling protocol:

Pretraining

• Tokenizer: Masked auto-encoding on all 200 tasks, minimizing masked-MSE plus $\pi^\star$5.
• Dynamics: Block-causal Transformer over (frozen) tokenizer latents, with a flow-matching objective ($\pi^\star$6, shortcut fraction $\pi^\star$7).
• Reward and behavior-cloning heads downstream, post-pretraining, where ground-truth rewards are present.

Evaluation Metrics

• PSNR on reconstructed and rollout frames
• Rollout $\pi^\star$8PSNR vs. frame-repeat baseline
• Action-shuffle ratio ($\pi^\star$9) for action marginalization
• Calibration: Spearman $N \times 16$0 for hallucination predictors vs. realized $N \times 16$1PSNR
• Downstream closed-loop reward via MPC+CEM (with $N \times 16$2, $N \times 16$3, $N \times 16$4, $N \times 16$5), normalized

Baselines

• Dreamer 4 architecture: Transformer-based tokenizer (12 layers, $N \times 16$6, 64$N \times 16$764 latents, $N \times 16$8M params) and block-causal dynamics transformer (16 layers, $N \times 16$9, 32 spatial tokens, 8 heads, $$C(D) = \sum_{(s,a) \in D} w(s,a)\,I[(s,a) \in {\rm Support}(D)]$$0M params). Total model, including reward/behavior heads, is $$C(D) = \sum_{(s,a) \in D} w(s,a)\,I[(s,a) \in {\rm Support}(D)]$$1M parameters. Training is performed on 8$$C(D) = \sum_{(s,a) \in D} w(s,a)\,I[(s,a) \in {\rm Support}(D)]$$2H100 (58 GPU‐days).

These protocols allow for systematic measurement of model fit, generalization, data efficiency, and robustness to hallucination (Hansen et al., 25 Jun 2026).

6. Access, Extension, and Recommended Practices

MMBench2 is distributed under the CC BY 4.0 license, available at https://nicklashansen.com/mmbench2. Installation is provided via pip (pip install mmb2) or by cloning the repository with associated data download scripts; simulators are available as pip submodules (mmb2[simulators]). Users are encouraged to cite: “Hansen, N. & Wang, X. (2024). Hallucination in World Models is Predictable and Preventable. ICCV.”

Recommended best practices include:

• Uniform-task sampling during pretraining to enhance state-action coverage.
• Normalization of all coverage and hallucination predictors by scene motion metric $$C(D) = \sum_{(s,a) \in D} w(s,a)\,I[(s,a) \in {\rm Support}(D)]$$3.
• For new tasks, logging at least 50 trajectories and employing normalized tokenizer residual $$C(D) = \sum_{(s,a) \in D} w(s,a)\,I[(s,a) \in {\rm Support}(D)]$$4 as a curiosity signal for targeted exploration.
• Evaluation should be on both “seen” and “unseen” tasks to accurately assess generalization capability.

Dataset extension requires task simulators to conform to the common API (reset, step, RGB at 15fps), with actions zero-padded to 16 dimensions and standardized logging schemas. Fine-tuning protocols on new corpora should employ coverage-aware sampling to maximize utility and minimize hallucination (Hansen et al., 25 Jun 2026).

7. Significance and Research Applications

MMBench2 enables detailed study of hallucination in generative world models, allowing isolation of perceptual, action-marginalized, and scene-diverging hallucination modes. Its explicit support for live replay, rich multimodal streams, and rigorous coverage metrics distinguishes it within the landscape of reinforcement learning and world-modeling benchmarks. MMBench2 facilitates the development and evaluation of data-centric, coverage-aware learning algorithms and offers a reproducible testbed for research in data-efficient transfer, generalization, and robust policy learning. A plausible implication is that future research leveraging the dataset's coverage signals and active data collection protocols may yield advances in scalable, general world modeling and robust reinforcement learning (Hansen et al., 25 Jun 2026).