---
title: 'MMBench2: Multimodal Benchmark Dataset'
url: https://www.emergentmind.com/topics/mmbench2-dataset
type: topic
---

# MMBench2: Multimodal Benchmark Dataset

MMBench2 is a large-scale, multimodal benchmark dataset designed for training and evaluating visual world models with a focus on action-controllable, future-predictive generation. It serves as a standardized resource for measuring model performance, diagnosing hallucination phenomena, and evaluating generalization and data efficiency across a diverse set of continuous control, manipulation, and synthetic environments. MMBench2 provides 427 hours of video data spanning 210 tasks, with rich, high-fidelity annotation and a live simulator suite, establishing a comprehensive infrastructure for systematic study and advancement of world modeling research [2606.27326].

## 1. Dataset Composition and Domains

MMBench2 comprises 65,600 expert and behaviorally diverse trajectories, totaling approximately 23 million RGB frames (224×224 resolution, 15 fps), collected across 210 tasks drawn from 10 domains. Of these, 20 million frames (about 370 hours) are allocated for pretraining and 3 million frames (about 57 hours) are held out for testing. Each task includes RGB observations, ground-truth continuous actions (padded to 16 dimensions with per-dimension validity mask), scalar per-step rewards, OpenAI-Gym-style Python API access to a live simulator, and tokenized language instructions (CLIP-ViT/B format) for the task.

The data covers 10 domains:

| Domain                | Example Tasks                    | Modality/Focus          |
|-----------------------|----------------------------------|-------------------------|
| DMControl             | Hopper-Stand, Acrobot Swing-Up   | Classic control         |
| DMControl-Extended    | Giraffe-Run, Spinner-Spin        | Expanded control        |
| Meta-World            | Hammer, Basket-Place             | Manipulation            |
| ManiSkill3            | Dexterous locomotion/manipulation| Locomotion/manipulation |
| MuJoCo                | Standard locomotion              | Locomotion              |
| Box2D                 | Walker/Lander variants           | Locomotion              |
| RoboDesk              | Tabletop push/slide              | Manipulation            |
| OGBench               | Goal-conditioned maze            | Navigation              |
| MiniArcade            | Whirlpool, Dungeon Explorer      | Bespoke arcade games    |
| Continuous Atari      | Atari with continuous actions     | Game-like and control   |

MMBench2's task taxonomy includes 200 pretraining tasks and 10 completely held-out transfer tasks for evaluating out-of-distribution generalization. Per-task frame counts are heavy-tailed, with the top 20 tasks accounting for 26% of the dataset, and per-episode lengths vary from 25 up to 1000 frames. This reflects diverse horizon and difficulty regimes [2606.27326].

## 2. Data Collection Protocols and Annotations

The data collection protocol leverages MMBench single-task expert policies for 200 tasks and augments the dataset with a broad spectrum of behaviors:

- **Random policies**: Uniform random actions sampled over $[-1,1]$.
- **No-op (all-zeros) actions**
- **Expert policy ($\pi^\star$) demonstrations**
- **Transformed expert**: Action scaling, axis flips, and dropout.
- **Structured noise**: Gaussian and Ornstein–Uhlenbeck processes around expert actions.
- **Curiosity-driven rollouts**: World-model planning that maximizes hallucination signal.
- **Human play**: Trajectories collected via a keyboard interface.

Behaviors are mixed within episodes to maximize state-action coverage. All episodes log synchronized RGB frames (224×224, 15 fps), actions, rewards, and per-step simulator metadata (such as ground-truth state, if needed). Annotation adheres to exact action vectors (no pseudo-labels), precise reward labeling, and explicit “policy_type” fields for expert/human identification in metadata.

RGB frames are provided as uint8 PNG or Torch tensors. Actions are stored as $N \times 16$ float32 with corresponding boolean masks, and trajectories are encoded as fixed-length buffers or ragged lists with explicit length headers [2606.27326].

## 3. Structure, Splits, and Formats

MMBench2 follows a hierarchical filesystem-based organization. Each task folder contains standardized subfolders for `train`, `val`, and `test` splits, with individual trajectories represented as directories containing frame sequences, action arrays, rewards, and metadata. Simulators are provided as Python packages or bindings, and task-specific CLIP-tokenized instructions are stored in a central JSON file.

Pretraining tasks feature 260 episodes partitioned 90% train, 5% validation, and 5% test by trajectory. The 10 held-out transfer tasks are split into 50 episodes each for train, validation, and test. Explicit lists of “seen” and “unseen” tasks are provided. All modality formats and metadata schemas are standardized across domains, supporting direct comparability and batching [2606.27326].

## 4. Coverage Metrics and Hallucination Predictors

A central theme in MMBench2 is quantitative assessment of state-action space coverage and prediction of hallucination risk in generative world models.

- **State-action coverage**: Defined as $C(D) = \sum_{(s,a) \in D} w(s,a)\,I[(s,a) \in {\rm Support}(D)]$, for dataset $D$ and weighting $w(s,a)$ (uniform or importance-weighted).
- **Density estimation**: $\hat p(s,a) = \frac{1}{N} \sum_{i=1}^N K((s,a),(s_i,a_i))$ via kernel smoothing or histogram binning.
- **Coverage-aware sampling**: Sampling task or state-action bins uniformly, i.e., $q(t) = 1/T$ for $T=210$, in contrast to frame-proportional sampling.

Three empirically validated hallucination predictors (each normalized by scene motion $m = {\rm RMS}(z_t-z_{t-1})$):

- **Tokenizer residual**: $u_r(z) = \|\hat z - {\rm Encode}({\rm Decode}(\hat z))\|_2$.
- **Flow instability**: $u_f = \frac{1}{K} \sum_{k=K/2}^K \|\hat x_1^{(k)} - \hat x_1^{(k-1)}\|$.
- **Inter-seed variance**: $u_s = \frac{1}{N} \sum_{i=1}^N \|\hat z_i - \bar z\|^2$.

These signals yield high calibration ($\rho \approx 0.8$ Spearman rank with rollout $\Delta$PSNR) and are used both for detection and active mitigation of hallucination [2606.27326].

## 5. Benchmarking Protocols and Model Baselines

A typical MMBench2 benchmarking pipeline uses a two-stage world modeling protocol:

**Pretraining**
- Tokenizer: Masked auto-encoding on all 200 tasks, minimizing masked-MSE plus $0.2 \cdot {\rm LPIPS}$.
- Dynamics: Block-causal Transformer over (frozen) tokenizer latents, with a flow-matching objective ($T=24$, shortcut fraction $\rho=0.25$).
- Reward and behavior-cloning heads downstream, post-pretraining, where ground-truth rewards are present.

**Evaluation Metrics**
- PSNR on reconstructed and rollout frames
- Rollout $\Delta$PSNR vs. frame-repeat baseline
- Action-shuffle ratio (${\rm MSE_{unshuffled}} / {\rm MSE_{shuffled}}$) for action marginalization
- Calibration: Spearman $\rho$ for hallucination predictors vs. realized $\Delta$PSNR
- Downstream closed-loop reward via MPC+CEM (with $H=32$, $K=16$, ${\rm pop}=32$, ${\rm iters}=3$), normalized

**Baselines**
- Dreamer 4 architecture: Transformer-based tokenizer (12 layers, $d_{\rm model} = 512$, 64$\times$64 latents, $\approx 100$M params) and block-causal dynamics transformer (16 layers, $d_{\rm model} = 1024$, 32 spatial tokens, 8 heads, $\approx 250$M params). Total model, including reward/behavior heads, is $\approx 350$M parameters. Training is performed on 8$\times$H100 (58 GPU‐days).

These protocols allow for systematic measurement of model fit, generalization, data efficiency, and robustness to hallucination [2606.27326].

## 6. Access, Extension, and Recommended Practices

MMBench2 is distributed under the CC BY 4.0 license, available at https://nicklashansen.com/mmbench2. Installation is provided via pip (`pip install mmb2`) or by cloning the repository with associated data download scripts; simulators are available as pip submodules (`mmb2[simulators]`). Users are encouraged to cite: “Hansen, N. & Wang, X. (2024). Hallucination in World Models is Predictable and Preventable. ICCV.”

Recommended best practices include:
- Uniform-task sampling during pretraining to enhance state-action coverage.
- Normalization of all coverage and hallucination predictors by scene motion metric $m$.
- For new tasks, logging at least 50 trajectories and employing normalized tokenizer residual $u_r^{\rm norm}$ as a curiosity signal for targeted exploration.
- Evaluation should be on both “seen” and “unseen” tasks to accurately assess generalization capability.

Dataset extension requires task simulators to conform to the common API (reset, step, RGB at 15fps), with actions zero-padded to 16 dimensions and standardized logging schemas. Fine-tuning protocols on new corpora should employ coverage-aware sampling to maximize utility and minimize hallucination [2606.27326].

## 7. Significance and Research Applications

MMBench2 enables detailed study of hallucination in generative world models, allowing isolation of perceptual, action-marginalized, and scene-diverging hallucination modes. Its explicit support for live replay, rich multimodal streams, and rigorous coverage metrics distinguishes it within the landscape of reinforcement learning and world-modeling benchmarks. MMBench2 facilitates the development and evaluation of data-centric, coverage-aware learning algorithms and offers a reproducible testbed for research in data-efficient transfer, generalization, and robust policy learning. A plausible implication is that future research leveraging the dataset's coverage signals and active data collection protocols may yield advances in scalable, general world modeling and robust reinforcement learning [2606.27326].

Source: https://www.emergentmind.com/topics/mmbench2-dataset