---
title: MMBench2 Visual World Modeling Benchmark
url: https://www.emergentmind.com/topics/mmbench2
type: topic
---

# MMBench2 Visual World Modeling Benchmark

Searching arXiv for MMBench2 and the cited paper to ground the article.
MMBench2 is a large-scale, massively multitask benchmark and dataset for visual world modeling, introduced to support both the training of high-capacity action-conditioned video world models and the systematic analysis and mitigation of hallucination in their rollouts [2606.27326]. It consists of **65,600 trajectories**, **23M frames**, or **427 hours** of RGB video at **224×224** resolution and **15 fps**, spanning **210 continuous-control tasks in 10 domains**, with **ground-truth actions and rewards** and **live simulators** for all tasks. Its stated goals are to understand hallucination in world models at a fine-grained, stage-by-stage level, develop and validate predictors that tell when and where a world model will hallucinate, and use those predictors for mitigation through coverage-aware sampling during pretraining and curiosity-based online data collection plus finetuning.

## 1. Definition and distinguishing characteristics

MMBench2 is positioned as both a dataset and a benchmark protocol for visual world modeling. Compared to prior datasets and benchmarks—offline RL datasets, robot imitation sets, video corpora, and the original MMBench—it is described as the first to jointly provide broad multi-domain continuous-control tasks, mixed-quality behaviors, full action and reward labels, and live environments tightly matched to the dataset. The mixed-quality behaviors explicitly include **random, noisy, expert, human, curiosity-driven** behaviors.

This design supports three linked uses. First, it enables tightly controlled training corpora for action-conditioned world models. Second, it supports offline and online probing of coverage gaps. Third, it connects offline hallucination analysis to downstream control performance. A common misconception addressed by the benchmark design is that visually fluent rollouts are necessarily dynamically faithful. In the benchmark’s framing, hallucination refers precisely to cases where outputs remain visually plausible and fluent while becoming decoupled from the ground-truth dynamics.

| Component | Value | Role |
|---|---:|---|
| Trajectories | 65,600 | Offline corpus |
| Frames | 23M | Training and evaluation data |
| Video duration | 427 hours | Scale of visual coverage |
| Tasks | 210 | Multitask scope |
| Domains | 10 | Cross-domain diversity |
| Labels | Actions, rewards | Action-conditioned and control evaluation |
| Simulators | Live for all tasks | Online collection and MPC evaluation |

The benchmark is therefore not limited to static reconstruction quality. It is structured to evaluate whether model fidelity, hallucination diagnostics, and mitigation strategies transfer to control in live environments.

## 2. Dataset composition, task structure, and splits

The MMBench2 corpus comprises **65,600 trajectories**, **427 hours** of video at **15 fps**, **224×224 RGB** observations, and **210 tasks** covering **10 domains**. Action spaces are continuous, with dimensionality from **1–16**, and are zero-padded to **16** with a validity mask. Each timestep exposes visual observation \(o_t \in \mathbb{R}^{224\times 224 \times 3}\), action \(a_t \in \mathbb{R}^{d_a}\), \(1 \le d_a \le 16\), reward \(r_t \in \mathbb{R}\), and a live simulator implementing transition dynamics
\[
s_{t+1} \sim p(s_{t+1} \mid s_t, a_t),\quad r_t = r(s_t, a_t).
\]
The low-dimensional state \(s_t\) is available but is **not used** for world model training; the setup is image-based.

The 10 domains are **DMControl**, **DMControl Extended**, **Meta-World**, **ManiSkill3**, **MuJoCo**, **MiniArcade**, **Box2D**, **RoboDesk**, **OGBench**, and **Continuous Atari (CALE)**. These domains cover locomotion, dexterous and tabletop manipulation, goal-conditioned navigation, arcade-style games and Atari-like tasks, and physics-driven control. This breadth matters because the dataset is intended to expose both perceptual variability and action-conditioned dynamics variability across heterogeneous environments.

The split structure is central to the benchmark. The **210 tasks** are divided into **200 pretraining tasks** and **10 held-out “unseen” tasks** for transfer and finetuning experiments. The pretraining corpus has approximately **260 episodes per task on average**, but episode lengths are highly heterogeneous, ranging from **25 steps** to **1000 steps**. The resulting frame distribution is heavy-tailed: the top **20 tasks** contribute approximately **26%** of all frames, while the bottom **20 tasks** contribute only **0.7%**. This non-uniformity is explicitly treated as central to the argument that coverage gaps drive hallucination.

The benchmark also specifies protocol splits. Pretraining uses **~20M frames** from the 200 tasks for tokenizer and dynamics training. Testing uses the remaining **~3M frames** from the same 200 tasks for evaluation of reconstructions, rollouts, and hallucination predictors. Transfer and finetuning experiments use **10 “seen” tasks** and **10 unseen tasks**. For finetuning, new data are collected using **expert, random, no-op, curiosity, human** policies, with **50 trajectories per task**.

## 3. Hallucination taxonomy and evaluation protocol

The benchmark formalizes world modeling as a three-stage pipeline:
1. **Encoder**: \(z_t = \mathrm{Encode}(o_t)\)
2. **Dynamics**: \(\hat z_{t+1} = f_\theta(z_{\le t}, a_{\le t})\)
3. **Decoder**: \(\hat o_{t+1} = \mathrm{Decode}(\hat z_{t+1})\)

Within this pipeline, “hallucination” denotes any failure where the output \(\hat o_{t+1:t+H}\) is visually plausible and fluent but decoupled from the true dynamics. The paper distinguishes **three hallucination modes**, each tied to a stage of the pipeline [2606.27326].

**Perceptual hallucination** occurs in the encoder–decoder stage alone, at horizon \(H=0\). The reconstruction of a single observation \(o_t\) is already wrong before any rollout dynamics are applied. Examples given include an unseen maze layout reconstructed as a different but plausible layout, or a novel object mapped onto a similar in-distribution object. Formally, if \(z_t = \mathrm{Encode}(o_t)\) and \(\tilde o_t = \mathrm{Decode}(z_t)\), perceptual hallucination is a large perceptual discrepancy between \(o_t\) and \(\tilde o_t\) that changes semantic structure.

**Action-marginalized hallucination** occurs in one-step dynamics prediction when the predicted next latent \(\hat z_{t+1}\) is essentially insensitive to the input action \(a_t\). The benchmark diagnoses this by comparing teacher-forced one-step prediction error using true actions versus batch-shuffled actions. It defines the **action shuffle ratio**
\[
\text{ASR} = \frac{\text{Flow MSE with shuffled actions}}{\text{Flow MSE with true actions}},
\]
and declares actions “ignored” when
\[
\text{ASR} \le 1.1.
\]

**Scene-diverging hallucination** arises in multi-step autoregressive rollout when error accumulation produces physically implausible events. The benchmark compares rollout quality against a trivial baseline that repeats the last true frame over the horizon. For rollout \(\hat o_{t+1:t+H}\) and ground truth \(o_{t+1:t+H}\), it defines
\[
\Delta\text{PSNR} = \text{PSNR}(\hat o_{t+1:t+H}, o_{t+1:t+H}) - \text{PSNR}(\mathrm{repeat}(o_t), o_{t+1:t+H}),
\]
and labels a rollout as scene-diverging when
\[
\Delta\text{PSNR} \le 0.
\]

The benchmark protocol evaluates world models at several levels. **Recon PSNR (dB)** measures single-frame reconstruction quality. **LPIPS** is used in tokenizer comparisons. **Rollout PSNR gain \(\Delta\text{PSNR}\)** measures rollout fidelity relative to the repeat-last-frame baseline. **ASR** measures action sensitivity. Binary hallucination labels are defined by the thresholds above for **action-ignored** and **scene-diverging** events. To connect model quality to decision making, the benchmark evaluates downstream control using **MPC with CEM**, with planning horizon \(H=32\) and replanning every **16 steps**, and reports normalized task score \(s \in [0,1]\).

## 4. Internal predictors and the coverage hypothesis

A central contribution associated with MMBench2 is the proposal of three internal signals that can be computed from the world model itself, without labels or extra training, and that correlate strongly with hallucination. The raw signals are normalized by scene motion \(m\), defined as RMS latent change at that step, estimated per-task over the dataset or online.

The first signal is the **tokenizer round-trip residual** \(u_r^{\text{norm}}\), defined from
\[
u_r = \left\| \hat z_{t+1} - \mathrm{Encode}\big(\mathrm{Decode}(\hat z_{t+1})\big) \right\|,
\qquad
u_r^{\text{norm}} = \frac{u_r}{m}.
\]
Its stated intuition is that if the decoded frame is off-manifold for the tokenizer, re-encoding it will push it back toward the latent manifold, producing a large residual. It is therefore associated with perceptual hallucination.

The second signal is **flow instability** \(u_f^{\text{norm}}\). The dynamics model is trained as a shortcut flow-matching model that predicts a clean latent through multiple Euler substeps. If \(\hat x_1^{(k)}\) denotes the predicted clean latent at substep \(k\), then schematically
\[
u_f \approx \frac{1}{K/2} \sum_{k=K/2}^K \left\|\hat x_1^{(k)} - \hat x_1^{(k-1)}\right\|,
\qquad
u_f^{\text{norm}} = \frac{u_f}{m}.
\]
Low values indicate rapid convergence to a stable prediction; high values indicate oscillation or drift.

The third signal is **inter-seed variance** \(u_s^{\text{norm}}\). For fixed context and action, the model runs \(N\) independent denoising trajectories with different seeds, producing \(\hat z^{(1)}_{t+1}, \dots, \hat z^{(N)}_{t+1}\), and defines
\[
u_s = \frac{1}{N} \sum_{i=1}^N \left\|\hat z^{(i)}_{t+1} - \bar z_{t+1}\right\|^2,
\qquad
\bar z_{t+1} = \frac{1}{N} \sum_{i=1}^N \hat z^{(i)}_{t+1},
\qquad
u_s^{\text{norm}} = \frac{u_s}{m}.
\]
Its stated interpretation is epistemic uncertainty: if seeds diverge, multi-step rollouts will fan out.

For approximately **9000 held-out 24-frame sequences**, Spearman correlation between rollout \(\Delta\text{PSNR}\) and each predictor is reported as about \(|\rho| \approx 0.80\), strongly negative. As binary classifiers, the reported **AUROC** values for detecting **action-ignored** events (\(\text{ASR} \le 1.1\)) are **0.887** for \(u_r^{\text{norm}}\), **0.868** for \(u_f^{\text{norm}}\), and **0.873** for \(u_s^{\text{norm}}\). For detecting **scene-diverging** events (\(\Delta\text{PSNR} \le 0\)), the corresponding AUROCs are **0.919**, **0.939**, and **0.934**. All three predictors are reported to outperform raw unnormalized versions, latent scene motion \(m\), kNN distance in latent space, and per-task frame count [2606.27326].

These signals are then embedded in a broader **data-centric perspective**. The paper’s central claim is that hallucination in world models is primarily a data coverage issue. Low-coverage regions of the state–action space are those where the empirical visitation density is small. Figure-based evidence is described for tasks such as point maze, cup catch, and lunar lander, where hallucination predictors cluster around the periphery of the visited state distribution. The article’s interpretation is therefore stage-specific: limited scene diversity degrades perceptual generalization, narrow action coverage encourages collapse toward average transitions, and one-step errors in poorly covered regions compound into scene divergence.

## 5. Coverage-aware training and curiosity-driven adaptation

MMBench2 is not only a diagnostic benchmark; it is also used to validate mitigation strategies. The first strategy is **coverage-aware training**, which changes how existing data are sampled. Because the dataset is highly imbalanced across tasks, frame-uniform sampling gives tasks probability proportional to frame count, while task-uniform sampling upweights rare tasks:
\[
p_\text{frame}(i,t) \propto 1 \quad \text{over all frames},
\]
so that \(p(i) \propto n_i\), whereas
\[
p_\text{task}(i) = \frac{1}{200},
\]
with \(t\) sampled uniformly within task \(i\). Explicit loss reweighting was also tested, but sampling rebalancing was found more effective.

Starting from a pretrained base model, the study extends training by **30k steps for tokenizer** and/or **30k for dynamics** under coverage-aware sampling. The reported variants are **Tok ft**, **Dyn ft**, and **Both**. Relative to the base model, the quantitative changes are as follows. For **Recon PSNR**, Tok ft yields **+0.46 dB**, Dyn ft **−0.01 dB**, and Both **+0.44 dB**. For **ASR**, Tok ft yields **+0.02**, Dyn ft **+0.27**, and Both **+0.29**. For **Rollout \(\Delta\text{PSNR}\)**, Tok ft yields **+0.42 dB**, Dyn ft **+0.68 dB**, and Both **+0.88 dB**. The hallucination predictors also improve: \(u_r^{\text{norm}}\) by **−0.20**, \(u_f^{\text{norm}}\) by **−0.07**, and \(u_s^{\text{norm}}\) by **−0.14**. The stated interpretation is that coverage-aware sampling improves perceptual reconstructions, action sensitivity, and multi-step rollout quality simultaneously, with finetuning both tokenizer and dynamics performing best.

The second strategy is **online mitigation via curiosity rewards and finetuning**. The benchmark mainly uses the tokenizer residual \(u_r^{\text{norm}}\) as a curiosity score. For a candidate action sequence \(a_{t:t+H}\), the model imagines a trajectory and computes
\[
J(a_{t:t+H}) = \sum_{\tau = t+1}^{t+H} u_r^{\text{norm}}(\hat z_\tau),
\]
then plans actions that maximize this score. The planner is **CEM**, with horizon **\(H=32\)**, replanning every **\(K=16\)** steps, population size **32**, **3 iterations**, **2 rollouts per candidate**, and warm-start from the **BC prior**.

For the transfer and finetuning experiments, data collection on **10 seen + 10 unseen tasks** uses **No-op actions**, **Random policy**, **Expert policy**, **Human play**, and **Curiosity policy**, with **50 trajectories per task**. Tokenizer and dynamics are then finetuned on these new trajectories, typically for **50k tokenizer** and **30k dynamics** steps, and evaluated both offline and in closed loop using MPC.

On the **10 unseen tasks**, the reported downstream normalized performance is **0.118** for the random policy baseline and **0.276** for the base pretrained model without extra data. Finetuning on curiosity data, with both tokenizer and dynamics finetuned, yields **Recon PSNR 36.05 dB**, **Rollout \(\Delta\text{PSNR}\) +3.00 dB**, **ASR 2.00**, \(u_r^{\text{norm}} = 1.144\), and **task performance 0.325**. For comparison, expert-policy finetuning gives **0.362**, human-play finetuning gives **0.362**, and finetuning on **all data combined** gives **Recon PSNR 37.91 dB**, **Rollout \(\Delta\text{PSNR}\) +4.02 dB**, \(u_r^{\text{norm}} = 0.975\), and **task performance 0.390**. The paper states that curiosity-based data reaches about **90%** of the performance achievable with expert or human trajectories, without human supervision or privileged policies.

## 6. Baseline model, findings, limitations, and research significance

The benchmark is instantiated with a **350M-parameter world model**, architecturally similar to **Dreamer 4**, comprising a **tokenizer** of approximately **100M parameters** and a **dynamics model** of approximately **250M parameters** [2606.27326]. The tokenizer is an encoder–decoder Transformer that patchifies each \(224\times224\) RGB frame with stride **14** into **256 patch tokens**, appends **64 learnable latent queries**, uses a Transformer encoder with \(d_\text{model}=512\), **8 heads**, **12 layers**, and MLP ratio **4**, and produces **64 latent tokens**, each **64-dimensional**, bounded by \(\tanh\):
\[
z \in [-1,1]^{64\times 64}.
\]
Its training objective is masked autoencoding with a mask ratio sampled from \(\mathcal{U}(0,0.9)\), reconstructing masked pixels only, with RMS-normalized MSE and LPIPS loss.

The dynamics model consumes one action token, one shortcut-conditioning token, **32 packed spatial latent tokens**, **4 register tokens**, and optional agent tokens for reward and BC heads. Its Transformer backbone uses block-causal layers with spatial self-attention, temporal causal self-attention, and MLP, with \(d_\text{model}=1024\), **8 heads**, **16 layers**, MLP ratio **4**, **RoPE**, **QK-normalization**, and **RMSNorm**. It is trained with the **shortcut flow-matching objective** with noise levels \(\sigma \in \{0,\dots,k_\text{max}\}\), \(k_\text{max}=64\), a self-consistency bootstrap fraction \(\rho_\text{self}=0.25\), and inference via **shortcut Euler** with step size \(d=0.125\), or **8 substeps per frame**. After pretraining, a reward predictor and a deterministic Gaussian **BC policy** over the **16-d padded action** are added. Training details include **300k** tokenizer steps, **180k** dynamics pretraining steps, sequence length \(T=24\), **AdamW**, learning rate **1e-4**, weight decay **1e-2**, and effective batch sizes **96** for tokenizer and **512** for dynamics.

The principal findings associated with MMBench2 are presented as follows. Hallucination is predictable: the three internal signals are highly correlated with rollout error and produce AUROC values above **0.86** across hallucination modes. Hallucination is coverage-driven: visual analyses show high predictor values concentrated in low-density regions of state space. Coverage-aware sampling works: a single data-centric sampling modification improves reconstructions, action sensitivity, rollout fidelity, and all three hallucination predictors. Curiosity-driven data collection is effective and data-efficient: using hallucination predictors as curiosity rewards to collect **50 trajectories** per unseen task improves unseen-task rollout quality and MPC performance. Pretrained world models transfer zero-shot, but finetuning helps: the base model reaches **0.276** on unseen tasks versus **0.118** for the random baseline, and targeted finetuning closes part of the remaining gap. The paper also reports that off-the-shelf tokenizers such as **Wan 2.1 VAE**, **SD-VAE**, and **Cosmos** perform worse than the in-domain tokenizer on training tasks, while **Wan 2.1** performs better on unseen tasks unless the in-domain tokenizer is finetuned, after which the in-domain tokenizer matches or outperforms it.

The stated limitations are equally important. The study operates at approximately **350M parameters** on **210 simulated tasks**, leaving open whether the findings scale to **billion-parameter** models or **real-world robotics** with noisy sensors and partial observability. Although MMBench2 spans many simulated domains, it does not include real-world data. Training requires substantial computation, reported as approximately **58 GPU days on 8×H100**. Downstream evaluation is centered on MPC-based control over relatively short horizons and specific task sets.

These design choices and findings suggest that MMBench2 functions as a benchmark-and-methodology package for studying hallucination in world models. It standardizes large-scale multitask visual world modeling with full action and reward labels and live simulators, formalizes hallucination into perceptual, action-marginalized, and scene-diverging modes, provides internal label-free predictors, and links offline fidelity metrics to online control performance. A plausible implication is that its main scientific contribution lies not only in scale, but in making coverage a measurable and actionable variable in world-model reliability.

Source: https://www.emergentmind.com/topics/mmbench2