Hallucination in World Models is Predictable and Preventable

Abstract: Modern generative world models render increasingly realistic action-controllable futures, yet they frequently hallucinate: rollouts remain visually fluent while drifting from the ground-truth dynamics. We hypothesize that hallucination concentrates in low-coverage regions of the state-action space, where lightweight data-centric signals can both detect it and guide mitigation. To test this, we introduce MMBench2, a 427-hour, 210-task dataset for visual world modeling with ground-truth actions, rewards, and live simulators, and train a 350M-parameter world model on it. We identify three distinct hallucination modes: perceptual, action-marginalized, and scene-diverging -- each anchored to a different stage of the pipeline, and develop three signals that accurately predict where the model will fail. To close coverage gaps at training time, we develop a coverage-aware sampling technique; to close them online, our hallucination predictors serve as curiosity rewards for targeted data collection, yielding a data-efficient finetuning recipe that adapts the pretrained world model to entirely unseen environments with as few as 50 real environment trajectories. Overall, our findings reveal that hallucination in world models is inherently a data coverage issue, and that the same signals used to detect it can also be used for mitigation. An interactive web version of our paper is available at https://www.nicklashansen.com/mmbench2

• The paper introduces a taxonomy of hallucination by systematically identifying predictive signals from internal model metrics such as tokenizer round-trip residual, flow instability, and inter-seed denoising variance.
• It demonstrates that coverage gaps in the diverse MMBench2 dataset are strongly correlated with rollout errors, highlighting the role of data distribution in model failures.
• The research validates that both coverage-aware sampling and curiosity-driven data collection efficiently mitigate hallucination, thus enhancing robust model-based control.

Predictable and Preventable Hallucination in World Models: An In-Depth Analysis

Introduction

The phenomenon of hallucination in generative world models—where rollouts appear visually plausible but deviate from ground-truth environment dynamics—presents a significant obstacle to robust model-based control. This work systematically investigates the underlying causes, identifiable predictive signals, and data-centric mitigations of hallucination in world models, advancing the discussion beyond model-induced limitations toward an evidence-based, coverage-driven perspective. The authors introduce MMBench2, an extensive multitask dataset with dense action and reward annotation, and empirically validate the role of data coverage in hallucination, proposing practical collection and sampling remedies.

Taxonomy and Characterization of Hallucination

The analysis yields a stage-wise taxonomy of hallucination, each mapped to distinct points in the generative pipeline:

1. Perceptual Hallucination: Emanates from the tokenizer (frozen encoder-decoder); out-of-distribution (OOD) states are projected onto near neighbors in the tokenizer's latent space, often causing structural misalignment in reconstructed frames. This effect is isolated to input stage corruption and is invariant to rollout horizon.
2. Action-Marginalized Hallucination: Dependent on the action-conditioning dynamics model, these hallucinations manifest as insensitivity to the provided actions; resulting rollouts are visually coherent yet not controllable, failing to respect the causal effect of interventions.
3. Scene-Diverging Hallucination: Occurs during multi-step rollouts when compounding errors in low-coverage state regions produce physically implausible predictions, such as objects reappearing or state resets inconsistent with the environment’s rules.

Figure 1: Hallucination taxonomy, visualizing perceptual, action-marginalized, and scene-diverging failure modes.

This taxonomy provides diagnostic clarity; interventions must be targeted at data and mechanisms at the respective pipeline stages.

MMBench2: A Diverse Dataset for Visual World Modeling

The MMBench2 dataset is engineered to facilitate the identification and mitigation of hallucination sources. It comprises 65,600 trajectories (~427 hours, 23 million frames at $224\times 224$, 15 fps) across 210 continuous-control tasks spanning ten domains, including manipulation, navigation, classic games, and open-loop control, with dense action and reward annotations and access to live simulators.

Figure 2: Representative sample of 36 out of 210 tasks in MMBench2, demonstrating extensive morphological and perceptual diversity.

Distributional skew is highlighted in Figure 3, where heavy tails (20 tasks comprising 26% of all frames) expose the risk of persistent coverage gaps in lower-task-count regions.

Figure 3: Dataset composition reveals the highly non-uniform distribution of state coverage by task.

Predictors of Hallucination: Mechanistic and Statistical Grounding

Three mechanistically distinct hallucination predictors are derived, each requiring only internal model signals—no additional labels or networks:

• Tokenizer Round-trip Residual ($u_r$): Quantifies divergence upon decoding and re-encoding predicted latents, flagging failures in perceptual fidelity (i.e., drift from the tokenizer’s manifold).
• Flow Instability ($u_f$): Measures variance between denoising predictions during Euler integration substeps; action-marginalized or weakly-conditioned predictions correspond to high $u_f$.
• Inter-seed Denoising Variance ($u_s$): Computes epistemic uncertainty via latent prediction variability across random noise seeds, pinpointing regions likely to exhibit scene divergence.

A critical empirical finding is that all three predictors are tightly correlated with realized rollout error (Spearman $\rho \approx 0.80$), validating their operational use as runtime hallucination detectors.

Figure 4: All three predictors robustly track realized rollout error, supporting their use as label-free detectors.

Coverage-Aware Training and Targeted Data Collection

Two principal mitigation strategies are explored:

• Coverage-aware Sampling: Adjusts pretraining data distribution to uniformly sample across tasks (rather than raw frames), thus addressing inter-task coverage bias. Empirically, coverage-aware sampling improves both low-level reconstruction (PSNR), action sensitivity, and rollout fidelity, confirming the centrality of distributional support over model capacity.
• Targeted Online Collection via Curiosity: The proposed predictors serve as curiosity signals to guide data collection toward historically low-coverage, high-hallucination regions, substantially improving performance with minimal new data. This mechanism is effective on both seen and entirely novel tasks, as evidenced by adaptation to unseen domains with as few as 50 real trajectories.

Figure 5: Hallucinations (high $u_r$) cluster in low-density state regions, mapping directly onto coverage gaps.

Figure 6: State density comparison under expert, curiosity-driven, and human data collection across tasks, highlighting the ability of hallucination-predictor-driven data collection to target coverage gaps.

Ablations show that, for zero-shot generalization, even strong pretrained models exhibit notable hallucination in OOD regimes; however, minimal targeted collection using model-internal uncertainty signals rapidly narrows the transfer gap, approaching human- or expert-collected data efficacy.

Comparative Analysis and Practical Implications

A controlled evaluation demonstrates that off-the-shelf tokenizers, pretrained on massive corpora, achieve superior perceptual generalization in the truly OOD regime, yet task-specific fine-tuning allows in-domain tokenizers to recover and surpass baselines. Thus, in-domain adaptation remains relevant until sufficient scale and heterogeneity are achieved in generic, foundational tokenizers.

Theoretical implications are clear: hallucination is fundamentally a coverage problem, not a capacity or architectural limitation, at the examined scales. These findings dovetail with ongoing work on uncertainty estimation, offline RL coverage theorems, and exploration in RL, emphasizing the importance of data curation, coverage-driven sampling, and active, curiosity-based acquisition policies.

Broader Impacts and Future Directions

Practically, these results motivate the adoption of coverage diagnostics as first-order components in world model pipelines, informing both training and deployment. The open-sourcing of MMBench2, the pretrained model, and interactive data collection interfaces provides an immediate platform for further research, including systematic scaling studies, exploration of real-world robotic perception, and probing extensibility to partially observable or highly stochastic domains.

Potential extensions include the integration of advanced OOD detectors, structured uncertainty quantification for risk-aware planning, and the development of more annotation-efficient collection protocols.

Conclusion

This work establishes that hallucination in modern world models is both predictable and preventable. Coverage-oriented metrics—extractable directly from model internals—enable both robust detection and efficient online mitigation. The MMBench2 dataset, alongside coverage-aware and curiosity-driven collection protocols, provides the empirical foundation to close coverage gaps and reliably adapt large-scale world models to previously unseen environments with minimal additional data.

The methodology reframes hallucination from an intractable nuisance into an actionable failure mode whose incidence—and solution—is fundamentally data-centric. The practical and theoretical trajectories outlined here pave the way for more reliable, adaptive, and scalable generative world models for downstream control and interactive tasks.