MPMWorlds: Material-Point-Method Simulations for Inferring and Extrapolating Physical Dynamics

Abstract: To study the ability to infer physical dynamics from videos and extrapolate them forward in time, we assemble a dataset of 2D Material Point Method (MPM) physical simulations covering rich physical phenomena such as deformable objects, fluids, kinetic objects, and emitters. We study code generation and video diffusion approaches on this dataset, identifying their strengths and weaknesses by varying the amount of physically relevant side information. The code generation model, beyond giving a working demonstration of automatic synthesis of MPM simulations, reveals that such an approach struggles with inferring physical parameters from visual input, but relative to video diffusion, produces physically and temporally stable extrapolations forward in time, while the video diffusion model more strongly identifies geometric properties from visual input but produces physically implausible extrapolations.

• The paper introduces 95,805 rendered 2D Material Point Method simulations across 9,204 scene templates, pairing executable Taichi code, structured configurations, and videos to evaluate physical inference and extrapolation.
• The benchmark shows code-generating VLMs produce more temporally stable rollouts than video diffusion models, achieving a 0.371 versus 0.862 anomaly rate under full configuration, while diffusion attains higher Mask IoU at 0.611 versus 0.512.
• The paper demonstrates that routing predictions through a prefix-quality gate combines both approaches, reducing anomaly rate to 0.001 and improving W-MAE to 0.144 while selecting video diffusion primarily for geometric fidelity.

Overview and motivation

MPMWorlds, introduced by Kovačič and Ellis (2606.01538), is a dataset of 2D Material Point Method (MPM) simulations designed to test whether generative models can infer the physical dynamics of a scene from partial observations and extrapolate them forward in time. The work is motivated by two questions: whether vision-LLMs (VLMs) can synthesize executable simulation code in place of hand-programmed simulators, and whether video diffusion models (VDMs) can serve as implicit world models without any simulation code at all. Unlike prior benchmarks centered on rigid-body dynamics or perceptual plausibility—such as Physion (Bear et al., 2021), VideoPhy (Bansal et al., 2024), or PhyWorldBench (Gu et al., 17 Jul 2025)—this benchmark targets long-horizon continuation of deformable, multi-material dynamics with paired executable programs and rendered rollouts.

The formal task is defined as follows: given a program $\rho$ producing a video $v$ and a prefix $v_{\leq t}$, a model must either recover an equivalent program $\hat\rho$ (code generation) or directly predict $\hat v = \hat v_{\leq t} \parallel \hat v_{>t}$ (video prediction). Models are evaluated under four input regimes that vary the amount of privileged side information $f(\rho)$: frames only, full scene configuration, configuration without material identities, and configuration without positional information.

Dataset construction

Each MPMWorlds instance comprises three artifacts: executable Taichi simulation code implementing the MPM state update without high-level APIs, a structured YAML scene configuration specifying object types, materials, geometry, initial conditions, and appearance, and a rendered 512×512 video at 30 FPS for 10 seconds on a high-contrast background. The final dataset contains 95,805 rendered simulations spanning 9,204 unique Scene Templates.

The construction pipeline is hierarchical and partially synthetic:

• Seed simulations: 35 manually written programs covering kinematic elements (pistons, gates, rotating objects), emitters, and rigid colliders.
• LLM-based expansion: Following the Self-Instruct paradigm (Shaghoulian, 2023), ChatGPT o3 and Gemini 2.5 Pro were prompted with seed examples to generate new programs, filtered by automated checks for runtime failures, trivial outputs, and numerical instability, plus LLM-based verification of program–config consistency. This yields roughly 1k Parent simulations, which are further modified into approximately 9k Child simulations.
• Automated perturbations: Numerical parameters such as object positions are perturbed under spatial constraints to reduce bias toward LLM-chosen initial configurations.

Scenes have a median of 3 objects; 22.5% contain at least two distinct material types and 5.3% contain three or more. Materials span liquids, particle- and FEM-based elastic bodies, plastic, snow, sand (Drucker–Prager elastoplasticity following Klár et al.), viscoplastic and viscoelastic media, plus rigid, kinematic, and emitter objects. One caveat worth noting: because all videos derive from ground-truth simulation code, the benchmark may structurally favor code-generation approaches in ways that may not transfer to natural video—a limitation the authors acknowledge explicitly.

Evaluation methodology

Two model classes are compared. The VLM baseline is Qwen2.5-VL-7B-Instruct, fine-tuned with full supervised finetuning to emit complete Taichi programs given prefix frames, optional side information, and a lightweight program scaffold fixing non-semantic structure. At inference, 15 candidate programs are sampled per example and selected via Best-of-$N$ using metrics computed only on the visible prefix. The VDM baseline is the 14B LongCat video continuation model, adapted with LoRA (full finetuning showed little difference) and conditioned on the same textual scene information where available; a single sample of 50 denoising steps calibrates inference compute against the VLM's 15-program budget.

Because evaluation lacks access to particle trajectories and many scenes involve topology-changing deformation, the authors propose five complementary metrics: Mask IoU (spatial occupancy overlap), Object Collapse Score (worst relative area loss of color-tracked regions within the prediction), Windowed Motion Activity Error (normalized difference in temporally smoothed frame-to-frame activity), Color Total Variation (foreground RGB histogram distance), and Temporal Anomaly Rate (fraction of samples whose maximum excess temporal jump exceeds ground truth). Pixel-level MSE and SEMD are reported but not used as primary metrics due to poor correlation with long-horizon physical consistency—an important methodological point, since standard pixel metrics would misleadingly favor the VDM, which achieves the best MSE (0.019 vs. 0.027 for the VLM under full configuration).

Test splits are partitioned at the Parent level so models never see held-out scene templates during training, with an additional modification validation split preserving partial template overlap to separate memorization from genuine generalization.

Principal findings

Complementary strengths and failure modes

The central empirical result is a sharp dissociation between the two model classes. Code generation substantially outperforms direct video prediction on temporal stability metrics: under full configuration, the VLM's anomaly rate is 0.371 versus 0.862 for the VDM, and its collapse score is 0.163 versus 0.366. Because generated programs execute inside a physically grounded simulator, successful generations preserve object persistence and temporal smoothness unless the code itself becomes numerically unstable. In contrast, VDM predictions frequently exhibit hallucinated motion, disappearing regions, and abrupt appearance changes.

The exception is mIoU, where the VDM consistently wins across all input regimes (0.611 vs. 0.512 under full configuration). Diffusion models preserve coarse spatial occupancy better, but these spatially plausible extrapolations deteriorate temporally. Temporal analysis reinforces this dissociation: moving-average W-MAE grows faster for the VDM than the VLM as the horizon extends, and cumulative collapse curves show VDM collapses occurring early and accumulating throughout the rollout while VLM rollouts stabilize after initial frames.

Sensitivity to structured side information

VLM performance is highly sensitive to structured scene information: removing material or positional information substantially degrades appearance fidelity (CTV rises from 0.188 to 0.267 when positions are removed) and motion consistency. Removing positions causes the VLM to hallucinate incorrect obstacle layouts that alter physical trajectories, indicating that visual state estimation—not dynamics modeling—is its principal bottleneck. Conversely, VDM performance changes comparatively little across input regimes, and in some cases slightly improves when material information is removed, suggesting current video diffusion models make limited use of explicit structured physical information even when provided, and may treat it as noise for pixel-space extrapolation.

Material-level analysis refines this picture. For VLMs, removing material information has little effect on liquid scenes but substantially degrades elastic/plastic, snow, and sand performance, implying fluid behavior is more readily inferred visually whereas history-dependent constitutive models require explicit specification. By material family, VLMs perform strongest on liquids and snow and worst on sand—the implementation-wise hardest constitutive model in the dataset—while VDMs perform worst on elastic/plastic scenes, where maintaining coherent trajectories of bouncing dynamic bodies requires temporally precise geometry and velocity propagation that diffusion rollouts fail to sustain.

Generalization and hybrid routing

Both model classes improve on the modification split relative to fully held-out templates, confirming novel scene structures are harder than variants of familiar ones. The improvement is larger for VLMs, and the VLM–VDM gap persists across splits, indicating it is not explained by memorization alone.

Exploiting the complementary profiles, the authors introduce a lightweight prefix-quality gate: a learned threshold on how well the VLM reconstructs the visible prefix decides whether to trust the VLM extrapolation or fall back to the VDM. Routing improves over either model alone on nearly every metric—for instance, under full configuration the gate achieves W-MAE 0.144 versus 0.254 (VLM) and 0.298 (VDM), and anomaly rate drops to 0.001 by routing 97% of samples to the VLM. Notably, mIoU routes predominantly to the VDM (only 23% VLM selection), yielding the best geometric score of all three options (0.633). This demonstrates that observable prefix reconstruction quality is predictive of long-horizon extrapolation quality, and provides a proof-of-concept for hybrid architectures combining visual state estimation with explicit dynamics representations.

Limitations and open questions

The authors are explicit about scope constraints. The dataset covers only 2D physics and omits materials such as hair, cloth, and highly stiff bodies. All videos originate from simulation code, so conclusions about the VLM–VDM comparison may not transfer to natural video, where no ground-truth program exists. Several questions remain open: whether the asymmetry in sensitivity to structured side information persists in 3D settings; whether VDMs can be trained to actually exploit textual physical parameters rather than ignoring them; and whether the prefix-quality gating result generalizes beyond simulation-derived video, since it depends on the availability of a well-defined reconstruction target.

Conclusion

MPMWorlds contributes a large-scale paired corpus of executable MPM simulations, videos, and structured configurations, together with a controlled evaluation of code generation versus video diffusion for physical inference and extrapolation. Its key finding—that executable program synthesis yields physically stable long-horizon continuations but fails at visual state estimation, while video diffusion preserves geometry but hallucinates dynamics—is quantified across materials, input regimes, and time horizons, and is converted into constructive guidance through a simple routing mechanism that outperforms both constituent models. The benchmark's restriction to synthetic 2D physics bounds the generality of these conclusions, but the controlled setting makes the dissociation between code-space and pixel-space prediction unusually clean.