ACWM-Phys: Action-Conditioned Video Benchmark

• ACWM-Phys is a benchmark that defines action-conditioned prediction by testing models on future visual signals using explicit control in diverse physical domains.
• It employs fully simulated environments with both in-distribution and out-of-distribution protocols to rigorously evaluate performance across rigid-body, deformable-object, particle, and kinematic dynamics.
• The benchmark offers a diagnostic tool to assess model generalization, interpolation, and the ability to learn transferable physical structures under controlled shifts.

ACWM-Phys is a benchmark for action-conditioned video world models (ACWMs) that evaluates future visual prediction under explicit control signals across diverse physical interaction regimes. It was introduced to address the narrow scope of prior ACWM evaluations, which are described as being largely restricted to egocentric navigation or task-specific robotics datasets with limited physical diversity. In contrast, ACWM-Phys is built in a fully simulated, controllable environment and spans rigid-body dynamics, deformable-object dynamics, particle dynamics, and kinematics, with explicit in-distribution (InD) and out-of-distribution (OoD) protocols designed to probe interpolation and generalization under controlled shifts in interaction patterns or scene configurations (Xue et al., 9 May 2026).

1. Conceptual scope and motivation

ACWM-Phys targets the setting in which a model predicts future observations conditioned on past observations and a sequence of actions. The paper characterizes this as the action-conditioned distribution

$$p(\mathbf{o}_{t+1:t+H}\mid \mathbf{o}_{1:t}, \mathbf{a}_{t:t+H-1}),$$

or, in latent space,

$$p(\mathbf{z}_{t+1:t+H}\mid \mathbf{z}_{1:t}, \mathbf{a}_{t:t+H-1}), \qquad \mathbf{z}_t=\mathcal{E}(\mathbf{o}_t).$$

This formulation is central to ACWMs because the benchmark is not passive video modeling: it evaluates whether a model can predict dynamics induced by control inputs rather than merely extrapolating visual motion (Xue et al., 9 May 2026).

The benchmark’s stated motivation is that existing evaluations do not adequately test generalized physical interaction. The benchmark therefore covers four classes of interaction identified in the paper as relevant to “world understanding”: rigid-body manipulation, deformable dynamics, particle systems, and kinematic control. The underlying concern is that diffusion-based world models may appear strong on narrow benchmarks while still relying heavily on visual appearance patterns and local motion statistics rather than learning transferable physical structure (Xue et al., 9 May 2026).

A central design principle is controllability. Because ACWM-Phys is simulation-based, it supports precise data collection, reproducible evaluation, and systematic analysis of model capabilities. This is important for OoD analysis because the shifts are introduced through physics-relevant parameters rather than through uncontrolled nuisance variation. A plausible implication is that ACWM-Phys is intended as a diagnostic benchmark for physically grounded action-conditioned prediction rather than as a proxy for end-to-end embodied intelligence.

2. Benchmark construction and task taxonomy

ACWM-Phys contains eight simulated environments organized into four categories, with roughly 15k trajectories in total. The appendix-level counts reported in the paper are 14,706 training trajectories, 750 InD test trajectories, and 750 OoD test trajectories (Xue et al., 9 May 2026).

The task taxonomy is as follows:

Category	Environments
Rigid-body dynamics	Push Cube; Stack Cube
Deformable-object dynamics	Push Rope; Cloth Move
Particle dynamics	Push Sand; Pour Water
Kinematics	Robot Arm; Reacher

The benchmark is intentionally varied in object types, control dimensionality, trajectory lengths, contact patterns, and the degree of geometric complexity. The trajectory horizons range from 50 to 300 frames. Task-specific training counts are also heterogeneous: Push Cube, Stack Cube, Push Rope, Cloth Move, Robot Arm, and Reacher each have about 1,987 training trajectories; Push Sand has 1,784; and Pour Water has 1,000 (Xue et al., 9 May 2026).

The physical regimes are differentiated not only by object class but by interaction structure. In the rigid-body category, Push Cube tests spatial transport with one to five cubes pushed by a circular pusher, while Stack Cube tests pick, transport, and placement. In kinematics, Robot Arm focuses on a 7-DoF articulated arm, whereas Reacher evaluates a lower-dimensional articulated system. The deformable tasks involve continuous deformation and contact-rich geometry: Push Rope uses flexible rope deformation in PyFlex, and Cloth Move pushes cloth over a fixed sphere using two arms. The particle tasks examine granular and fluid behavior through Push Sand and Pour Water (Xue et al., 9 May 2026).

The simulator stack is also task-specific. The benchmark uses PyFlex for rope, cloth, sand, and water; Isaac Sim for Robot Arm; MuJoCo for Reacher; and the paper also references planning with cuRobo for Robot Arm. Each trajectory includes image observations, actions, and evaluation labels / split labels (Xue et al., 9 May 2026).

3. Action-space design and controlled distribution shifts

A defining feature of ACWM-Phys is its carefully designed action space, which varies across environments according to control semantics rather than following a uniform action format. The paper stresses that action dimensionality ranges from low-dimensional simple controls to high-dimensional structured controls, and that this strongly affects generalization (Xue et al., 9 May 2026).

The environment-specific action definitions reported in the paper are:

Environment	Action space
Push Cube	$\mathbf{a}\in\mathbb{R}^2$, absolute 2D target position of the pusher
Stack Cube	$\mathbf{a}\in\mathbb{R}^7$, delta 6-DoF end-effector pose plus gripper open/close
Push Rope	$\mathbf{a}\in\mathbb{R}^2$, horizontal pole displacement
Cloth Move	$\mathbf{a}\in\mathbb{R}^3$ in the main benchmark; full 8-D per-arm action space studied in ablation
Push Sand	$\mathbf{a}\in\mathbb{R}^7$, 3D board pose delta plus orientation information
Pour Water	$\mathbf{a}\in\mathbb{R}^4$, Cartesian motion and tilt-angle deltas
Robot Arm	$\mathbf{a}\in\mathbb{R}^7$, per-joint angle delta for a 7-DoF Franka Panda arm
Reacher	$\mathbf{a}\in\mathbb{R}^2$, joint torques for a two-link planar arm

The InD/OoD protocol is a major part of the benchmark design. Training is performed only on InD data, with evaluation on both InD and OoD. The OoD split is not random; it targets a specific generalization dimension for each environment. Reported examples include unseen cube counts or positions near table edges/corners in Push Cube, held-out target placement directions in Stack Cube, rope length changes in Push Rope, cloth size and initial configuration changes in Cloth Move, substantial particle count increases in Push Sand, water level/quantity shifts in Pour Water, workspace expansion in Robot Arm, and goal-region shifts to unseen corners in Reacher (Xue et al., 9 May 2026).

Several OoD shifts are given numerically. For Push Rope, training lengths are in $$p(\mathbf{z}_{t+1:t+H}\mid \mathbf{z}_{1:t}, \mathbf{a}_{t:t+H-1}), \qquad \mathbf{z}_t=\mathcal{E}(\mathbf{o}_t).$$0 m and the OoD condition is fixed at $$p(\mathbf{z}_{t+1:t+H}\mid \mathbf{z}_{1:t}, \mathbf{a}_{t:t+H-1}), \qquad \mathbf{z}_t=\mathcal{E}(\mathbf{o}_t).$$1 m. For Push Sand, the OoD split uses roughly double the particle count of the InD maximum. For Robot Arm, the joint-angle range expands from $$p(\mathbf{z}_{t+1:t+H}\mid \mathbf{z}_{1:t}, \mathbf{a}_{t:t+H-1}), \qquad \mathbf{z}_t=\mathcal{E}(\mathbf{o}_t).$$2 to $$p(\mathbf{z}_{t+1:t+H}\mid \mathbf{z}_{1:t}, \mathbf{a}_{t:t+H-1}), \qquad \mathbf{z}_t=\mathcal{E}(\mathbf{o}_t).$$3. Because the benchmark is fully controlled, these shifts are described as exact and reproducible (Xue et al., 9 May 2026).

This design is important because it separates two questions that are often conflated in video prediction: interpolation within a familiar regime, and generalization to altered physical or geometric structure. ACWM-Phys treats these as separate evaluation axes.

4. Evaluation methodology and the ACWM-DiT baseline

The paper evaluates models using standard image prediction metrics—MSE, SSIM, and PSNR—and introduces Masked-MSE (M-MSE) to emphasize dynamic regions rather than allowing static background pixels to dominate the error (Xue et al., 9 May 2026). Given ground truth $$p(\mathbf{z}_{t+1:t+H}\mid \mathbf{z}_{1:t}, \mathbf{a}_{t:t+H-1}), \qquad \mathbf{z}_t=\mathcal{E}(\mathbf{o}_t).$$4 and prediction $$p(\mathbf{z}_{t+1:t+H}\mid \mathbf{z}_{1:t}, \mathbf{a}_{t:t+H-1}), \qquad \mathbf{z}_t=\mathcal{E}(\mathbf{o}_t).$$5, the motion map and weight are defined as

$$p(\mathbf{z}_{t+1:t+H}\mid \mathbf{z}_{1:t}, \mathbf{a}_{t:t+H-1}), \qquad \mathbf{z}_t=\mathcal{E}(\mathbf{o}_t).$$6

and the weighted metric is

$$p(\mathbf{z}_{t+1:t+H}\mid \mathbf{z}_{1:t}, \mathbf{a}_{t:t+H-1}), \qquad \mathbf{z}_t=\mathcal{E}(\mathbf{o}_t).$$7

The role of this metric is explicitly to better capture errors in moving foreground objects (Xue et al., 9 May 2026).

The standardized baseline investigated in the paper is ACWM-DiT, described as a latent video diffusion transformer rather than a novel architecture proposal. The model predicts future latent video tokens conditioned on past frames, a future action sequence, and the diffusion timestep. Its components are a frozen causal video VAE, a DiT-style transformer denoiser, an action embedding module, and flow matching training (Xue et al., 9 May 2026).

The latent prediction formulation is

$$p(\mathbf{z}_{t+1:t+H}\mid \mathbf{z}_{1:t}, \mathbf{a}_{t:t+H-1}), \qquad \mathbf{z}_t=\mathcal{E}(\mathbf{o}_t).$$8

and the flow-matching objective uses interpolation between Gaussian noise $$p(\mathbf{z}_{t+1:t+H}\mid \mathbf{z}_{1:t}, \mathbf{a}_{t:t+H-1}), \qquad \mathbf{z}_t=\mathcal{E}(\mathbf{o}_t).$$9 and the target future latent $\mathbf{a}\in\mathbb{R}^2$0: $\mathbf{a}\in\mathbb{R}^2$1

$\mathbf{a}\in\mathbb{R}^2$2

The paper notes that the reported setting primarily uses the current frame as condition and generates future frames in a fixed-horizon rollout (Xue et al., 9 May 2026).

The visual encoder is a pretrained Wan 2.1 causal VAE with $\mathbf{a}\in\mathbb{R}^2$3 spatial compression, $\mathbf{a}\in\mathbb{R}^2$4 temporal compression, and a 16-channel latent space. For video $\mathbf{a}\in\mathbb{R}^2$5, the encoder produces

$\mathbf{a}\in\mathbb{R}^2$6

The VAE is frozen during training. The denoiser is a bidirectional DiT backbone with alternating spatial self-attention and temporal self-attention, together with RoPE positional encoding. Actions are embedded by an MLP, temporally downsampled by a strided 1D convolution, and combined with the timestep embedding as

$\mathbf{a}\in\mathbb{R}^2$7

with conditioning injected into each DiT block via AdaLN. The paper also studies an alternative cross-attention action-conditioning mechanism (Xue et al., 9 May 2026).

Model scales are DiT-S at about 200M parameters, DiT-M at about 600M parameters, and DiT-L at about 800M parameters. The main results are reported with ACWM-DiT-S. Training uses AdamW, learning rate $\mathbf{a}\in\mathbb{R}^2$8, gradient clipping 1.0, 100k training steps, batch size 4, and 8 H100 GPUs. Flow matching uses 1000 noise levels, shift parameter $\mathbf{a}\in\mathbb{R}^2$9, and a Gaussian weighting envelope centered at noise step 500. Input resolution is usually $\mathbf{a}\in\mathbb{R}^7$0, except Push Sand at $\mathbf{a}\in\mathbb{R}^7$1, with latent length $\mathbf{a}\in\mathbb{R}^7$2. Inference is reported with 50 diffusion steps, while the appendix suggests 5–10 steps are often enough for near-saturated performance (Xue et al., 9 May 2026).

5. Empirical findings on interpolation, OoD transfer, and ablations

The principal empirical finding is that OoD generalization depends not only on the physical regime but also on effective task complexity. The paper states that models generalize well on visually simple, low-dimensional interactions with clear geometric structure, but suffer larger drops on deformable contacts, high-dimensional control, and complex articulated motion (Xue et al., 9 May 2026).

In-distribution performance is reported as strong across all tasks. The easiest tasks are Push Rope and Reacher, which have very high SSIM and low M-MSE, while harder InD tasks include Stack Cube and Cloth Move because of large robot-arm motion and extensive deformation with contact dynamics. Under OoD evaluation, degradation occurs across all tasks but is most severe for Robot Arm and Cloth Move. More stable OoD behavior is reported for Push Cube and Reacher, which the paper characterizes as being governed by low-dimensional geometric constraints (Xue et al., 9 May 2026).

The paper’s interpretive conclusion is that task complexity matters more than the coarse physics category itself. Good transfer is associated with tasks that are visually simple, low-dimensional, and geometrically constrained; poor transfer is associated with deformable contacts, high-DoF articulated motion, and dense particle systems. The benchmark therefore distinguishes between a model that can render plausible motion and a model that has internalized physical structure. The paper explicitly states that the observed pattern suggests that the model still relies heavily on visual appearance patterns instead of fully learning the underlying physics (Xue et al., 9 May 2026).

Several ablations support this interpretation.

First, model scale improves both InD and especially OoD performance. On Cloth Move and Robot Arm, the reported ordering is DiT-S < DiT-M < DiT-L, although the paper also states that returns diminish at the current data scale (Xue et al., 9 May 2026).

Second, action conditioning mechanism matters. AdaLN is reported as sufficient for low-dimensional actions such as those in Push Cube and Push Rope, where cross-attention yields little or no gain. By contrast, cross-attention helps significantly for high-dimensional actions, with a large gain on Robot Arm and a modest OoD gain on Cloth Move. This suggests that explicit action-token binding is useful when control is high-dimensional and structured (Xue et al., 9 May 2026).

Third, latent-space choice matters. A temporal causal VAE outperforms a frame-independent image VAE. The paper specifically reports that WanVAE with $\mathbf{a}\in\mathbb{R}^7$3 temporal compression outperforms FLUX VAE with $\mathbf{a}\in\mathbb{R}^7$4 temporal compression on Pour Water and Robot Arm, indicating that temporally aware latent representations help even in visually noisy particle-dynamics settings (Xue et al., 9 May 2026).

Fourth, data efficiency varies by task geometry. Pour Water is described as relatively data-efficient because it follows a repeatable motion, whereas Push Cube degrades more sharply when data is reduced because it requires broad coverage of workspace and trajectories (Xue et al., 9 May 2026).

Fifth, action dimensionality is a double-edged variable rather than a monotonic benefit. In Cloth Move, the richer full per-arm 8-D action space improves OoD substantially, because it helps the model infer cloth dynamics better. In Push Cube, adding a second pusher increases complexity and hurts performance. The paper’s conclusion is that larger action spaces are harder to model but can improve generalization when they provide richer control signals (Xue et al., 9 May 2026).

6. Scientific significance, limitations, and relation to adjacent work

The scientific significance of ACWM-Phys lies in benchmark design rather than in proposing a new predictive architecture. The paper explicitly argues that world models should not be judged only by visually plausible video. A model may produce realistic-looking trajectories while still failing to represent the underlying structure, control, and dynamics required for physically grounded generalization. On this view, ACWM-Phys functions as a stress test for whether action-conditioned generative models actually learn transferable physical interaction rules (Xue et al., 9 May 2026).

The benchmark also formalizes several practical lessons. It indicates that temporal latent representations are preferable to frame-wise encoders for action-conditioned dynamics, that cross-attention is valuable for high-dimensional control, and that scaling helps but does not eliminate failure modes in deformable and high-DoF regimes. These findings are presented as guidance for future physically grounded world models rather than as claims that the benchmark itself solves those problems (Xue et al., 9 May 2026).

The paper is explicit about its limitations. ACWM-DiT is not real-time; ACWM-Phys is simulation-based; and the evaluation target is video prediction, not closed-loop control or planning. The benchmark is therefore best understood as a diagnostic tool for model analysis. This suggests a boundary on interpretation: strong performance on ACWM-Phys would support action-conditioned predictive competence under controlled physical variation, but would not by itself establish robust decision-making or embodied agency (Xue et al., 9 May 2026).

Within the broader world-model literature represented in the supplied corpus, ACWM-Phys occupies a complementary position. A related but methodologically distinct line is ContactGaussian-WM, which studies a differentiable physics-grounded rigid-body world model learned from sparse and contact-rich videos and emphasizes unified Gaussian geometry, differentiable collision detection, and closed-form contact dynamics (Wang et al., 11 Feb 2026). ACWM-Phys, by contrast, is a benchmark spanning rigid, deformable, particle, and kinematic regimes for action-conditioned video prediction. The juxtaposition highlights two different research roles: benchmark construction for controlled evaluation on the one hand, and physics-grounded model design for specific contact-rich dynamics on the other.

Taken together, the evidence reported for ACWM-Phys supports a narrow but consequential conclusion: current diffusion-based ACWMs can achieve strong prediction on simple rigid or low-dimensional kinematic tasks, yet remain substantially weaker on deformable objects, particle systems, high-dimensional control, and complex articulated motion. The benchmark’s central contribution is to make those generalization limits measurable under reproducible, physics-relevant distribution shifts (Xue et al., 9 May 2026).