---
title: 'ACWM-Phys: Action-Conditioned Video Benchmark'
url: https://www.emergentmind.com/topics/acwm-phys
type: topic
---

# ACWM-Phys: Action-Conditioned Video Benchmark

ACWM-Phys is a benchmark for **action-conditioned video world models (ACWMs)** that evaluates future visual prediction under **explicit control signals** across diverse physical interaction regimes. It was introduced to address the narrow scope of prior ACWM evaluations, which are described as being largely restricted to **egocentric navigation** or **task-specific robotics datasets** with limited physical diversity. In contrast, ACWM-Phys is built in a **fully simulated, controllable environment** and spans **rigid-body dynamics, deformable-object dynamics, particle dynamics, and kinematics**, with explicit **in-distribution (InD)** and **out-of-distribution (OoD)** protocols designed to probe interpolation and generalization under controlled shifts in interaction patterns or scene configurations [2605.08567].

## 1. Conceptual scope and motivation

ACWM-Phys targets the setting in which a model predicts future observations conditioned on past observations and a sequence of actions. The paper characterizes this as the action-conditioned distribution
\[
p(\mathbf{o}_{t+1:t+H}\mid \mathbf{o}_{1:t}, \mathbf{a}_{t:t+H-1}),
\]
or, in latent space,
\[
p(\mathbf{z}_{t+1:t+H}\mid \mathbf{z}_{1:t}, \mathbf{a}_{t:t+H-1}), \qquad \mathbf{z}_t=\mathcal{E}(\mathbf{o}_t).
\]
This formulation is central to ACWMs because the benchmark is not passive video modeling: it evaluates whether a model can predict dynamics induced by control inputs rather than merely extrapolating visual motion [2605.08567].

The benchmark’s stated motivation is that existing evaluations do not adequately test generalized physical interaction. The benchmark therefore covers four classes of interaction identified in the paper as relevant to “world understanding”: **rigid-body manipulation**, **deformable dynamics**, **particle systems**, and **kinematic control**. The underlying concern is that diffusion-based world models may appear strong on narrow benchmarks while still relying heavily on **visual appearance patterns and local motion statistics** rather than learning transferable physical structure [2605.08567].

A central design principle is controllability. Because ACWM-Phys is simulation-based, it supports **precise data collection, reproducible evaluation, and systematic analysis** of model capabilities. This is important for OoD analysis because the shifts are introduced through physics-relevant parameters rather than through uncontrolled nuisance variation. A plausible implication is that ACWM-Phys is intended as a diagnostic benchmark for *physically grounded* action-conditioned prediction rather than as a proxy for end-to-end embodied intelligence.

## 2. Benchmark construction and task taxonomy

ACWM-Phys contains **eight simulated environments** organized into **four categories**, with roughly **15k trajectories** in total. The appendix-level counts reported in the paper are **14,706 training trajectories**, **750 InD test trajectories**, and **750 OoD test trajectories** [2605.08567].

The task taxonomy is as follows:

| Category | Environments |
|---|---|
| Rigid-body dynamics | Push Cube; Stack Cube |
| Deformable-object dynamics | Push Rope; Cloth Move |
| Particle dynamics | Push Sand; Pour Water |
| Kinematics | Robot Arm; Reacher |

The benchmark is intentionally varied in **object types, control dimensionality, trajectory lengths, contact patterns, and the degree of geometric complexity**. The trajectory horizons range from **50 to 300 frames**. Task-specific training counts are also heterogeneous: **Push Cube**, **Stack Cube**, **Push Rope**, **Cloth Move**, **Robot Arm**, and **Reacher** each have about **1,987 training trajectories**; **Push Sand** has **1,784**; and **Pour Water** has **1,000** [2605.08567].

The physical regimes are differentiated not only by object class but by interaction structure. In the rigid-body category, **Push Cube** tests spatial transport with one to five cubes pushed by a circular pusher, while **Stack Cube** tests pick, transport, and placement. In kinematics, **Robot Arm** focuses on a **7-DoF articulated arm**, whereas **Reacher** evaluates a lower-dimensional articulated system. The deformable tasks involve continuous deformation and contact-rich geometry: **Push Rope** uses flexible rope deformation in PyFlex, and **Cloth Move** pushes cloth over a fixed sphere using two arms. The particle tasks examine granular and fluid behavior through **Push Sand** and **Pour Water** [2605.08567].

The simulator stack is also task-specific. The benchmark uses **PyFlex** for rope, cloth, sand, and water; **Isaac Sim** for **Robot Arm**; **MuJoCo** for **Reacher**; and the paper also references planning with **cuRobo** for **Robot Arm**. Each trajectory includes **image observations, actions, and evaluation labels / split labels** [2605.08567].

## 3. Action-space design and controlled distribution shifts

A defining feature of ACWM-Phys is its **carefully designed action space**, which varies across environments according to control semantics rather than following a uniform action format. The paper stresses that action dimensionality ranges from **low-dimensional simple controls** to **high-dimensional structured controls**, and that this strongly affects generalization [2605.08567].

The environment-specific action definitions reported in the paper are:

| Environment | Action space |
|---|---|
| Push Cube | \(\mathbf{a}\in\mathbb{R}^2\), absolute 2D target position of the pusher |
| Stack Cube | \(\mathbf{a}\in\mathbb{R}^7\), delta 6-DoF end-effector pose plus gripper open/close |
| Push Rope | \(\mathbf{a}\in\mathbb{R}^2\), horizontal pole displacement |
| Cloth Move | \(\mathbf{a}\in\mathbb{R}^3\) in the main benchmark; full 8-D per-arm action space studied in ablation |
| Push Sand | \(\mathbf{a}\in\mathbb{R}^7\), 3D board pose delta plus orientation information |
| Pour Water | \(\mathbf{a}\in\mathbb{R}^4\), Cartesian motion and tilt-angle deltas |
| Robot Arm | \(\mathbf{a}\in\mathbb{R}^7\), per-joint angle delta for a 7-DoF Franka Panda arm |
| Reacher | \(\mathbf{a}\in\mathbb{R}^2\), joint torques for a two-link planar arm |

The InD/OoD protocol is a major part of the benchmark design. Training is performed only on **InD** data, with evaluation on both **InD** and **OoD**. The OoD split is not random; it targets a specific generalization dimension for each environment. Reported examples include **unseen cube counts** or **positions near table edges/corners** in **Push Cube**, held-out **target placement directions** in **Stack Cube**, **rope length changes** in **Push Rope**, **cloth size and initial configuration changes** in **Cloth Move**, substantial **particle count increases** in **Push Sand**, **water level/quantity shifts** in **Pour Water**, **workspace expansion** in **Robot Arm**, and **goal-region shifts to unseen corners** in **Reacher** [2605.08567].

Several OoD shifts are given numerically. For **Push Rope**, training lengths are in \([2.0, 2.8]\) m and the OoD condition is fixed at \(3.1\) m. For **Push Sand**, the OoD split uses roughly **double the particle count** of the InD maximum. For **Robot Arm**, the joint-angle range expands from \([-0.95, 0.58]\) to \([-1.15, 0.88]\). Because the benchmark is fully controlled, these shifts are described as **exact and reproducible** [2605.08567].

This design is important because it separates two questions that are often conflated in video prediction: interpolation within a familiar regime, and generalization to altered physical or geometric structure. ACWM-Phys treats these as separate evaluation axes.

## 4. Evaluation methodology and the ACWM-DiT baseline

The paper evaluates models using standard image prediction metrics—**MSE**, **SSIM**, and **PSNR**—and introduces **Masked-MSE (M-MSE)** to emphasize dynamic regions rather than allowing static background pixels to dominate the error [2605.08567]. Given ground truth \(\mathbf{o}_{1:T}\) and prediction \(\hat{\mathbf{o}}_{1:T}\), the motion map and weight are defined as
\[
m_{h,w} = \max_{t \in [1,T],\, c} \left|\mathbf{o}_{t,c,h,w} - \mathbf{o}_{1,c,h,w}\right|,
\qquad
w_{h,w} = 0.01 + m_{h,w},
\]
and the weighted metric is
\[
\text{M-MSE} = \frac{ \sum_{t,c,h,w} w_{h,w} \left(\hat{\mathbf{o}}_{t,c,h,w} - \mathbf{o}_{t,c,h,w}\right)^2 }{ \sum_{t,c,h,w} w_{h,w} }.
\]
The role of this metric is explicitly to better capture errors in **moving foreground objects** [2605.08567].

The standardized baseline investigated in the paper is **ACWM-DiT**, described as a **latent video diffusion transformer** rather than a novel architecture proposal. The model predicts future latent video tokens conditioned on **past frames**, a **future action sequence**, and the **diffusion timestep**. Its components are a **frozen causal video VAE**, a **DiT-style transformer denoiser**, an **action embedding module**, and **flow matching** training [2605.08567].

The latent prediction formulation is
\[
p(\mathbf{z}_{t+1:t+H}\mid \mathbf{z}_{1:t}, \mathbf{a}_{t:t+H-1}),
\]
and the flow-matching objective uses interpolation between Gaussian noise \(\mathbf{z}_0 \sim \mathcal{N}(0,I)\) and the target future latent \(\mathbf{z}_1=\mathbf{z}^{\mathrm{fut}}\):
\[
\mathbf{z}_\tau = \alpha(\tau)\mathbf{z}_0 + \beta(\tau)\mathbf{z}_1,
\]
\[
\mathcal{L}_{\mathrm{ACWM}}= \mathbb{E} \left\| \mathbf{v}_\theta(\mathbf{z}_\tau,\tau,\mathbf{h}_t) -\dot{\alpha}(\tau)\mathbf{z}_0 -\dot{\beta}(\tau)\mathbf{z}_1 \right\|_2^2,
\qquad
\mathbf{h}_t = \{\mathbf{z}_{1:t}, \mathbf{a}_{t:t+H-1}\}.
\]
The paper notes that the reported setting primarily uses the **current frame as condition** and generates future frames in a **fixed-horizon rollout** [2605.08567].

The visual encoder is a pretrained **Wan 2.1 causal VAE** with **\(8\times\) spatial compression**, **\(4\times\) temporal compression**, and a **16-channel latent space**. For video \(\mathbf{o}_{1:T}\in\mathbb{R}^{T\times 3\times H\times W}\), the encoder produces
\[
\mathbf{z}\in\mathbb{R}^{T_l\times \frac{H}{8}\times \frac{W}{8}\times 16}.
\]
The VAE is **frozen** during training. The denoiser is a **bidirectional DiT** backbone with alternating **spatial self-attention** and **temporal self-attention**, together with **RoPE** positional encoding. Actions are embedded by an MLP, temporally downsampled by a **strided 1D convolution**, and combined with the timestep embedding as
\[
\mathbf{c} = \mathbf{c}_t + \hat{\mathbf{a}},
\]
with conditioning injected into each DiT block via **AdaLN**. The paper also studies an alternative **cross-attention** action-conditioning mechanism [2605.08567].

Model scales are **DiT-S** at about **200M parameters**, **DiT-M** at about **600M parameters**, and **DiT-L** at about **800M parameters**. The main results are reported with **ACWM-DiT-S**. Training uses **AdamW**, learning rate \(10^{-4}\), gradient clipping **1.0**, **100k** training steps, batch size **4**, and **8 H100 GPUs**. Flow matching uses **1000 noise levels**, shift parameter \(s=5.0\), and a Gaussian weighting envelope centered at noise step 500. Input resolution is usually **\(240\times240\)**, except **Push Sand** at **\(240\times400\)**, with latent length **\(T_l = 37\)**. Inference is reported with **50 diffusion steps**, while the appendix suggests **5–10 steps are often enough** for near-saturated performance [2605.08567].

## 5. Empirical findings on interpolation, OoD transfer, and ablations

The principal empirical finding is that **OoD generalization depends not only on the physical regime but also on effective task complexity**. The paper states that models generalize well on **visually simple, low-dimensional interactions with clear geometric structure**, but suffer larger drops on **deformable contacts, high-dimensional control, and complex articulated motion** [2605.08567].

In-distribution performance is reported as strong across all tasks. The easiest tasks are **Push Rope** and **Reacher**, which have very high **SSIM** and low **M-MSE**, while harder InD tasks include **Stack Cube** and **Cloth Move** because of large robot-arm motion and extensive deformation with contact dynamics. Under OoD evaluation, degradation occurs across all tasks but is most severe for **Robot Arm** and **Cloth Move**. More stable OoD behavior is reported for **Push Cube** and **Reacher**, which the paper characterizes as being governed by **low-dimensional geometric constraints** [2605.08567].

The paper’s interpretive conclusion is that task complexity matters more than the coarse physics category itself. Good transfer is associated with tasks that are **visually simple**, **low-dimensional**, and **geometrically constrained**; poor transfer is associated with **deformable contacts**, **high-DoF articulated motion**, and **dense particle systems**. The benchmark therefore distinguishes between a model that can render plausible motion and a model that has internalized physical structure. The paper explicitly states that the observed pattern **suggests that the model still relies heavily on visual appearance patterns instead of fully learning the underlying physics** [2605.08567].

Several ablations support this interpretation.

First, **model scale** improves both InD and especially OoD performance. On **Cloth Move** and **Robot Arm**, the reported ordering is **DiT-S < DiT-M < DiT-L**, although the paper also states that returns diminish at the current data scale [2605.08567].

Second, **action conditioning mechanism** matters. **AdaLN** is reported as sufficient for low-dimensional actions such as those in **Push Cube** and **Push Rope**, where cross-attention yields little or no gain. By contrast, **cross-attention** helps significantly for high-dimensional actions, with a large gain on **Robot Arm** and a modest OoD gain on **Cloth Move**. This suggests that explicit action-token binding is useful when control is high-dimensional and structured [2605.08567].

Third, **latent-space choice** matters. A **temporal causal VAE** outperforms a frame-independent image VAE. The paper specifically reports that **WanVAE** with **\(4\times\)** temporal compression outperforms **FLUX VAE** with **\(1\times\)** temporal compression on **Pour Water** and **Robot Arm**, indicating that temporally aware latent representations help even in visually noisy particle-dynamics settings [2605.08567].

Fourth, **data efficiency** varies by task geometry. **Pour Water** is described as relatively data-efficient because it follows a repeatable motion, whereas **Push Cube** degrades more sharply when data is reduced because it requires broad coverage of workspace and trajectories [2605.08567].

Fifth, **action dimensionality** is a double-edged variable rather than a monotonic benefit. In **Cloth Move**, the richer full per-arm **8-D action space** improves OoD substantially, because it helps the model infer cloth dynamics better. In **Push Cube**, adding a second pusher increases complexity and hurts performance. The paper’s conclusion is that larger action spaces are harder to model but can improve generalization when they provide **richer control signals** [2605.08567].

## 6. Scientific significance, limitations, and relation to adjacent work

The scientific significance of ACWM-Phys lies in benchmark design rather than in proposing a new predictive architecture. The paper explicitly argues that world models should not be judged only by **visually plausible video**. A model may produce realistic-looking trajectories while still failing to represent the underlying **structure, control, and dynamics** required for physically grounded generalization. On this view, ACWM-Phys functions as a stress test for whether action-conditioned generative models actually learn transferable physical interaction rules [2605.08567].

The benchmark also formalizes several practical lessons. It indicates that **temporal latent representations** are preferable to frame-wise encoders for action-conditioned dynamics, that **cross-attention** is valuable for high-dimensional control, and that scaling helps but does not eliminate failure modes in deformable and high-DoF regimes. These findings are presented as guidance for future physically grounded world models rather than as claims that the benchmark itself solves those problems [2605.08567].

The paper is explicit about its limitations. **ACWM-DiT is not real-time**; **ACWM-Phys is simulation-based**; and the evaluation target is **video prediction**, not **closed-loop control or planning**. The benchmark is therefore best understood as a **diagnostic tool** for model analysis. This suggests a boundary on interpretation: strong performance on ACWM-Phys would support action-conditioned predictive competence under controlled physical variation, but would not by itself establish robust decision-making or embodied agency [2605.08567].

Within the broader world-model literature represented in the supplied corpus, ACWM-Phys occupies a complementary position. A related but methodologically distinct line is **ContactGaussian-WM**, which studies a **differentiable physics-grounded rigid-body world model** learned from sparse and contact-rich videos and emphasizes unified Gaussian geometry, differentiable collision detection, and closed-form contact dynamics [2602.11021]. ACWM-Phys, by contrast, is a **benchmark** spanning rigid, deformable, particle, and kinematic regimes for **action-conditioned video prediction**. The juxtaposition highlights two different research roles: benchmark construction for controlled evaluation on the one hand, and physics-grounded model design for specific contact-rich dynamics on the other.

Taken together, the evidence reported for ACWM-Phys supports a narrow but consequential conclusion: current diffusion-based ACWMs can achieve strong prediction on simple rigid or low-dimensional kinematic tasks, yet remain substantially weaker on **deformable objects**, **particle systems**, **high-dimensional control**, and **complex articulated motion**. The benchmark’s central contribution is to make those generalization limits measurable under reproducible, physics-relevant distribution shifts [2605.08567].

Source: https://www.emergentmind.com/topics/acwm-phys