ACWM-Phys: Investigating Generalized Physical Interaction in Action-Conditioned Video World Models

Abstract: Action-conditioned world models (ACWMs) have shown strong promise for video prediction and decision-making. However, existing benchmarks are largely restricted to egocentric navigation or narrow, task-specific robotics datasets, offering only limited coverage of the rich physical interactions required for generalized world understanding. We introduce ACWM-Phys, a new benchmark for evaluating action-conditioned prediction under diverse physical dynamics in a clean, controllable simulation environment with a carefully designed action space. ACWM-Phys contains training and evaluation data spanning rigid-body dynamics, kinematics, deformable-object interactions, and particle dynamics. To evaluate both interpolation and generalization, we design in-distribution and out-of-distribution protocols with controlled shifts in interaction patterns or scene configurations. By building the benchmark in a fully controllable simulator, ACWM-Phys enables precise data collection, reproducible evaluation, and systematic analysis of model capabilities for physically grounded world modeling. Through systematic experiments on ACWM-DiT, we find that OoD generalization depends not only on the physical regime but also on effective task complexity: models generalize well on visually simple, low-dimensional interactions with clear geometric structure, but suffer larger drops on deformable contacts, high-dimensional control, and complex articulated motion. This suggests that the model still relies heavily on visual appearance patterns instead of fully learning the underlying physics. Ablations show that cross-attention improves high-dimensional action conditioning, causal VAEs outperform frame-wise encoders, and larger action spaces are harder to model but can improve generalization by providing richer control signals. These findings guide the design of physically grounded world models.

• The paper introduces ACWM-Phys, a novel simulation-based benchmark that evaluates physical generalization in action-conditioned video world models over diverse interaction regimes.
• It employs eight varied environment configurations—from rigid-body to kinematics—to assess both in-distribution and out-of-distribution performance using metrics like SSIM and MSE.
• The study presents ACWM-DiT, a latent video diffusion transformer baseline, and ablation analyses that highlight key factors affecting model performance in complex physical dynamics.

ACWM-Phys: Benchmarking Generalized Physical Interaction in Action-Conditioned Video World Models

Motivation and Problem Statement

Action-conditioned world models (ACWMs) have increasingly become critical components for visual dynamic prediction and policy learning. However, current benchmarks for ACWM evaluation lack coverage of core physical interaction regimes—specifically, their generalization capability over rigid, deformable, particle-based, and kinematic environments remains almost entirely unexamined. They often restrict to egocentric navigation or narrow robotics manipulation, creating benchmarks with limited diversity and minimal physically meaningful distributional shifts. The paper "ACWM-Phys: Investigating Generalized Physical Interaction in Action-Conditioned Video World Models" (2605.08567) addresses this gap by introducing ACWM-Phys—a systematic, simulation-based benchmark tailored for large-scale, quantitative analysis of ACWM performance and generalization across a curated suite of physically diverse tasks.

ACWM-Phys Benchmark Design

ACWM-Phys consists of eight environment configurations, grouped into four major interaction regimes: rigid-body, deformable-object, particle-system, and kinematics. Each environment is defined with precise control over action spaces, distribution shifts, and dynamic regimes, allowing controlled experiments along both in-distribution (InD) and out-of-distribution (OoD) axes (Figure 1).

Figure 1: ACWM-Phys provides diverse physical scenes across InD and OoD protocols, enabling precise study of ACWMs' learning and generalization under varying physical dynamics.

The environments include:

• Rigid-Body: Push Cube and Stack Cube.
• Deformable: Push Rope and Cloth Move.
• Particle: Push Sand and Pour Water.
• Kinematics: Robot Arm and Reacher.

Each task varies object configuration, number, and workspace geometry in OoD settings, challenging the world model to extrapolate physical rules rather than memorize appearance-based statistics. The dataset comprises more than 15k high-resolution simulated trajectories with paired actions and pixel observations (Figure 2).

Figure 2: Four representative frames per task; environments are grouped by interaction category to emphasize structural diversity across the benchmark suite.

ACWM-DiT: Latent Video Diffusion Transformer Baseline

The authors instantiate a strong baseline, ACWM-DiT, using a bidirectional DiT backbone with interleaved spatial and temporal self-attention, operating in a compressed latent space. The architecture follows principles from recent flow-matched video diffusion models, leveraging a pretrained causal VAE for compact spatiotemporal encoding. Actions are embedded via a two-stage MLP+convolution module and injected through AdaLN or cross-attention, providing flexible and scalable conditional control (Figure 3).

Figure 3: The ACWM-DiT latent diffusion transformer architecture, integrating noisy latent tokens with temporally compressed action embeddings.

Training is performed per-environment with 100k steps, distributed over 8 H100 GPUs, using a flow-matching objective with a Gaussian-weighted denoising schedule.

Experimental Analysis: Physical Generalization and Failure Modes

In-Distribution Performance: ACWM-DiT achieves high-fidelity prediction in visually simple, repetitive environments (e.g., SSIM 0.988 for Push Rope, 0.992 for Reacher), even in presence of moderate motion. However, tasks with high-dimensional control or large dynamic deformation (Stack Cube, Cloth Move) are more challenging (SSIM drop, higher MSE on motion-relevant regions).

Out-of-Distribution Generalization: The model's generalization capability is acutely sensitive to the complexity of state-action space and dynamic regime. Significant performance degradation occurs for high-dimensional, contact-rich, or particle-dominated scenes (e.g., $\Delta$SSIM $=-0.067$ on Robot Arm; large MSE increments for Cloth Move and Push Sand on OoD). By contrast, low-dimensional, geometrically simple tasks (Push Cube, Reacher) retain low error under OoD.

Qualitative analyses (e.g., Figure 1, Figure 2 in the appendix) indicate that, under OoD, model predictions remain visually plausible but typically underestimate or hallucinate physically sensitive quantities (e.g., incorrect cloth extent, under-predicted water or sand volume) (see case studies in the appendix).

Ablation Studies: Model and Protocol Insights

Multiple targeted ablations illuminate the mechanisms behind empirical trends:

• Model Scale: Increased capacity favors improved OoD robustness, with diminishing returns between DiT-M and DiT-L at this data scale.
• Action Conditioning Mechanisms: Cross-attention surpasses AdaLN for high-dimensional action spaces (e.g., strong gains on Robot Arm), but shows little effect in low-dimensional settings.
• Latent Space Choice: Causal VAEs with temporal coupling yield consistently superior results over frame-independent VAEs in both InD and OoD, especially for temporally structured tasks.
• Action Dimensionality: Enlarged action spaces pose a learning challenge but, for tasks where actions provide more informative disambiguation (e.g., full dual-arm control on Cloth Move), can significantly enhance generalization.

Performance and Inference Dynamics

The authors provide empirical curves of SSIM and PSNR with respect to diffusion steps. Most tasks saturate after 5-10 steps, indicating high sample efficiency in inference (Figure 4 and Figure 5).

Figure 4: SSIM vs. diffusion steps; InD test (blue), OoD test (red). Performance saturates quickly, indicating sufficient denoising in a few steps.

Figure 5: PSNR vs. diffusion steps; InD test (blue), OoD test (red). The gain per additional inference step rapidly diminishes.

However, the bidirectional flow-matched diffusion design, while effective, remains computationally slower than forward-only/auto-regressive variants. The paper highlights this as a current limitation for downstream real-time or closed-loop applications.

Implications and Future Directions

Theoretical Significance: This work rigorously demonstrates that current video-based ACWMs, despite strong in-distribution accuracy, often fail to extrapolate physical laws under distributional shift—exposing their reliance on visual pattern matching rather than true generalized physical simulation.

Practical Implications: The ACWM-Phys benchmark, with its clean splits and calibrated OoD axes, provides a critical platform for diagnosing bottlenecks in interaction-centric generative models. As visual world models evolve toward richer planning and control applications, systematic evaluation across regimes (particularly those reflecting real-world robot and manipulation complexity) will be indispensable for progress in both algorithmic development and policy learning.

Bridging to Control and Real Environments: The current version is simulation-only and assesses only open-loop video prediction. Integrating these diagnostics with closed-loop policy evaluation, sim-to-real transfer, and robust high-dimensional planning remains open yet essential.

Recommendations for Future Research:

• Incorporate real-sensor noise, actuation uncertainty, and perceptual aliasing.
• Architectures that explicitly encode physical priors (e.g., graph dynamics, constraint-based regularizers) may better capture environment invariances.
• Investigate hybrid training strategies that blend visual realism with symbolic or structured physics.

Conclusion

ACWM-Phys constitutes a substantial step forward in standardizing the evaluation of action-conditioned video world models across diverse physical interaction regimes. The authors' thorough investigation via ACWM-DiT and carefully engineered ablation studies surface both the progress and limitations in physical generalization of pixel-based world models. Future research on more physically informed, robust, and efficient architectures will be essential to close the gap between visual generative models and genuine, generalizable simulation engines for downstream embodied intelligence.