PhysiFormer: Learning to Simulate Mechanics in World Space

Abstract: We present PhysiFormer, a diffusion transformer for physically-plausible 3D object motion. Unlike video world models that operate in view-dependent pixel space, PhysiFormer represents objects as 3D meshes expressed in world coordinates. Given the initial vertex positions and velocities, as well as object material type, rigid or elastic, the model samples future vertex trajectories. While related neural physics approaches build on ad-hoc latent spaces or explicitly enforce rigidity and causality, PhysiFormer shows that excellent results can be obtained without any such inductive biases, by casting vertex trajectory prediction as a single denoising diffusion process directly in world coordinates. The probabilistic formulation captures uncertainty in the learned dynamics, enabling diverse plausible futures from initial conditions, making this framework potentially useful for applications with unobserved uncertainty. The model features attention factorised over time, space, and objects for efficiency, enabling permutation-invariant multi-object reasoning without needing explicit object encoding. Trained on over 100k simulated trajectories, PhysiFormer generates rigid and elastic mechanics, and generalises to mixed-material settings, unseen real-world geometries, and larger object counts. It substantially outperforms autoregressive baselines in trajectory accuracy, rigidity preservation, and momentum-based physical consistency. Our results position coordinate-space diffusion as a promising step toward view-invariant, geometry-aware world modelling for robotics, graphics, and physical design. Visualisations, code, and models are available at https://yimingc9.github.io/physiformer.

• The paper introduces a diffusion transformer that predicts complete mesh vertex trajectories in one pass using geometry-native inputs.
• It leverages factorized spatial, temporal, and per-object attention with a flow-matching objective to capture both rigid and elastic dynamics.
• Empirical results demonstrate significant improvements in shape coherence, momentum drift, and generalization to mixed-material and high-resolution scenarios.

World-Space Diffusion Transformers for 3D Physics Simulation: A Review of "PhysiFormer: Learning to Simulate Mechanics in World Space"

Introduction and Motivation

"PhysiFormer: Learning to Simulate Mechanics in World Space" (2606.27364) addresses the problem of simulating physically-plausible 3D object dynamics generatively and efficiently from purely geometric world-space representations. Conventionally, neural world models for physical systems operate on video or pixel spaces, thereby entangling geometric, photometric, and view-dependent cues and limiting generalization and physical faithfulness. By shifting the generative modeling target to 3D mesh trajectories—explicit vertex motion in world coordinates—PhysiFormer offers a direct, view-agnostic, and geometry-native approach to mechanical prediction, eliminating dependence on engineered object encodings, latent-space simulators, or video-based surrogates.

Figure 1: Overview: Given initial per-vertex positions $X_0$ and velocities $V_0$, and material conditions (rigid, deformable, or mixed), PhysiFormer predicts future vertex trajectories in a single pass, supporting arbitrary mesh topology at inference.

The authors introduce a diffusion transformer that outputs full future mesh vertex trajectories for scenes containing multiple rigid, elastic, or mixed-material objects, attaining strong physical consistency without explicit inductive biases for structure (e.g., rigidity) or causality. Notably, the model’s probabilistic nature supports sampling diverse, plausible futures under uncertainty—a crucial property for agent-centric simulation, graphics, and robotics.

Model Formulation and Diffusion Objective

At the modeling core is an unconditioned score-based diffusion process, parameterized by a transformer. Unlike AR models, which iteratively evolve the system by predicting the next state from the previous, PhysiFormer conditions on the initial vertex positions, velocities, and provided per-object material embeddings and then samples the complete future trajectory in one forward ODE solve. The authors employ a flow-matching objective (JiT-style x-prediction with v-loss), which allows the network to denoise noised mesh vertex sequences by learning their flow velocity during trajectory refinement. This hybridizes the favorable properties of diffusion (robust generation, ODE inference, uncertainty quantification) with the explicit world-space geometric structure.

Figure 2: Architecture: Input mesh vertex coordinates (with noise injected by the flow-matching schedule) are linearly embedded, conditioned on initial geometry, velocity, and material embeddings, and processed by a diffusion transformer with global registers and factorized attention.

Encoding is accomplished by flattening the $T \times N \times 3$ space-time vertex tensor into a token stream, where $T$ is the number of time steps, $N$ the number of mesh vertices, and each frame is projected into a high-dimensional feature space. Conditioning on the first-frame positions/velocities and material type is done via additive embeddings. The absence of object-identity tokens is mitigated via the architectural design, which leverages attention factorization.

Transformer Architecture and Structured Attention

Computational feasibility and inductive structure are achieved through factorized self-attention modules. Rather than standard quadratic self-attention over all space-time tokens, PhysiFormer alternates spatial, temporal, and per-object (object-level) attention, allowing joint reasoning over vertices in a frame, the time axis for each vertex, and inter-object relationships, with special care to preserve permutation invariance with respect to object ordering.

Rotary positional encoding (RoPE) is used for temporal and spatial axes, with spatial RoPE parameterized by relative 3D positions, allowing the model to exploit geometric locality and translation invariance. This, combined with shared register tokens for global context aggregation, allows the model to learn object identities and long-range dependencies without explicit object labels or hand-coded inductive biases.

Figure 3: Qualitative comparison: Against AR baselines on 10k-rigid dataset, AR models suffer from error accumulation and rigidity loss, while PhysiFormer maintains shape coherence and plausible multi-object motion.

Dataset Construction and Training

Training relies on synthetic datasets generated with the Genesis simulator, capturing rigid and elastic body mechanics with a variety of primitive and nontrivial mesh templates, initial conditions, and physical environments. Four datasets span floor and airborne initializations, varied object counts (including out-of-distribution quantities at test time), and both uniform and mixed material compositions. All mesh trajectories are provided as ground truth for supervision, and all training scenes are restricted to single-material settings (with generalization evaluated on mixed-material scenes).

Training uses AdamW, bf16 AMP, and 50-step Heun ODE sampling at inference for stable and efficient trajectory synthesis.

Empirical Results and Generalization

Quantitative evaluation demonstrates that PhysiFormer significantly outperforms strong autoregressive mesh- and particle-based baselines (including TIE variants) on mean squared trajectory error, rigidity preservation (Kabsch alignment deviation), and momentum drift across both short (10-frame) and long (49-frame) prediction horizons. On the 10k rigid-body dataset, the one-shot model attains order-of-magnitude lower error on shape and momentum metrics, and retains physically plausible coherence, where all AR baselines degrade to deformation or divergence under multi-step rollout.

Figure 4: The model generalizes to mixed-material scenes and complex real-world geometries not seen in training, inferring plausible multi-object, multi-material dynamics directly.

Beyond quantitative benchmarks, the diffusion transformer generalizes robustly beyond its training domain: to complex real-world geometries with higher mesh resolutions, mixed combinations of rigid and elastic objects per scene, and to larger object counts than ever observed during training. The architecture’s factorized attention directly enables scaling to more objects via implicit object tokenization.

Figure 5: Generalization: -L-10k predicts trajectories for unseen geometries and out-of-distribution object counts (t = 0, 15, 30, 48). Top: PhysiFormer. Bottom: Best AR model, which fails to preserve shape over long horizons.

Ablations and Physically-Faithful Sampling

Ablations show that reducing the diffusion process’s noise scale (to 0.1, much less than standard Gaussian) is critical when operating in raw world-coordinate space—in alignment with prior findings that coordinate diffusion benefits from reduced noise variance due to the strong conditional dependency on initial states.

When compared to physics simulators, the model demonstrates several empirical advantages: it requires only simple geometric and velocity initialization (no full physical parameters or topology required), maintains a fixed inference cost regardless of modelled material types or object complexity, and is robust to contact and collision artifacts where simulators may fail (including objects escaping boundary boxes at high speed).

Discussion, Limitations and AI Implications

PhysiFormer’s diffusion-in-trajectory-space paradigm establishes a clear path toward geometry-consistent, view-invariant, and agent-ready world models that avoid the pitfalls of video-based physics surrogates—namely, lack of physical consistency, view-dependence, and poor generalization. The strong empirical claims—substantial superiority over equivalent AR baselines in physical consistency, rigidity, and long-range prediction—are robustly supported and motivate further exploration in scaling this methodology for higher-resolution meshes, longer trajectories, and denser heterogeneous scenes.

Despite these strengths, limitations remain: the model is bound by fixed trajectory length and mesh resolution, and physical artifacts such as rare object interpenetration and orientation discontinuities can occur due to the lack of explicit constraint objectives. Potential future extensions include variable-length prediction, mesh upsampling or compression, and explicit contact consistency terms. Furthermore, integrating this coordinate-space diffusion framework into embodied simulation pipelines (robotics, design, content creation) opens prospects for efficient, physical-aware generation, uncertainty-aware planning, and agent-centric reasoning over 3D environments.

Conclusion

PhysiFormer demonstrates that generative mesh trajectory diffusion in world-space, parameterized by a factorized attention transformer, can efficiently and robustly simulate diverse, physically-plausible 3D object dynamics. It subsumes both rigid and elastic mechanics, generalizes across scene complexity and object count, and decisively outperforms autoregressive mesh-based and particle-based baselines on shape, motion, and physicality. This work substantiates the position that coordinate-space diffusion, when combined with strong conditional structure and transformer-based reasoning, is a viable and promising approach for geometry-level world modeling in physical AI systems (2606.27364).