PhysiFormer: Diffusion Transformer for 3D Mesh Motion

• PhysiFormer is a diffusion Transformer that models physically-plausible 3D object motion by directly predicting full mesh trajectories in world coordinate space.
• Its factorized attention mechanism over time, space, and objects reduces computational cost while preserving rigidity and momentum consistency.
• The model generalizes across mixed-material settings and unseen geometries, outperforming autoregressive baselines in trajectory accuracy and physical plausibility.

PhysiFormer is a diffusion Transformer for physically-plausible 3D object motion that models one or more triangle meshes directly in world coordinate space rather than in view-dependent pixel space. Conditioned on initial vertex positions $X_0$, initial velocities $V_0$, and per-object material labels $y$ indicating rigid or elastic behavior, it samples future vertex trajectories and thereby learns the conditional distribution $p(X \mid X_0, V_0, y)$ over mesh motion. The model is formulated as full-trajectory denoising diffusion in raw coordinates, uses factorised attention over time, space, and objects, and is trained on over 100k simulated trajectories generated in Genesis; reported results show generalisation to mixed-material settings, unseen real-world geometries, and larger object counts, while outperforming autoregressive baselines in trajectory accuracy, rigidity preservation, and momentum-based physical consistency (Chen et al., 25 Jun 2026).

1. Problem formulation and representational scope

PhysiFormer treats a scene as a triangular mesh with $N$ vertices and faces $\mathcal{F} \subset \{1,\ldots,N\}^3$, with multiple objects represented as connected components of the mesh. The state variable is the vertex-position field $X(t) \in \mathbb{R}^{N\times 3}$, and instantaneous velocity is defined as $V(t)=dX/dt|_t \in \mathbb{R}^{N\times 3}$. Time is discretized into $T$ steps, yielding $X,V \in \mathbb{R}^{T\times N\times 3}$ with slices $V_0$0 and $V_0$1. The conditioning variables are the initial conditions $V_0$2 and per-object material labels $V_0$3, broadcast to vertices, that specify whether an object is rigid or elastic (Chen et al., 25 Jun 2026).

This formulation departs from video world models that operate in pixel space. By expressing objects as meshes in a global frame, the method removes ambiguities due to viewpoint, lighting, and occlusions, and makes physical invariants such as rigidity and momentum easier to measure. PhysiFormer predicts future positions directly as a one-shot trajectory sample $V_0$4; when velocities are needed, they may be recovered by finite differences, $V_0$5. The paper contrasts this with autoregressive baselines that instead model $V_0$6 and are therefore exposed to rollout error accumulation.

The world-space choice is also tied to the model’s probabilistic interpretation. Because the learned dynamics are expressed as a trajectory distribution rather than as a single deterministic rollout, the framework can represent uncertainty in unobserved physical properties such as mass and friction. A plausible implication is that the coordinate-space representation and the probabilistic trajectory model are jointly responsible for the reported ability to generate diverse plausible futures from identical initial conditions.

2. Diffusion formulation in coordinate space

PhysiFormer follows the “Just image Transformer” training objective via flow matching in raw coordinates rather than in latent codes. Let $V_0$7 denote a clean trajectory sample, flattened to a vector, and let $V_0$8 be noise. The interpolation between data and noise is

$V_0$9

so that $y$0 is pure noise at $y$1 and clean data at $y$2. The flow velocity is

$y$3

The model predicts clean data $y$4 and plugs it into the velocity field. Training minimizes the $y$5-loss

$y$6

During training, $y$7 is sampled from a logit-normal distribution with $y$8 and $y$9. During inference, samples are obtained by integrating the ODE $p(X \mid X_0, V_0, y)$0 from $p(X \mid X_0, V_0, y)$1 to $p(X \mid X_0, V_0, y)$2 using a Heun integrator with 50 steps, producing $p(X \mid X_0, V_0, y)$3 (Chen et al., 25 Jun 2026).

The paper characterizes this as $p(X \mid X_0, V_0, y)$4-prediction with $p(X \mid X_0, V_0, y)$5-loss. Its stated rationale is that predicting $p(X \mid X_0, V_0, y)$6, which lies on a lower-dimensional data manifold, simplifies learning relative to direct velocity prediction. PhysiFormer therefore uses diffusion not as image synthesis in disguise, but as sequence-level generative modeling of vertex trajectories in world coordinates.

All positions and velocities are expressed in a bounded world coordinate box $p(X \mid X_0, V_0, y)$7, and time is discretized at $p(X \mid X_0, V_0, y)$8 for 49 frames. Noise is sampled as $p(X \mid X_0, V_0, y)$9 and scaled by noise_scale = 0.1; the paper reports that this narrower noise stabilizes training in raw coordinate space and stabilizes the coordinate-derived RoPE during early sampling.

3. Transformer architecture and factorised attention

The noisy trajectory input $N$0 is linearly embedded per vertex per timestep to $N$1 and flattened into $N$2 tokens. Initial positions and velocities are separately embedded and broadcast-summed onto each vertex token so that initial conditions remain distinct from the noised trajectory input. Material conditioning is implemented with a two-layer material MLP that embeds rigid versus elastic as inputs $N$3 or $N$4 and broadcasts the resulting embedding to the corresponding object’s vertices. In addition, 16 learned global register tokens are prepended and then replicated and consolidated across factorized attentions to aggregate context (Chen et al., 25 Jun 2026).

The central architectural device is factorized attention. A naïve DiT over $N$5 tokens would incur a cost of $N$6. PhysiFormer instead alternates three attention modes:

• full spatial attention, with cost $N$7;
• object-level spatial attention, grouping vertices by object and attending within each object at each time;
• temporal attention, with cost $N$8.

The resulting total complexity is reduced to $N$9. Object-level attention is especially significant: it implicitly encodes object identity, encourages per-object coherence, and remains permutation-invariant with respect to object order, without requiring explicit object-ID tokens.

Spatio-temporal positional encoding is also split by attention type. Temporal attention uses standard 1D RoPE over time indices. Spatial attentions use coordinate-conditioned RoPE following RenderFormer: per-vertex $\mathcal{F} \subset \{1,\ldots,N\}^3$0 coordinates are multiplied by log-spaced base-2 frequencies, converted to sinusoidal phases, and used to drive block-wise $\mathcal{F} \subset \{1,\ldots,N\}^3$1 rotations of query/key channel pairs. When the number of coordinate-derived phases differs from the per-head rotary dimension, phases are padded or truncated; zero padding yields identity rotations for unmatched channels.

The Transformer outputs predicted clean trajectories $\mathcal{F} \subset \{1,\ldots,N\}^3$2 through a linear head back to $\mathcal{F} \subset \{1,\ldots,N\}^3$3. At inference time, the provided face connectivity $\mathcal{F} \subset \{1,\ldots,N\}^3$4 is used to assemble the 4D mesh motion, and first-frame positions are clamped to $\mathcal{F} \subset \{1,\ldots,N\}^3$5. The backbone is a DiT-L with factorized attention, a “4×6 layers” stack, hidden dimension $\mathcal{F} \subset \{1,\ldots,N\}^3$6, and 16 register tokens.

4. Data generation, training regime, and reproducibility

Training data are generated in Genesis. Each scene is stored as a tuple $\mathcal{F} \subset \{1,\ldots,N\}^3$7 with $\mathcal{F} \subset \{1,\ldots,N\}^3$8, trajectory $\mathcal{F} \subset \{1,\ldots,N\}^3$9 for $X(t) \in \mathbb{R}^{N\times 3}$0, face connectivity $X(t) \in \mathbb{R}^{N\times 3}$1, and per-vertex material labels $X(t) \in \mathbb{R}^{N\times 3}$2, where 0 denotes rigid and 1 denotes elastic. The simulated environment is a bounded $X(t) \in \mathbb{R}^{N\times 3}$3 box with gravity on, friction minimized for near-elastic collisions, and standard damping retained for numerical stability; objects may collide with one another and with the box walls. Elastic objects use a fixed Young’s modulus that is constant across scenes (Chen et al., 25 Jun 2026).

Four datasets, totaling more than 100k scenes, are used:

Dataset	Scenes	Characteristics
D1	10k	Rigid floor-start; 1–5 convex objects from 15 templates; 4–20 vertices per object; $X(t) \in \mathbb{R}^{N\times 3}$4 vertices per scene
D2	15k	Rigid floor-start; 1–5 objects from 25 convex + 10 concave templates; 4–88 vertices per object; $X(t) \in \mathbb{R}^{N\times 3}$5 vertices per scene
D3	60k	Airborne-start rigid; 35k with 1–5 objects and 25k with 6–10 objects; in each group, 10k include nonzero initial angular velocity
D4	20k	Elastic; 1–5 objects; 10k floor-start and 10k airborne-start; templates as D2

Initial conditions are randomized in two modes. In floor-start scenes, objects are placed non-overlapping on the floor with random initial linear velocity, some zero; a $X(t) \in \mathbb{R}^{N\times 3}$6 orientation jitter allows unstable stationary objects to fall under gravity. In airborne-start scenes, objects are spawned in air with similar randomization. Splits are stratified train/validation/test with a fixed seed.

Optimization uses AdamW, EMA decay 0.9999, AMP bf16, and Flash SDPA for attention. Training runs on $X(t) \in \mathbb{R}^{N\times 3}$7 NVIDIA H100 94GB with effective batch size 64 and base learning rate $X(t) \in \mathbb{R}^{N\times 3}$8. The curriculum is staged: pretraining of “−L-10k” on D1 for 70k iterations with a 780-step warm-up and cosine decay to $X(t) \in \mathbb{R}^{N\times 3}$9; finetuning on rigid D1+D2+D3 for 27k iterations; and finetuning on elastic D4 for 12k iterations with a 60/40 rigid/elastic sampling ratio. Inference uses EMA weights, clamps $V(t)=dX/dt|_t \in \mathbb{R}^{N\times 3}$0, and integrates with 50 Heun steps.

The implementation details reported for reproduction include the DiT-L backbone, factorized attention that alternates full spatial, object-level spatial, and temporal modes, coordinate-conditioned RoPE for spatial attention, 1D RoPE for temporal attention, the embedding stack for trajectory tokens and conditioning variables, the logit-normal $V(t)=dX/dt|_t \in \mathbb{R}^{N\times 3}$1 schedule, and sampling with Heun integration.

5. Evaluation protocol and empirical results

Three evaluation families are reported. Trajectory accuracy is measured by per-vertex mean-square error

$V(t)=dX/dt|_t \in \mathbb{R}^{N\times 3}$2

Rigidity preservation is evaluated by fitting a best rigid transform to each object via the Kabsch algorithm and averaging the residual across time and connected components. Momentum-based physical consistency is assessed by a momentum drift ratio that compares the predicted system-momentum drift to the ground-truth drift; values closer to 1 are better. For computational feasibility, constant masses are assumed (Chen et al., 25 Jun 2026).

On the D1 benchmark with 250 test samples, PhysiFormer −L-10k is compared against autoregressive Transformer baselines and TIE. The reported 10-frame and 49-frame results are:

Model	10-frame horizon	49-frame horizon
PhysiFormer −L-10k	MSE 0.0953; Rigidity 0.0411; Momentum 4.45	MSE 9.55; Rigidity 0.185; Momentum 1.91
$V(t)=dX/dt|_t \in \mathbb{R}^{N\times 3}$3	MSE 1.78; Rigidity 0.928; Momentum 21.7	MSE 217; Rigidity 143; Momentum 11.9
$V(t)=dX/dt|_t \in \mathbb{R}^{N\times 3}$4	MSE 0.896; Rigidity 0.0960; Momentum 3.82	MSE 101; Rigidity 27.6; Momentum 8.37
$V(t)=dX/dt|_t \in \mathbb{R}^{N\times 3}$5	MSE 1.13; Rigidity 0.0846; Momentum 3.69	MSE 117; Rigidity 18.5; Momentum 7.95
TIE$V(t)=dX/dt|_t \in \mathbb{R}^{N\times 3}$6	MSE 0.157; Rigidity 0.328; Momentum 2.64	MSE 17.1; Rigidity 31.0; Momentum 2.91
TIE$V(t)=dX/dt|_t \in \mathbb{R}^{N\times 3}$7	MSE 1.73; Rigidity 0.137; Momentum 2.06	MSE 14.8; Rigidity 20.6; Momentum 2.29

The qualitative characterization in the paper is that autoregressive rollouts accumulate errors: stationary objects drift, meshes deform, and objects can escape the bounding volume. Even strong autoregressive baselines, including TIE$V(t)=dX/dt|_t \in \mathbb{R}^{N\times 3}$8, are reported to exhibit severe shape deformation over long horizons, whereas PhysiFormer maintains object rigidity and plausible long-horizon motion.

Ablations focus on properties specific to coordinate-space diffusion. Noise scaling at noise_scale = 0.1 yields the best quality for coordinate-space training; smaller scales hurt generalization, while larger scales cause jitter and harder denoising. Replacing factorized per-object attention with full spatial plus temporal attention and learned object-ID embeddings yields similar performance on 49-frame rigid trajectories—MSE $V(t)=dX/dt|_t \in \mathbb{R}^{N\times 3}$9 versus $T$0, momentum drift 1.53 versus 1.70, rigidity loss $T$1 versus $T$2—but object-ID embeddings do not extrapolate to more objects because unseen IDs have no embeddings.

6. Generalisation, uncertainty, efficiency, and failure modes

PhysiFormer trained on single-material scenes is reported to generalise to mixed-material scenes that jointly simulate rigid and elastic objects, to unseen real-world meshes with far more complex geometries than the training templates, and to larger object counts, including inference on 15 rigid objects despite training on at most 10. The paper attributes this object-count extrapolation to object-level attention, which scales while remaining permutation-invariant (Chen et al., 25 Jun 2026).

The diffusion formulation is used not only for accuracy but also for uncertainty modeling. The paper states that uncertainty in implicit properties such as mass, friction, and contact angles leads to diverse plausible futures. For PhysiFormer −L-10k, statistics across five generations are reported. For MSE, the mean is 0.0883 at 10 frames and 9.55 at 49 frames, with standard deviations 0.293 and 13.5. For rigidity loss, the mean is 0.0407 at 10 frames and 0.185 at 49 frames, with standard deviations 0.0210 and 1.22. For momentum drift ratio, the mean is 4.42 at 10 frames and 1.94 at 49 frames, with standard deviations 0.049 and 0.027. The reported high variance in MSE, coupled with physically plausible rigid motion differences relative to ground truth, indicates that the model’s stochasticity is not treated as numerical noise alone.

The efficiency argument is twofold. First, attention factorization reduces computational cost from $T$3 to $T$4 and lowers memory usage. Second, sampling cost is fixed by the number of ODE steps rather than by trajectory length in an autoregressive loop. A simulator comparison is also provided: on an 80-thread Xeon Gold 6338 CPU node, Genesis rigid-body simulation averages 1–6.5 s per sample for 1–10 objects, and elastic-body simulation averages 20–36 s for 1–5 objects, excluding rendering. On a single H100 GPU, PhysiFormer generates elastic scenes in roughly one fifth of the time at 25 steps.

The reported limitations are specific. Trajectory length and mesh resolution are fixed at $T$5 and the training resolution; longer horizons and spatial compression are identified as beneficial future directions. Because the model has no explicit collision or contact constraints, occasional spurious contacts, interpenetration, and rare orientation discontinuities occur. Material extrapolation is limited by the conditioning scheme, which distinguishes only two material classes, rigid and elastic. The paper suggests that contact-focused training and physics-aware losses may mitigate these issues.

The application domains named are robotics world modeling, where geometry-aware and view-invariant dynamics can support planning and simulation without camera confounders; graphics and animation, where the model enables efficient rollout of deformable and contact-rich scenes; and physical design optimization, where learned surrogate dynamics can accelerate evaluations. The reported results position coordinate-space diffusion as a step toward view-invariant, geometry-aware world modelling for robotics, graphics, and physical design.

7. Nomenclature and disambiguation

“PhysiFormer” in this usage refers specifically to the mesh-world diffusion model for rigid and elastic 3D mechanics introduced in “PhysiFormer: Learning to Simulate Mechanics in World Space” (Chen et al., 25 Jun 2026). It should be distinguished from several similarly named systems in unrelated subfields.

“PhysFormer: A Physics-Embedded Generative Model for Physically Self-Consistent Spectral Synthesis” is a physics-embedded conditional generative model for high-resolution spectral synthesis and parameter inversion in radiative-transfer settings, not a mechanics world model (Wang et al., 2 Mar 2026). “PhysicsFormer: An Efficient and Fast Attention-Based Physics Informed Neural Network for Solving Incompressible Navier Stokes Equations” is a transformer-based PINN for Burgers’ equation and incompressible Navier–Stokes equations; the source notes that it is “also written as PhysiFormer in some references,” which creates an additional naming collision (Barman et al., 7 Jan 2026). “PhysFormer” and “PhysFormer++” are video-transformer architectures for remote photoplethysmography from facial video, centered on temporal-difference attention and physiological measurement rather than 3D object mechanics (Yu et al., 2021, Yu et al., 2023). “Spiking-PhysFormer” is an rPPG variant that introduces SNN components for energy-efficient inference, again in a camera-based physiological-measurement setting rather than mesh-based physics simulation (Liu et al., 2024).

This naming overlap suggests that “PhysiFormer” is not a stable cross-domain family name but a label reused across mechanics simulation, PDE modeling, spectral synthesis, and physiological sensing. In bibliographic or technical discussion, the arXiv identifier is therefore the most reliable disambiguator.