PhysCtrl: Physics-Grounded Image-to-Video

• PhysCtrl is a physics-grounded framework that integrates physical parameters and external forces into a diffusion-based generative model for video synthesis.
• It models object dynamics as 3D point trajectories and leverages a diffusion process with physics-inspired regularizers to ensure motion plausibility.
• The framework bridges simulation data with a downstream image-to-video module, achieving improved metrics like 77% vIoU and low Chamfer-Distance for enhanced physical realism.

PhysCtrl is a framework for physics-grounded image-to-video generation that introduces explicit control through physical parameters and applied forces while retaining a generative formulation for motion synthesis. It models object dynamics as 3D point trajectories and learns their distribution with a diffusion model conditioned on material identity, force inputs, and scene constraints. The framework is trained on a large-scale synthetic corpus generated by physics simulators and couples its learned dynamics with a downstream image-to-video model, so that the synthesized video follows trajectories intended to be both controllable and physically plausible (Wang et al., 24 Sep 2025).

1. Problem setting and physical representation

PhysCtrl addresses a specific deficit in contemporary video generation: strong photorealism does not by itself ensure physical plausibility or 3D controllability. Its formulation therefore begins with an explicit representation of object motion in three dimensions. An object is represented by a set of $N$ points in $\mathbb R^3$ over frames $f=0,\ldots,F$, with initial state $P^0=\{x_p^0\in\mathbb R^3\}_{p=1}^N$, per-frame point sets $P^f=\{x_p^f\}_{p=1}^N$, and full trajectory $\mathcal P\in\mathbb R^{F\times N\times 3}$ (Wang et al., 24 Sep 2025).

The conditioning variables encode both material properties and external actuation. PhysCtrl uses

$c=\{P^0,f,D,h,[mat],E,\nu\},$

where $f\in\mathbb R^3$ is an external point force, $D\in\mathbb R^3$ is the drag point on $P^0$ where the force is applied, $\mathbb R^3$0 is the floor height, $\mathbb R^3$1 is a one-hot material indicator, and $\mathbb R^3$2 and $\mathbb R^3$3 are Young’s modulus and Poisson ratio for elastic and plasticine objects. The material palette comprises four classes: elastic, sand, plasticine, and rigid (Wang et al., 24 Sep 2025).

The physical basis is stated through continuum mechanics and the Material Point Method (MPM). The governing equations include conservation of momentum and mass,

$\mathbb R^3$4

with Cauchy stress $\mathbb R^3$5 derived from an energy density $\mathbb R^3$6 and deformation gradient $\mathbb R^3$7. The discrete MPM updates are written through particle-to-grid and grid-to-particle transfers, but PhysCtrl does not execute these simulators at inference time; instead, simulator outputs supervise the learned generative model during training (Wang et al., 24 Sep 2025).

This design makes the 3D trajectory, rather than pixels, the primary locus of physical control. A plausible implication is that PhysCtrl treats rendering as a downstream problem and reserves the core learning capacity for dynamics.

2. Diffusion-based generative physics model

At the center of PhysCtrl is a generative physics network implemented as a diffusion model over trajectories. The forward process follows a DDPM-style noising kernel,

$\mathbb R^3$8

with closed-form sampling

$\mathbb R^3$9

The reverse model is a U-like diffusion transformer $f=0,\ldots,F$0 that predicts the denoised trajectory conditioned on the physical control vector $f=0,\ldots,F$1 (Wang et al., 24 Sep 2025).

Training uses the signal-prediction variant of score matching: $f=0,\ldots,F$2 A cosine-based schedule is used for $f=0,\ldots,F$3, with $f=0,\ldots,F$4 and $f=0,\ldots,F$5 (Wang et al., 24 Sep 2025).

PhysCtrl supplements the diffusion objective with a velocity consistency term,

$f=0,\ldots,F$6

which constrains first-order temporal differences between predicted and reference trajectories (Wang et al., 24 Sep 2025).

The result is not a classical simulator and not a purely appearance-driven video prior. It is a learned distribution over physically conditioned 3D motion fields. This suggests that PhysCtrl occupies an intermediate regime between simulation-supervised motion synthesis and controllable video generation.

3. Architecture and physics-based regularization

PhysCtrl introduces a spatiotemporal attention block specifically designed to emulate particle interactions. Each transformer block alternates between spatial self-attention across points within a frame and temporal self-attention across frames for a fixed point (Wang et al., 24 Sep 2025).

For spatial attention, the token matrix for frame $f=0,\ldots,F$7 is

$f=0,\ldots,F$8

formed by concatenating point embeddings with a learned physics token $f=0,\ldots,F$9. Queries, keys, and values are computed by linear projection,

$P^0=\{x_p^0\in\mathbb R^3\}_{p=1}^N$0

followed by

$P^0=\{x_p^0\in\mathbb R^3\}_{p=1}^N$1

Adaptive layer normalization aligns geometry tokens and conditioning tokens before attention (Wang et al., 24 Sep 2025).

For temporal attention, each point $P^0=\{x_p^0\in\mathbb R^3\}_{p=1}^N$2 yields a trajectory token sequence

$P^0=\{x_p^0\in\mathbb R^3\}_{p=1}^N$3

with an analogous attention computation

$P^0=\{x_p^0\in\mathbb R^3\}_{p=1}^N$4

The ordering “spatial $P^0=\{x_p^0\in\mathbb R^3\}_{p=1}^N$5 temporal” is explicitly described as physically motivated: first gather neighbor forces, then integrate forward (Wang et al., 24 Sep 2025).

Training adds three physics-inspired regularizers. The deformation-gradient consistency term is

$P^0=\{x_p^0\in\mathbb R^3\}_{p=1}^N$6

where

$P^0=\{x_p^0\in\mathbb R^3\}_{p=1}^N$7

approximates the MPM gradient step. Grid velocities $P^0=\{x_p^0\in\mathbb R^3\}_{p=1}^N$8 are reconstructed by one P2G+G2P pass on predicted points. Floor interaction is enforced through

$P^0=\{x_p^0\in\mathbb R^3\}_{p=1}^N$9

The total training objective is

$P^f=\{x_p^f\}_{p=1}^N$0

These terms are intended to enforce physical plausibility beyond what a generic diffusion prior would learn from trajectory statistics alone (Wang et al., 24 Sep 2025).

A common misconception would be to equate such regularization with exact simulation. PhysCtrl instead uses simulator-derived inductive bias and loss shaping; the paper’s formulation remains generative and learned rather than analytically integrated at deployment.

4. Training data and image-to-video pipeline

The training corpus is synthetic and large scale. PhysCtrl uses 550 K simulated animations in total: 150 K elastic objects with varying drag forces, and 100 K each of gravity-only trajectories for elastic, sand, plasticine, and rigid materials. Each simulation contains $P^f=\{x_p^f\}_{p=1}^N$1 points sampled uniformly on the mesh surface and $P^f=\{x_p^f\}_{p=1}^N$2 frames. MPM, following Jiang et al. 2016, is used for elastic, plasticine, and sand, while a rigid-body solver is used for rigid objects. Elastic parameters are sampled from $P^f=\{x_p^f\}_{p=1}^N$3 and $P^f=\{x_p^f\}_{p=1}^N$4, with force magnitude $P^f=\{x_p^f\}_{p=1}^N$5, where $P^f=\{x_p^f\}_{p=1}^N$6 is total weight. Data augmentation consists of random $P^f=\{x_p^f\}_{p=1}^N$7-rotations and initial noise sampled from $P^f=\{x_p^f\}_{p=1}^N$8 (Wang et al., 24 Sep 2025).

Component	Specification
Dataset scale	550 K simulated animations
Materials	elastic, sand, plasticine, rigid
Per-simulation resolution	$P^f=\{x_p^f\}_{p=1}^N$9 points, $\mathcal P\in\mathbb R^{F\times N\times 3}$0 frames

The image-to-video system is a multistage pipeline. First, the object in the input image is segmented with SAM. Second, 20 novel-view images are synthesized via SV3D, and three are selected for orbit coverage. Third, multi-view Gaussian reconstruction with LGM produces a 3D point cloud $\mathcal P\in\mathbb R^{F\times N\times 3}$1, which is downsampled to $\mathcal P\in\mathbb R^{F\times N\times 3}$2. The learned diffusion model then samples the trajectory $\mathcal P\in\mathbb R^{F\times N\times 3}$3. Each 3D point $\mathcal P\in\mathbb R^{F\times N\times 3}$4 is projected into the input image with known camera intrinsics to obtain $\mathcal P\in\mathbb R^{F\times N\times 3}$5, from which a per-pixel trajectory field $\mathcal P\in\mathbb R^{F\times N\times 3}$6 is formed. Finally, a pretrained video diffusion model, “Diffusion as Shader” by Gu et al. 2025, is conditioned on this dense motion prior through cross-attention during denoising, producing an RGB video consistent with the sampled 3D physics trajectory (Wang et al., 24 Sep 2025).

This pipeline makes the physical controller upstream of appearance generation. The trajectory field acts as an explicit motion prior rather than an implicit prompt.

5. Evaluation, ablations, and subsequent extensions

PhysCtrl is evaluated at both the dynamics level and the video level. For generative dynamics, the reported metrics are volume-IoU (vIoU\uparrow), Chamfer-Distance (CD$\mathcal P\in\mathbb R^{F\times N\times 3}$7), and $\mathcal P\in\mathbb R^{F\times N\times 3}$8-correspondence error Corr$\mathcal P\in\mathbb R^{F\times N\times 3}$9, averaged over frames. Against Motion2VecSets and MDM, PhysCtrl reports vIoU $c=\{P^0,f,D,h,[mat],E,\nu\},$0, CD $c=\{P^0,f,D,h,[mat],E,\nu\},$1, and Corr $c=\{P^0,f,D,h,[mat],E,\nu\},$2. Ablations removing spatial attention, temporal attention, or the physics loss each yield drops in vIoU and CD, indicating that all three contribute to performance (Wang et al., 24 Sep 2025).

Evaluation setting	Reported result
Generative dynamics	vIoU 77.0%, CD 0.0030, Corr 0.0016
User study	81% for physics plausibility, 66% for video quality
GPT-4o assessment	PhysCtrl wins SA, PC, and VQ

For image-to-video generation, GPT-4o ratings are reported for Semantic-Adherence (SA), Physical-Commonsense (PC), and Video-Quality (VQ) on a 5-point Likert scale, with PhysCtrl winning all three. In a user study of $c=\{P^0,f,D,h,[mat],E,\nu\},$3 users $c=\{P^0,f,D,h,[mat],E,\nu\},$4 clips, PhysCtrl is chosen $c=\{P^0,f,D,h,[mat],E,\nu\},$5 of the time for physics plausibility and $c=\{P^0,f,D,h,[mat],E,\nu\},$6 for video quality, versus $c=\{P^0,f,D,h,[mat],E,\nu\},$7 and $c=\{P^0,f,D,h,[mat],E,\nu\},$8 for the nearest competitor (Wang et al., 24 Sep 2025).

A later framework, TelePhysics, is described as implementing “PhysCtrl”-style capabilities in a training-free, multi-object, scene-level formulation. TelePhysics reconstructs a unified 3D scene from a single image, uses the Genesis engine with rigid-body dynamics, MPM, and position-based dynamics, and decouples simulation from rendering to deliver approximately 15 FPS interactive previews. Its reported quantitative benchmarks include semantic adherence $c=\{P^0,f,D,h,[mat],E,\nu\},$9 versus PhysCtrl’s $f\in\mathbb R^3$0, physical commonsense $f\in\mathbb R^3$1 versus $f\in\mathbb R^3$2, video quality $f\in\mathbb R^3$3 versus $f\in\mathbb R^3$4, and human Borda preference approximately $f\in\mathbb R^3$5 versus $f\in\mathbb R^3$6 for the next best method (Zhang et al., 19 May 2026).

The relationship between the two systems is methodologically informative. PhysCtrl is a learned generative model for single-object physics-grounded motion; TelePhysics instead prioritizes training-free scene reconstruction and real-time interaction. This suggests a divergence between model-based and simulation-backed routes to controllable physical video generation.

6. Scope, limitations, and open directions

PhysCtrl is explicitly limited to the single-object setting. Multi-body interactions, fluids, and granular coupling are not covered. Its material palette is restricted to elastic, sand, plasticine, and rigid; boundary conditions beyond a flat floor, including obstacles and collisions, are only approximated (Wang et al., 24 Sep 2025).

These constraints bound the interpretation of the reported results. “Physics-grounded” in this framework means that the generated trajectories are conditioned on physical parameters, supervised by simulator outputs, and regularized by physics-inspired losses; it does not imply arbitrary scene interaction or full generality over constitutive laws. The paper identifies several future directions: multi-object and contact-rich scenes, learned simulators that capture energy conservation more explicitly, and real-world domain adaptation from video (Wang et al., 24 Sep 2025).

Within the recent literature, PhysCtrl is most usefully understood as a trajectory-centric bridge between 3D physical simulation data and controllable video synthesis. Its core contribution is not a new physics engine but a generative model whose latent control space is organized around material response, force application, and geometric evolution in $f\in\mathbb R^3$7.