Physics-Aware Video Generation

• Physics-aware video generation is a computational technique that integrates real-world physical laws into video synthesis to produce sequences with accurate dynamics.
• Architectural innovations combine simulation-based methods, diffusion models, and reinforcement learning to enforce physical realism in object interactions.
• The field uses specialized datasets, benchmarks, and evaluation metrics to overcome specification bottlenecks and enhance both semantic fidelity and physical plausibility.

Physics-aware video generation refers to the task of synthesizing video sequences that not only achieve high photorealistic fidelity but also exhibit behaviors, interactions, and dynamics that strictly adhere to real-world physical laws. The field addresses the chronic shortcomings of conventional generative video models, which, although visually impressive, routinely violate physics—manifesting as objects floating, passing through each other, displaying unphysical inertia, or emitting implausible trajectories. This article provides a comprehensive review of the principles, methodologies, modalities of supervision, architectural innovations, evaluation protocols, benchmark datasets, and outstanding challenges associated with physics-aware video generation.

1. Theoretical Motivation and Problem Formulation

The classical video generation pipeline relies on unconditional or conditional diffusion or autoregressive models, which are trained to match the empirical distribution of video frames or sequences. These models are known to excel at visual realism but lack mechanisms to enforce or understand physical principles such as Newtonian dynamics, conservation of momentum, collision law, or energy dissipation. The underlying issue is a “specification bottleneck”: the training data does not contain explicit quantitative physics, and text prompts are under-specified, prompting the model to hallucinate plausible but inconsistent dynamics (Feng et al., 18 May 2026).

To overcome these limitations, recent works define the physics-aware video generation objective as learning a policy $\pi_\theta(v|p)$ that maximizes both semantic fidelity to a user prompt $p$ and a physically grounded reward signal $R_{\text{phys}}$ associated with the output video $v$: $$\max_\theta\; \mathbb{E}_{p}\Big[\mathbb{E}_{v\sim\pi_\theta(\cdot|p)}\log\pi_\theta(v|p) + \gamma\,R_{\text{phys}}(v, p)\Big]$$ where $\gamma$ balances visual-textual alignment with physical plausibility (Wang et al., 6 Nov 2025).

Physical realism encompasses several subtasks—rigid-body interactions, articulated motion, collision resolution, non-penetration, frictional effects, fluid flow, soft-body deformation, and consistency with conservation laws—all of which are fundamentally distinct from pixel-level or perceptual losses.

2. Architectural and Algorithmic Principles

Physics-aware video generation methods can be broadly classified by their architectural strategy and degree of physics integration:

• Model-Based Simulation and Rendering: Some pipelines explicitly decouple perception, simulation, and rendering. PhysGen (Liu et al., 2024) factors the process as: (1) extracting geometry and physics parameters from an image via instance segmentation and foundation models; (2) executing a 2D rigid-body physics simulation using Newton-Euler equations ($F$, $\tau$, $M$, $I$ parameters quantitatively inferred from the image); (3) rendering via affine warping, relighting, and video diffusion-based refinement. All simulation proceeds in pixel–centimeter units with physical realism injected at the simulation and compositing stages.
• Physics-Infused Diffusion Guidance: Several works, e.g., VLIPP (Yang et al., 30 Mar 2025) and TrajVLM-Gen (Yang et al., 1 Oct 2025), use a vision–LLM (VLM), often prompted with chain-of-thought (CoT) prompts, as a “motion planner” that predicts physically plausible object trajectories. These momentum-aware or physically conditioned trajectories are then injected—either as optical flow, bounding boxes, or trajectory tokens—into the video generative model via cross-attention or structured noise (Yang et al., 30 Mar 2025, Yang et al., 1 Oct 2025).
• Physics Feature Distillation and Latent Guidance: Other approaches (e.g. Phantom (Shen et al., 9 Apr 2026), PhysVideoGenerator (Satish et al., 7 Jan 2026)) jointly model visual and latent “physics” representations. In Phantom, a dual-branch flow-matching mechanism is employed, with one branch learning visual recurrence and the other learning physics recurrence (trained using features from a self-supervised physics-aware encoder such as V-JEPA2). Cross-attention between the branches ensures mutual refinement and consistency.
• Reinforcement Learning and Preference Optimization: Some frameworks optimize models or representation modules via reinforcement learning with physics-structured rewards. Methods like PhysMaster (Ji et al., 15 Oct 2025), PhysRVG (Zhang et al., 16 Jan 2026), and PhyGDPO (Cai et al., 31 Dec 2025) introduce reinforcement objectives based on DPO-type (Direct Preference Optimization) or GRPO (Group Relative Policy Optimization), using either human or model-based feedback on physical plausibility to guide the generator and its physics module.
• Inference-Time Physics Guidance: Several methods achieve physics compliance without any training or fine-tuning. These techniques employ light-weight test-time interventions: (1) constructing counterfactual “physics-violating” prompts using LLMs; (2) synchronously denoising two parallel chains with Synchronized Decoupled Guidance, such that implausible content is suppressed at every denoising step (Hao et al., 29 Sep 2025), or (3) sampling and verifying motion plans via sketch-guided VLM analysis before passing them to the main video generator (Huang et al., 21 Nov 2025).

3. Physics Supervision, Reward Formulation, and Evaluation Metrics

In order to enforce, evaluate, or optimize for physical realism, models use a variety of physics-centric signals:

Reward Type / Metric	Mathematical Formulation	Reference(s)
Trajectory Offset (TO)	$p$0	(Zhang et al., 16 Jan 2026)
Collision-Weighted Penalty	Per-frame weights $p$1 for collisions (sharp $p$2 changes)	(Zhang et al., 16 Jan 2026)
Groupwise DPO Loss	Plackett–Luce loss with VLM-based physics scores $p$3	(Cai et al., 31 Dec 2025)
Intra-Object Stability	$p$4	(Wang et al., 6 Nov 2025)
Mechanics Verification	Custom modules for coarse-checking collisions, penetraion, and Newtonian adherence	(Wang et al., 6 Nov 2025, Huang et al., 21 Nov 2025)
Human and VLM Judgments	VideoPhy2, VBench, and user studies (SA/PC/hard-action scores, binary pass/fail)	(Zhang et al., 27 May 2025, Feng et al., 18 May 2026)

Metrics such as Fréchet Video Distance (FVD), Temporal LPIPS, and VBench scores are often retained for perceptual evaluation, but physics realism requires the introduction of human and vision-LLM raters, classifying outputs according to “physical commonsense,” “mechanics adherence,” and “phenomena detection” (Zhang et al., 27 May 2025, Cai et al., 31 Dec 2025, Huang et al., 21 Nov 2025).

Experiments unequivocally show that physics-aware models—across simulation-based, planning, and RL-optimized paradigms—achieve superior scores on all physical correctness dimensions and can even increase perceptual or semantic alignment scores beyond baseline video generators (Feng et al., 18 May 2026, Zhang et al., 16 Jan 2026, Yang et al., 30 Mar 2025).

4. Data, Benchmarks, and Modalities

Recent advances have been catalyzed by curated, physics-centric datasets and custom benchmarks:

• Specialized Test Suites: PhyGenBench (phenomena-centric; mechanics, optics, thermal, materials) (Zhang et al., 27 May 2025), VideoPhy/VideoPhy2 (action- and phenomenon-centric, broad domain coverage) (Zhang et al., 27 May 2025, Cai et al., 31 Dec 2025), Physics-IQ (real videos annotated for physical comprehension) (Yang et al., 30 Mar 2025), WorldModelBench and PhyWorldBench (planning and long-horizon physical tasks) (Huang et al., 21 Nov 2025), PhysRVGBench (rigid-body tasks with annotated collisions) (Zhang et al., 16 Jan 2026).
• Dataset Construction: Large-scale curation pipelines such as PhyAugPipe (Cai et al., 31 Dec 2025) employ multimodal VLMs with chain-of-thought prompting to mine physics-rich, multiparticipant, and causally explicit clips from open video collections, filtering for high “physics richness” and semantic diversity. Balanced sampling strategies ensure coverage across dozens of physically complex categories.
• Driving and Controlled-Scenario Datasets: GenieDrive (Yang et al., 14 Dec 2025) builds on NuScenes to provide ground-truth 4D occupancy maps, enabling supervision “in the world model latent” rather than just pixels.

This proliferation of standardized datasets enables rigorous quantitative comparisons and drives architectural innovation centered on real-world physical fidelity.

5. Conditioning and Integration of Physics Knowledge

Conditioning mechanisms for physics-aware video generation include:

• Explicit Simulation: Model-based pipelines infer mass, friction, and restitution from static images and run classical rigid-body simulations for subsequent rendering (Liu et al., 2024).
• Trajectory Guidance: Physics-aware trajectories (bounding boxes, centroids, optical flow) predicted by VLM planners or by distillation from video foundation models are injected as conditioning signals—either as structured noise, attention masks, or cross-modal embeddings—into diffusion or transformer modules (Yang et al., 1 Oct 2025, Yang et al., 30 Mar 2025, Zhang et al., 29 May 2025).
• Latent Physical Representation: Architectures like Phantom (Shen et al., 9 Apr 2026) learn joint visual–physics latent spaces, while PhysVideoGenerator (Satish et al., 7 Jan 2026) trains a lightweight predictor to regress physics tokens from noisy diffusion latents for direct cross-attention injection during generation.
• Prompt Design and RL: Prompt-based approaches (PhyPrompt (Wu et al., 3 Mar 2026), PhyGDPO (Cai et al., 31 Dec 2025)) utilize physics-focused Chain-of-Thought LLMs that enrich prompts with force, material, and inter-object interaction details using RL curricula that gradually shift from semantic scaffolding to physical refinement.
• Negative and Counterfactual Prompting: At inference, several algorithms steer away from implausible trajectories by generating and conditioning on carefully designed “physics-violating” prompts and using synchronously denoised dual chains (SDG) (Hao et al., 29 Sep 2025, Saurabh et al., 27 Mar 2026).

A consistent finding is that explicit, temporally and spatially localized physics signals—whether simulated, predicted, or VLM-derived—substantially improve both physics fidelity and generalization.

6. Practical Impact, Limitations, and Future Directions

Physics-aware video generation has significant implications for embodied AI, robotics simulation, driving world models, scientific visualization, AR/VR, and downstream causal understanding. State-of-the-art models (e.g., PhysVideo (Wang et al., 19 Mar 2026), RealWonder (Liu et al., 5 Mar 2026), GenieDrive (Yang et al., 14 Dec 2025), PhyGDPO (Cai et al., 31 Dec 2025)) demonstrate robust real-world deployment potential, including interactive control, long-horizon rollout, and sim-to-real transfer.

However, limitations persist:

• Specification Bottleneck: Many methods still rely on incomplete prompts or lack real physical parameterization (mass, coefficients, environmental variables) (Feng et al., 18 May 2026).
• Data and Generalization: Physics-rich annotations and supervision are costly; transfer to visually or physically out-of-distribution scenes remains imperfect (Zhang et al., 29 May 2025).
• Computational Complexity: Physics-guided methods may incur overheads in inference or planning, though sketch-based and LoRA/SR approaches have reduced these constraints (Huang et al., 21 Nov 2025, Cai et al., 31 Dec 2025).

Notable research directions include the integration of explicit differentiable simulators into generative pipelines, learning physical priors jointly with text–video modeling, scaling physics-annotated datasets, and advancing world model representation for causal and multi-modal reasoning.

Physics-aware video generation is thus a rapidly maturing field, marked by the convergence of advanced generative modeling, structured physical knowledge, reinforcement learning, vision–language reasoning, and principled evaluation. It is poised to enable robust, reliable, and physically grounded synthesis for a diverse array of world-modeling and control applications.