Physics-Grounded Video Forecasting

• Physics-grounded video forecasting is a field that predicts future video frames by leveraging physical states, geometric representations, and motion laws instead of solely relying on raw pixel data.
• It employs methods such as simulator-mediated state rollouts, object-centric latent predictions, and trajectory-conditioned generation to enhance realism and causal accuracy.
• This approach enables improved evaluation metrics, realistic simulations in robotics and driving, and scalable extensions to text-to-video synthesis and anomaly detection.

Physics-grounded video forecasting denotes a family of methods that predict future visual observations by anchoring prediction in physical state, geometry, motion law, or verifiable physical structure rather than only in unconstrained pixel statistics. In current literature, the term covers several neighboring formulations: simulator-mediated future-frame prediction from inferred object state and parameters (Jaques et al., 2019, Kandukuri et al., 2020), 3D- or occupancy-grounded world models that roll out future state and then render video (Xue et al., 2023, Yang et al., 14 Dec 2025, Wang et al., 11 Feb 2026), object-centric latent predictors with explicit position–velocity structure (Gupta et al., 2021), and trajectory- or prompt-conditioned generators that improve physical alignment without a full simulator (Feng et al., 9 Jul 2025, Saurabh et al., 27 Mar 2026). Adjacent work on video-based physical state estimation, anomaly detection, and grounded video reasoning shows that forecasting belongs to a broader program of learning visual models whose outputs are constrained by physical dynamics, causal progression, and what–when–where grounding (Kamagata et al., 11 Jun 2026, Li et al., 5 Mar 2025, Osmanli et al., 23 Apr 2026).

1. Conceptual scope and task formulations

A useful distinction separates methods that forecast future frames directly from methods that first infer a physically meaningful intermediate state and then forecast through that state. In the first case, the output is a future image or video sequence, often conditioned on past frames, timestamps, prompts, or controls. In the second, the output may be a latent physical state, an object trajectory, a 3D point cloud, or a 4D occupancy field, from which future video is rendered afterward (Khurana et al., 2024, Yang et al., 14 Dec 2025).

The boundary of the field is porous. Some work is explicitly future-frame prediction, as in object-centric latent forecasting with a kinematic prior $p_{t+1}=p_t+v_t$ (Gupta et al., 2021), canonical-view billiards prediction from 10 observed frames to 10 predicted frames (Wang et al., 2018), or timestamp-conditioned long-horizon future-frame generation from past observations (Khurana et al., 2024). Other work is forecasting-adjacent: coastal video models that estimate nearshore wave peak period $T_p$ from 60-frame sliding windows perform clip-level regression rather than future RGB synthesis, yet still learn from video dynamics to recover a physical quantity summarizing temporal behavior (Kamagata et al., 11 Jun 2026).

Recent generative work extends the scope further. Trajectory-guided image-to-video systems predict coarse future motion trajectories and then use them to control generation (Yang et al., 1 Oct 2025, Feng et al., 9 Jul 2025). Physics-aware text-to-video systems inject chunk-level physical descriptions, latent predictive tokens, or agentically constructed conditioning into frozen generators to reduce physically implausible motion (Satish et al., 7 Jan 2026, Saurabh et al., 27 Mar 2026, Feng et al., 18 May 2026). This suggests that “forecasting” in the literature now ranges from explicit state rollout to controlled future synthesis, with the common denominator being an attempt to tie video evolution to some structured account of physical dynamics.

2. Physical representations and latent state

The choice of representation is the main axis along which physics grounding varies. A recurrent theme is that future prediction improves when the model operates in a state space closer to the scene’s underlying mechanics than raw pixels permit. One early strategy is canonicalization: real monocular billiards video is translated into a synthetic domain, warped into an allocentric canonical view, forecast in that normalized space by a Recurrent Latent Variation Network, and then mapped back to the original camera view and real image domain (Wang et al., 2018). The physical claim is not explicit conservation-law enforcement, but the removal of nuisance appearance and viewpoint complexity before dynamics prediction.

Object-centric latent models push the same idea toward structured state. In ICSWM, the latent space is organized around objects, positions, and velocities, with Gaussian-shaped mask priors and the explicit kinematic relation $p_{t+1}=p_t+v_t$ (Gupta et al., 2021). Physics-as-Inverse-Graphics goes further by inferring per-object coordinates, velocities, and global physical parameters from video alone, then simulating with known equations of motion and rendering back into pixels through a coordinate-consistent decoder (Jaques et al., 2019). In related work on differentiable physics for action-conditioned prediction, the latent state of object $i$ is $$\mathbf{s}_i=(\mathbf{x}_i^\top,\boldsymbol{\xi}^\top)^\top$$, where pose, velocity, mass, and friction become explicit prediction variables rather than opaque latent codes (Kandukuri et al., 2020).

Three-dimensional representations make the physical state more explicit still. 3D-IntPhys learns a conditional NeRF-style visual frontend and converts the inferred 3D field into explicit point clouds, then predicts future evolution with a graph neural dynamics model over those points (Xue et al., 2023). ContactGaussian-WM uses a unified Gaussian representation for both visual appearance and collision geometry, with world state $s_t=\{q_t,v_t\}$ and learnable physical parameters $\theta=(M,\mu,K,D)$, so future video is rendered from explicit rigid-body pose, velocity, friction, stiffness, and damping (Wang et al., 11 Feb 2026). GenieDrive adopts 4D semantic occupancy as its intermediate world state, forecasting 6 future occupancy frames from 4 past frames and compressing occupancy into a latent tri-plane representation that uses only 58% of the latent size of previous methods (Yang et al., 14 Dec 2025).

An alternative to explicit state variables is to forecast invariant modalities. Timestamp-conditioned diffusion models improve long-horizon future prediction by forecasting grayscale or pseudo-depth rather than RGB, with pseudo-depth obtained from ZoeDepth and treated as a geometry-grounded target (Khurana et al., 2024). This remains weaker than explicit mechanics, but it moves prediction toward scene layout and away from texture.

3. Forecasting mechanisms and control paradigms

Physics-grounded forecasting methods differ as much in their transition models as in their state representations. One major family uses differentiable simulation. Physics-as-Inverse-Graphics combines inverse graphics with a differentiable physics engine, uses Euler integration with known force laws, and trains from video reconstruction and prediction without supervision of object masks, positions, velocities, or physical parameters (Jaques et al., 2019). “Learning to Identify Physical Parameters from Video Using Differentiable Physics” embeds a constrained rigid-body simulator with an LCP-based contact and friction model inside an action-conditioned video prediction network, allowing future states and future frames to be forecast from inferred pose, velocity, mass, and friction (Kandukuri et al., 2020). ContactGaussian-WM replaces LCP-style contact with a complementarity-free differentiable rigid-body contact model and rolls out future state through explicit collision geometry and frictional dynamics before Gaussian rendering (Wang et al., 11 Feb 2026).

A second family learns neural dynamics in structured latent spaces without a full analytic simulator. ICSWM uses a GNN over object latents augmented with velocities and a contrastive latent prediction objective (Gupta et al., 2021). GMG, designed for meteorological and other non-rigid spatiotemporal fields, augments an ST-ConvLSTM backbone with a Global Focus Module for long-range dependencies and a Motion Guided Module for deformation, growth, and dissipation processes that local convolutions or sliding windows struggle to capture (Du et al., 14 Mar 2025). This is physics-inspired rather than physics-constrained: the model uses architectural inductive biases tailored to teleconnections and non-rigid motion, but no explicit governing equations.

A third family makes trajectories the central physical variable. “Physics-Grounded Motion Forecasting via Equation Discovery for Trajectory-Guided Image-to-Video Generation” tracks points with CoTracker, keeps the top-5 most dynamic trajectories, fits symbolic functions $x_t=f_i^x(t)$ and $y_t=f_i^y(t)$ by retrieval-initialized symbolic regression, extrapolates them forward, and uses the resulting trajectories to control an off-the-shelf image-to-video model (Feng et al., 9 Jul 2025). TrajVLM-Gen similarly predicts future trajectories first, but with a vision-LLM that outputs chain-of-thought reasoning plus 2D bounding-box sequences $[p_1,\dots,p_n]$, then injects those trajectories into an OpenSora-based video generator through text conditioning and trajectory-aware attention (Yang et al., 1 Oct 2025).

A fourth family improves physical plausibility by modifying conditioning inside a large generator. PhysVideoGenerator regresses V-JEPA 2 physical representations from noisy diffusion latents and injects the predicted physics tokens into the temporal attention layers of Latte through a dedicated cross-attention block, showing that diffusion latents contain enough information to recover V-JEPA 2 physical representations and that joint diffusion-plus-physics optimization remains stable over 50 epochs (Satish et al., 7 Jan 2026). PhysVid instead conditions a video generator on temporally contiguous chunks, each annotated with physics-grounded descriptions of states, interactions, and constraints, and adds negative physics prompts—descriptions of locally relevant law violations—at inference time to steer generation away from implausible trajectories (Saurabh et al., 27 Mar 2026).

The most explicit recent response to under-specified conditioning is agentic. NEWTON argues that text prompts are a lossy compression of the physical world and derives three requirements for useful physics conditioning—sufficiency, dynamism, and verifiability—then trains a planner to orchestrate keyframe generation, scientific computation, and prompt refinement in a multi-turn loop with verifier feedback (Feng et al., 18 May 2026). This suggests a general systems view in which future video generation is only one action inside a larger planning-and-verification pipeline.

4. Evaluation, grounding, and diagnostics

Evaluation in physics-grounded video forecasting is unusually heterogeneous because methods target different intermediate variables. State-space and world-model papers report errors on physical trajectories and on rendered video. ContactGaussian-WM evaluates cumulative translation error $T_p$0, rotation error $T_p$1, and image PSNR $T_p$2, thereby measuring state forecasting and video forecasting together (Wang et al., 11 Feb 2026). GenieDrive evaluates occupancy reconstruction and forecasting with mIoU and IoU, reports 42.59 mIoU and 51.80 IoU for average forecasting, and adds video metrics such as FVD, mIoU, and mAP for the occupancy-guided renderer (Yang et al., 14 Dec 2025). Equation-discovery-based forecasting uses normalized tree edit distance and MSE for recovered motion equations, then evaluates downstream video with FVD, FID, Subject consistency, Smooth, and trajectory error TraErr (Feng et al., 9 Jul 2025).

For multimodal or probabilistic predictors, evaluation often emphasizes structured or invariant targets rather than raw RGB fidelity. Timestamp-conditioned depth forecasting uses scale-and-shift invariant $T_p$3 error for pseudo-depth, PSNR for luminance and RGB, and Top-$T_p$4 metrics plus average trajectory error for long-horizon forecasting (Khurana et al., 2024). In coastal video state estimation, transformer-based architectures outperformed in terms of the accuracy of the instantaneous prediction, while lightweight recurrent-convolutional architectures achieved higher temporal stability and operational oceanographic skill, measured with Scatter Index and Willmott Skill Score (Kamagata et al., 11 Jun 2026). This is a recurring pattern: pointwise regression accuracy and physically meaningful temporal behavior need not rank models the same way.

Grounding diagnostics have become an evaluation topic in their own right. “Grounding Video Reasoning in Physical Signals” defines a what–when–where structure with a_what, a_when, and a_where, evaluates them with Acc, tIoU, and sIoU, and aggregates them through Logarithmic Geometric Mean; across models and prompt families, spatial grounding is the weakest across settings (Osmanli et al., 23 Apr 2026). This matters for forecasting because a model can name a future event correctly while still failing to localize when or where it occurs.

Physics anomaly and explanation benchmarks extend this line of thinking. Phys-AD introduces more than 6400 videos across 22 real-world object categories and 47 types of anomalies, explicitly requiring visual reasoning that combines physical knowledge and video content to determine object abnormality; it also introduces the Physics Anomaly Explanation metric PAEval for assessing whether visual-language foundation models can provide accurate explanations for underlying physical causes (Li et al., 5 Mar 2025). Although anomaly detection is not forecasting, the benchmark underscores the same principle: physically grounded video models should be evaluated not only on detection or generation quality, but on causal and localized physical consistency.

5. Domains and empirical regimes

The empirical landscape of physics-grounded video forecasting is broad. Controlled classical mechanics remains a common regime because it permits interpretable state spaces and known equations of motion. Physics-as-Inverse-Graphics studies spring systems, three-body gravity, and pendulum control (Jaques et al., 2019). Equation-discovery-based forecasting evaluates spring-mass, damped spring-mass, two-body, projectile motion, single pendulum, double pendulum, and fluid motion (Feng et al., 9 Jul 2025). Real-video intuitive physics prediction has also been demonstrated on billiards, where allocentric canonicalization and synthetic-to-real transfer make future motion prediction tractable from monocular video (Wang et al., 2018).

Robotics and manipulation motivate contact-rich world models. ContactGaussian-WM targets quasi-dynamic pushing and high-dynamic free-fall/rebound, learns from sparse and contact-rich video sequences, and reports robust generalization in both simulation and real-world evaluations, including data synthesis and real-time MPC (Wang et al., 11 Feb 2026). 3D-IntPhys addresses complex scenes involving fluids, granular materials, and rigid objects, including FluidPour, FluidCubeShake, and GranularPush, and shows that explicit 3D point-based state improves long-horizon future predictions and extrapolation (Xue et al., 2023).

Driving has emerged as a major application domain because it requires long-horizon, action-conditioned, multi-view consistency. GenieDrive forecasts 4D occupancy and then generates six-view driving videos, reporting a 7.2% improvement in forecasting mIoU at 41 FPS with 3.47M parameters, together with a 20.7% reduction in FVD for occupancy-guided video generation (Yang et al., 14 Dec 2025). The physical grounding here is geometry- and control-centered rather than Newtonian.

Environmental and scientific forecasting form another important regime. GMG targets radar precipitation, radar echo forecasting, and WeatherBench global 2 m temperature prediction, arguing that local kernels and sliding windows miss teleconnections and that weather fields require models adapted to non-rigid growth and dissipation (Du et al., 14 Mar 2025). Coastal wave monitoring is adjacent rather than direct future-frame prediction, but it demonstrates a practically important physics-guided video-to-state mapping for surf-zone dynamics and long-term passive monitoring (Kamagata et al., 11 Jun 2026).

Generative benchmarks emphasize broader physical commonsense. PhysVid reports improvements on VideoPhy and VideoPhy2 through temporally local physics-aware conditioning (Saurabh et al., 27 Mar 2026), while NEWTON improves joint accuracy on VideoPhy-2 from 21.4% to 29.7% on LTX-Video and from 30.7% to 37.4% on Veo-3.1 without modifying either generator (Feng et al., 18 May 2026). These results are not forecasting benchmarks in the narrow sense, but they demonstrate that physically grounded control remains a live problem even when the output is open-ended video rather than a deterministic future rollout.

6. Limitations and open directions

The field’s main limitations follow directly from its design choices. Methods with the strongest physical interpretability often require the strongest assumptions. Physics-as-Inverse-Graphics assumes that the family of governing equations is known (Jaques et al., 2019). Differentiable rigid-body approaches assume known actions, structured object models, or restricted camera and rendering conditions (Kandukuri et al., 2020). ContactGaussian-WM is currently restricted to rigid bodies, calibrated cameras, and deterministic forecasting (Wang et al., 11 Feb 2026). 3D-IntPhys requires multi-view RGB, camera poses, and instance masks, and is demonstrated in simulation rather than unconstrained real-world video (Xue et al., 2023). GenieDrive is specialized to the structured sensor geometry and occupancy supervision of driving (Yang et al., 14 Dec 2025).

At the opposite end, weakly constrained generation methods scale more easily but provide weaker guarantees. TrajVLM-Gen is “physics-aware” mainly through learned motion priors, heuristic physical labels, and trajectory consistency during generation, not through explicit physical law modeling (Yang et al., 1 Oct 2025). PhysVid improves physical commonsense through local descriptions of states, interactions, and constraints, but its grounding remains semantic rather than mechanistic (Saurabh et al., 27 Mar 2026). PhysVideoGenerator establishes the feasibility of regressing predictive world-model features from noisy diffusion latents, yet does not establish that generated videos are more physically plausible or that the learned tokens correspond to interpretable physical state (Satish et al., 7 Jan 2026).

Evaluation reveals another persistent limitation: grounding is weaker than semantics. In physically grounded video reasoning, prompt-family robustness is selective rather than universal, perturbation gains cluster in weak original cases, and spatial grounding is the weakest across settings (Osmanli et al., 23 Apr 2026). A plausible implication is that future-frame forecasting systems that appear semantically correct may still fail at physically localizing contact, onset, or future interaction regions.

Recent work increasingly treats this as a conditioning problem rather than only a model-capacity problem. NEWTON’s specification bottleneck diagnosis—together with its requirements of sufficiency, dynamism, and verifiability—suggests that physically grounded forecasting will likely require hybrid systems in which explicit world-state representations, adaptive external tools, and sample-level verification are combined rather than treated as mutually exclusive alternatives (Feng et al., 18 May 2026). A plausible synthesis is already visible across the literature: infer or construct a structured physical state, roll it forward with either analytic or learned dynamics, condition rendering on that state, and evaluate the resulting future not only for visual fidelity but for physically grounded what–when–where consistency (Jaques et al., 2019, Yang et al., 14 Dec 2025, Osmanli et al., 23 Apr 2026).