Inference-time Physics Alignment of Video Generative Models with Latent World Models

Abstract: State-of-the-art video generative models produce promising visual content yet often violate basic physics principles, limiting their utility. While some attribute this deficiency to insufficient physics understanding from pre-training, we find that the shortfall in physics plausibility also stems from suboptimal inference strategies. We therefore introduce WMReward and treat improving physics plausibility of video generation as an inference-time alignment problem. In particular, we leverage the strong physics prior of a latent world model (here, VJEPA-2) as a reward to search and steer multiple candidate denoising trajectories, enabling scaling test-time compute for better generation performance. Empirically, our approach substantially improves physics plausibility across image-conditioned, multiframe-conditioned, and text-conditioned generation settings, with validation from human preference study. Notably, in the ICCV 2025 Perception Test PhysicsIQ Challenge, we achieve a final score of 62.64%, winning first place and outperforming the previous state of the art by 7.42%. Our work demonstrates the viability of using latent world models to improve physics plausibility of video generation, beyond this specific instantiation or parameterization.

• The paper demonstrates that using latent world models as physics-aware rewards substantially improves physical plausibility in video generation.
• The methodology decouples the video generator from the reward model and employs gradient-based guidance and Best-of-N sampling techniques.
• Empirical evaluations on benchmarks show notable gains in physics metrics and human preference, validating the effectiveness of inference-time alignment.

Inference-Time Physics Alignment for Video Generative Models Using Latent World Models

Motivation and Problem Formulation

Recent advances in large-scale video generative models have substantially improved perceptual video quality but have lagged in terms of physics plausibility. Generated videos commonly exhibit implausible object dynamics, incorrect temporal coherence, or violations of fundamental physical principles. While previous research predominantly attributed these deficiencies to pre-training limitations, this work instead reframes the problem as an inference-time alignment challenge, hypothesizing that physically plausible samples can be uncovered within the data manifold by leveraging stronger physics-aware reward signals.

The central proposition is to use latent world models, specifically VJEPA-2, as reward models to guide or select among denoising trajectories during video generation. This approach enables scaling of computational resources at inference-time—rather than retraining the generative model—and admits both differentiable (gradient-based) and non-differentiable (gradient-free) sampling schemes.

Methodology

The method decouples the generative model from the reward model. The generative backbone can be any text- or video-conditioned video diffusion model, such as MAGI-1 (autoregressive) or vLDM (holistic). The reward function is derived from a latent world model, instantiated via VJEPA-2’s “surprise” score, which functions as a proxy for physical plausibility. Specifically, for a generated video, VJEPA-2 predicts future latent encodings from context frames; the cosine similarity (surprise) between predicted and actual future encodings quantifies the adherence to learned physical dynamics.

Figure 1: VJEPA-2 computes surprise via a sliding window to evaluate physical plausibility and this signal is used to guide or select candidate generations.

Sampling is then performed according to a tilted distribution:

$p^*(x) \propto w(x) p(x)$

where $p(x)$ is the base video generator and $w(x) = \exp(\lambda r(x))$ or $w(x) = [F(r(x))]^{N-1}$ are weighting functions induced by the differentiable reward $r(x)$. Three principal strategies are adopted:

• Guidance ($\nabla$): Trajectory-level gradient-based steering using the reward’s score to bias the generative process at each denoising step.
• Best-of-N (BoN): Particle-based generation of $N$ samples with selection of the highest-reward sample.
• Guidance + BoN: Combining both for maximal effect; guidance focuses the search, while BoN further sharpens selection post-hoc.

  Figure 2: Qualitative improvements—generations guided by the latent world model more faithfully respect real-world physics than baseline samples.

Empirical Results

PhysicsIQ Benchmark

Performance is evaluated on the PhysicsIQ benchmark, which measures spatiotemporal IoU, spatial IoU, weighted IoU, and pixelwise MSE, providing a composite physics plausibility score. Notably, using VJEPA-2 surprise as the reward, combined with guidance and BoN, yields a new state-of-the-art PhysicsIQ score of 62.0%, surpassing the previous best by 6.78% for V2V and approximately 2% for I2V settings. This margin is not matched by reward signals based on VLMs (e.g., Qwen2.5-VL or Qwen3-VL) or VideoMAE. The scaling plot further demonstrates strong performance increases as the number of particles or guidance strength is raised.

Figure 3: Left: Substantial margin over previous SOTA on PhysicsIQ; Right: VJEPA reward outperforms VLM-based reward selection for inference-time alignment.

Figure 4: Increasing the number of particles and using guidance consistently pushes the PhysicsIQ score distribution toward higher plausibility.

Evaluations and Ablations

Human preference studies corroborate automatic results, showing an 11.4% improvement in win rate for physics plausibility and quality over baseline strategies, with preference for videos generated using VJEPA-2 reward alignment. Importantly, perceptual visual quality, motion smoothness, and subject/background consistency (measured by VBench suite) are preserved or even slightly improved, indicating that physics alignment does not trade off against visual realism.

Figure 5: Human study interface for pairwise video comparison on physics, quality, and prompt alignment.

Further analysis shows that physics improvements are robust across VJEPA architecture hyperparameters, and physics consistency improves monotonically as search/exploration budget increases. In text-to-video (T2V) settings using the VideoPhy benchmark, physics consistency improves by 6.9–8.1% with modest declines in semantic adherence, reflecting the text-agnostic nature of the current reward formulation.

Comparison to Prior Approaches

Unlike prior work that modifies training objectives to inject explicit physics signals, distills from motion priors, or applies VLM-based prompt rewriting, this framework provides a clean separation: a fixed pre-trained video diffusion model is inference-aligned using a world-model-based reward, requiring no generator re-training. Alternative reward signals—pixel-space autoencoder surprise (VideoMAE), VLM physics judgments, or classifier-free guidance—fail to match the transferability and effect of VJEPA-based alignment.

Limitations and Failure Modes

Analysis reveals three main limitations:

1. World Model Expressivity: Physics coverage is currently limited by the training data and design of the latent reward model. Failure cases include complex phenomena (e.g., rapid transitions, subtle material interactions, mirror reflections, siphon effects) that VJEPA-2 does not robustly encode.
2. Search Efficiency and Compute: Guidance and BoN scale well with compute allocation, but gradient-based steering incurs significant runtime/memory cost when used extensively.
3. Reward Semantics: The surprise score, while effective, is not fully disentangled from non-physical factors, and does not account for text-driven semantic alignment, especially important for open-domain T2V.

   Figure 6: Representative failure cases—abrupt physical events and advanced effects often remain unimproved even under VJEPA-2 guidance.

Implications and Future Work

The results demonstrate that substantial improvements in physics plausibility can be realized through purely inference-time techniques, assuming access to powerful physics-aware reward models. This has direct implications for downstream applications in robotics, autonomous driving, and simulation, where correctness of generated video dynamics is paramount. Scaling of inference-time search is viable, allowing dynamic compute-budget tradeoffs at deployment.

Further methodological advancements could include:

• Development of more expressive world models with broader coverage, capable of capturing complex material phenomena and rapid state transitions.
• Reward models incorporating conditioning semantics to jointly ensure adherence to both physics and prompt constraints.
• More efficient search or hybrid algorithms balancing gradient-based and candidate-based selection for improved compute-performance scaling.

Conclusion

This work establishes inference-time physics alignment using latent world models as an effective paradigm for enhancing physical plausibility in video generation. Leveraging VJEPA-2 as a reward model, the approach achieves state-of-the-art results on standard benchmarks and demonstrates significant practical viability—without altering or retraining the core generative model. The presented framework opens a pathway for improved world modeling, robust generation under physical laws, and more reliable downstream utility of video generative models.