CausalMotion: Structured Physical Reasoning as Keyframe and Trajectory Guidance for Training-Free Video Generation

Abstract: Recent advances in diffusion-based video generation have significantly improved visual quality and short-term temporal coherence. However, existing methods still struggle to produce videos with physically consistent and causally plausible dynamics, especially in scenarios involving long-horizon interactions. This limitation arises from the fact that video diffusion models primarily learn physical consistency implicitly, while vision-LLMs can directly model physical laws. Based on this idea, in this work, we propose \textbf{CausalMotion}, a training-free framework that injects explicit physical reasoning into video generation through structured intermediate representations. Our key idea is to decouple reasoning from generation by leveraging a vision-LLM to decompose a text prompt into a sequence of causally consistent keyframes and object-centric motion trajectories. These representations are then aligned and integrated as soft constraints to guide a pretrained video diffusion model during inference. This design enables explicit modeling of object dynamics and causal transitions without requiring additional training or supervision. Extensive experiments show that our method consistently improves physical plausibility and temporal coherence, particularly in dynamics-intensive scenarios, while maintaining high perceptual video quality.

• The paper introduces a training-free framework that combines VLM-generated causal keyframes with physics-aware object trajectories to guide pretrained video diffusion models during inference.
• CausalMotion achieves a 0.65 average on PhyGenBench, surpassing VLIPP at 0.60 and LTX-Video at 0.39, while raising VBench quality from 80.57% to 82.52%.
• The method’s keyframe reasoning is essential, but its benefits decrease for appearance-dominant changes and complex long-horizon scenes, while inference takes approximately 448 seconds per sample.

CausalMotion addresses a persistent weakness of diffusion-based video generation: models such as LTX-Video, Wan, and CogVideoX learn physical dynamics implicitly from statistical correlations in training data and therefore produce visually plausible but causally invalid sequences—missing intermediate states, temporal discontinuities, and object interpenetration. The paper proposes a training-free framework that decouples reasoning from generation: a vision-LLM (VLM) decomposes the prompt into causally ordered keyframes and physics-aware object trajectories, which are then injected as soft constraints into a pretrained video diffusion model at inference time. The approach requires no fine-tuning, no physics simulator, and no additional supervision.

Motivation and positioning

The authors ground the problem in documented failure modes of current generators (e.g., Runway Gen-3 lighting a candle lowered into water) and cite evidence that video models do not reliably learn physical principles. They position CausalMotion against three families of prior work:

• Simulator-based methods (PhysGen, NewtonGen, PhysGen3D): physically grounded but constrained to simplified assumptions such as rigid-body dynamics, with substantial computational overhead.
• Training-based alignment (PISA, Force Prompting, NewtonRewards, VideoDPO, PhyGDPO): inject physical priors via losses or RL rewards but require large-scale training and supervision whose quality bounds performance.
• Foundation-model guidance (VLIPP, PhyT2V, ProPhy, VChain): closest to this work; these use VLMs to supply intermediate representations, but typically without explicit trajectory-level latent control.

CausalMotion's distinguishing claim is that combining structured keyframe reasoning with dense, physics-aware trajectory constraints at the latent level yields larger gains than either mechanism alone—an ablation result that supports this claim directly.

Method

The pipeline has three stages.

Keyframe generation via Visual Thought Reasoning. A VLM first infers the consequence—the most likely final state of the described scenario—as a global causal anchor. It then autoregressively produces a chain of visual thoughts: for each step $i$, conditioned on the previous image, caption, and the consequence, it emits a textual instruction $\mathit{txt}_i$ rendered into image $\mathit{img}_i$ by gpt-image-1.5. This yields causally consistent intermediate states covering all transitions implied by the outcome.

Physical state and trajectory planning. Key objects are localized in the initial keyframe with GroundingDINO/SAM-2-style segmentation. Each object is assigned a Physical State Vector comprising position, extent, velocity, and contact relations. Qwen2.5-VL-72B acts as a high-level planner: it identifies applicable physical laws (gravity, momentum conservation, fluid behavior), infers interactions, and predicts sparse states at roughly 13 key timesteps. These are densified to 121 frames via regime-adaptive interpolation—linear for uniform motion, kinematic with acceleration for accelerated motion—avoiding the artifacts of naive interpolation.

Time alignment and latent-space guidance. Keyframes are mapped to trajectory timesteps by a greedy, monotonic assignment maximizing IoU between each keyframe's object boxes and the interpolated trajectory. During denoising with LTX-Video, per-frame trajectories are projected into the latent temporal space; appearance anchors cropped from neighboring keyframes are blended with propagated local appearance into a dynamic reference $P^{\mathrm{ref}}_{i,\tau}$, and a Gaussian soft mask centered on the trajectory applies localized latent updates $$L_\tau \leftarrow (1-\alpha_s M)\odot L_\tau + \alpha_s M \odot P^{\mathrm{ref}}_{i,\tau}$$. Because updates are spatially localized, the global generative distribution is preserved while motion and appearance follow the planned dynamics. Keyframes additionally condition LTX-Video at their aligned timesteps with fixed strength ($\lambda = 0.9$).

Results

On PhyGenBench, evaluated under the PhyGenEval protocol across mechanics, optics, thermal, and material categories, CausalMotion achieves an average score of 0.65 versus 0.60 for VLIPP, 0.55 for PhyGDPO, and 0.39 for the LTX-Video baseline—a 67% relative improvement over its own backbone. Gains are largest in dynamics-intensive categories: mechanics (+0.06 over VLIPP) and thermal (+0.08). Notably, these gains come entirely from inference-time guidance rather than training.

Model	Mechanics	Optics	Thermal	Material	Avg
LTX-Video-T2V	0.35	0.45	0.36	0.38	0.39
VideoDPO	0.48	0.60	0.47	0.58	0.54
PhyGDPO	0.55	0.60	0.58	0.47	0.55
VLIPP	0.55	0.71	0.60	0.53	0.60
CausalMotion	0.61	0.71	0.68	0.61	0.65

On VBench (20 prompts shared with VChain), the method scores 82.52%, exceeding LTX-Video (80.57%) and Wan2.1-T2V-1.3B (76.21%)—an important negative-control result indicating that physical constraints do not degrade perceptual quality. Under Gemini 2.5 Flash VLM-as-judge evaluation, the overall score rises from 2.70 (LTX-Video) to 3.58, with large gains in semantic alignment (1.68 → 3.70) attributed to keyframe grounding and improved temporal consistency attributed to trajectory stabilization. Qualitative examples (milk poured into coffee, melting ice cream) show continuous state transitions where baselines exhibit liquid accumulating above cups or frozen upper scoops.

Ablations

Removing any component degrades performance, but unevenly. Dropping keyframe reasoning is catastrophic (average falls to 0.471), confirming it as the primary source of causal grounding. Removing trajectory physical-state planning costs little on average (0.642) and improves thermal slightly (0.689 vs. 0.678); removing latent trajectory guidance similarly yields 0.645 and improves material (0.625 vs. 0.608). The authors explain this candidly: material scenarios often involve stationary objects undergoing gradual appearance change (e.g., oxidizing metal), where trajectory constraints over-constrain the generator without benefit. This is a genuine boundary of applicability—the framework targets object-centric spatial dynamics, not appearance-only evolution.

An appendix analysis of trajectory–keyframe alignment shows frame-wise IoU degrading over time and high variance of mean IoU across prompts, indicating that alignment quality depends on scene complexity and interaction density, with failures concentrated in long-horizon or complex multi-object scenarios.

Limitations

The paper concedes two principal limitations. First, trajectory planning quality is bounded by the VLM's reasoning capability, and errors can accumulate in highly interactive scenes. Second, the alignment and guidance mechanisms presuppose explicit object motion; benefits diminish when dynamics are primarily photometric. A practical cost consideration is runtime: approximately 448 seconds per sample, dominated by API-based keyframe generation (271 s), with trajectory planning (69 s) and guided LTX-Video inference (108 s) contributing the remainder. The authors note this could be reduced with locally deployed models, though they do not evaluate such a configuration.

Conclusion

CausalMotion demonstrates that structured physical reasoning—causally ordered keyframes plus physics-aware trajectories—can be injected into a pretrained video diffusion model purely at inference time, yielding state-of-the-art PhyGenBench performance (0.65 average) while improving rather than sacrificing VBench perceptual quality. The main open questions left by the work are whether the framework extends beyond object-centric motion to appearance-dominant transformations, how robustness degrades under VLM reasoning errors in multi-object long-horizon scenes, and whether the ~7.5-minute-per-sample overhead can be reduced enough for practical deployment.