KineMask: Physics-Guided Video Diffusion

• KineMask is a physics-guided video diffusion framework offering explicit per-pixel velocity control to synthesize realistic rigid-body interactions from static images.
• It integrates dense velocity mask inputs with high-level text conditioning using a two-stage ControlNet training strategy to propagate motion cues through diffusion steps.
• Empirical evaluations demonstrate superior performance in motion fidelity and physical plausibility, making it valuable for applications such as robotics simulation and embodied decision-making.

KineMask is a physics-guided video diffusion framework that enables explicit, object-level kinematic control in video generation, particularly for synthesizing physically plausible rigid-body interactions. Developed to overcome the limitations of traditional video diffusion models—namely their lack of physically grounded control—KineMask integrates dense per-pixel velocity conditioning and high-level textual prompts, thereby producing realistic sequences of object motion, collision, and resultant effects from a single image and specified initial object velocity. Built on the CogVideoX image-to-video backbone, KineMask combines a ControlNet branch for velocity mask input with a text-conditioning mechanism, facilitating the generation of complex dynamical phenomena and supporting advanced applications such as robotics simulation and embodied decision-making (Romero et al., 2 Oct 2025).

1. Methodological Foundations and Objectives

KineMask seeks to reconcile the gap between unconstrained video diffusion, where motion is emergent from model priors, and explicit physics simulation, where outcomes are dictated by dynamical solvers. Two primary objectives define the framework: (1) providing user-controlled initialization of object velocities in 3D (projected to the image domain) and supporting automatic simulation of object futures—including collisions and momentum transfer; and (2) maintaining high visual fidelity by leveraging large pretrained video diffusion models (VDMs) while enforcing physically plausible, rigid-body dynamics through per-pixel mask conditioning. Earlier approaches predominately offer high-level semantic control, such as text prompts or keyframes, whereas KineMask introduces dense, low-level kinematic control via RGB “velocity masks” $M_t \in \mathbb{R}^{H \times W \times 3}$, where each object pixel encodes its instantaneous $(x, y, z)$ velocity (Romero et al., 2 Oct 2025).

2. Architecture and Training Paradigm

KineMask utilizes CogVideoX-I2V-5B as its generative backbone. At each timestep $t$ in the denoising diffusion process, the model predicts noise conditioned on the noisy video tensor $x_t \in \mathbb{R}^{F \times H \times W \times C}$, a reference image $y$ (the first video frame), a text-derived scene embedding $c \in \mathbb{R}^d$, and a control embedding from a ControlNet $\psi_{\phi}(M)$ that consumes velocity masks. The training objective adopts the standard DDPM form: $$\mathcal{L}_{\mathrm{diff}}(\theta) = \mathbb{E}_{x_0,c,y,M,t,\epsilon}\left[\left\|\epsilon - \epsilon_\theta(x_t, c, y, \psi_\phi(M))\right\|^2\right].$$

A two-stage ControlNet training strategy underpins effective mask-based guidance:

• Stage 1: Supervised Motion—Utilizing a synthetic dataset of videos with complete per-frame velocity masks, the ControlNet is trained to map velocity masks to denoising guidance, receiving clean dense supervision at all frames.
• Stage 2: Self-Supervised Synthesis—Recognizing that only the initial-frame velocity mask will be available at inference, later-stage training randomly zeroes out masks after the first $f^\ast$ frames. The ControlNet is fine-tuned to propagate the initial velocity cue through the entire diffusion sequence, synthesizing plausible object interactions without future mask supervision.

The total loss combines standard diffusion and dense control objectives, with mask and velocity loss terms weighted equally ($$\lambda_{\mathrm{mask}}=1, \lambda_{\mathrm{vel}}=1$$).

3. Physics-Guided Components

3.1 Learned Mask Inference

At inference, only the initial velocity mask $(x, y, z)$0 is provided. The trained ControlNet predicts a learned embedding $(x, y, z)$1, which directs the denoising backbone to move controlled object pixels according to the inferred kinematic trajectory corresponding to realistic rigid-body motion.

3.2 Implicit Rigid-Body Dynamics and Collisions

The framework eschews explicit coupling to physics solvers. Instead, it learns rigid-body integration—including velocity updates and collision-induced impulses—by example from synthetic Blender-rendered data. Standard updates are governed by: $(x, y, z)$2 but when collisions occur, learned operators induce velocity jumps consistent with conservation of momentum. The ControlNet and diffusion backbone jointly implement an implicit collision operator: $(x, y, z)$3 yielding plausible post-collision object dynamics.

4. Conditioning Mechanisms: Low-Level and High-Level Guidance

KineMask combines dense low-level kinematic control from velocity masks with textual high-level scene descriptions. Scene captions are generated for synthetic videos by a vision-LLM (Tarsier) at training, encompassing all object motions and collisions. During inference, descriptive prompts are generated using GPT-5, conditioned on the initial frame and velocity cue. The embeddings from these captions are integrated into the backbone through cross-attention alongside the ControlNet embedding, which enables the synthesis of videos with specified secondary effects (e.g., shattering, rippling), extending the diversity and realism of generated dynamics beyond geometric constraints.

5. Experimental Protocol and Empirical Evaluation

5.1 Datasets

Three principal datasets underpin the evaluation protocol:

• Simple Motion: 10,000 training and 100 test videos of solitary objects (cubes, cylinders) moving without collision.
• Interactions: 10,000 training and 100 test videos with multiple objects and collision dynamics, rendered on randomized backgrounds (AmbientCG).
• Real World: 50 web-sourced or ChatGPT-generated real images, evaluated for generalization; ground-truth videos or masks are not available.

5.2 Metrics

Evaluation employs:

• Mean Squared Error (MSE) between generated and ground-truth frames ($(x, y, z)$4 and $(x, y, z)$5).
• Fréchet Video Distance (FVD) for overall video quality.
• Fréchet Video Motion Distance (FVMD) for motion realism.
• Intersection over Union (IoU) between predicted and true object masks for geometric accuracy.

5.3 Results

KineMask consistently outperforms state-of-the-art baselines (CogVideoX, Wan2.2-I2V, Force Prompting) across all key quantitative metrics on the Interactions test set. Notably, mask-based control enables superior results even when KineMask is trained exclusively on Simple Motion data. Human preference studies on real-world images show a substantial margin in favor of KineMask for motion fidelity, physical plausibility, and interaction realism. Qualitative results indicate precise execution of collisions, causal object pushes, multi-object cascades, and generation of plausible secondary effects (Romero et al., 2 Oct 2025).

Dataset	MSE	FVD	FVMD	IoU
Interactions	Lower	Lower	Lower	Higher
Simple Motion	Lower	Lower	Lower	Higher

6. Limitations and Future Directions

Currently, KineMask controls only per-pixel velocity. Accurate simulation of real-world dynamics requires further parameters, such as per-pixel mass, friction, restitution, and explicit modeling of non-rigid and fluid effects. Extensions to the ControlNet embedding to accommodate these factors, or hybridization with differentiable physics simulators, are identified as promising avenues. More advanced multimodal LLM integration could enable richer, goal-directed planning (e.g., specifying objectives like “slide the cup so that the water spills toward the blue book”). KineMask’s ability to predict object interactions given minimal input positions it as a viable world-model module for robot planning and scenario prediction, where rapid, differentiable, and physical rollouts are needed (Romero et al., 2 Oct 2025).