PhyCo: Physically-Controlled Video Generation

• PhyCo is a framework for generative video modeling that introduces continuous, interpretable physical controls to address appearance synthesis and physical consistency in motion.
• It leverages a physics-rich simulation dataset and a multi-branch ControlNet to condition a pretrained video diffusion model on pixel-aligned physical property maps.
• A vision-language guided reward optimization refines physical realism by reducing errors in friction, restitution, deformation, and applied force during generation.

PhyCo is a framework for generative video modeling that introduces continuous, interpretable, and physically grounded control into motion synthesis. It was proposed to address a specific deficiency of modern video diffusion models: strong appearance synthesis paired with weak physical consistency, including object drift, unrealistic rebound in collisions, and material responses that do not match underlying properties. The framework combines a large-scale physics-rich simulation dataset, physics-supervised fine-tuning of a pretrained diffusion model through a ControlNet conditioned on pixel-aligned physical property maps, and VLM-guided reward optimization that supplies differentiable feedback from targeted physics queries. In the reported formulation, PhyCo produces physically consistent and controllable outputs through variations in friction, restitution, deformation, and force, without any simulator or geometry reconstruction at inference (Narayanan et al., 30 Apr 2026).

1. Conceptual scope and system decomposition

PhyCo is organized around three components: dataset construction, physics-supervised generative modeling, and reward-based refinement. The proposal is explicitly framed as a method for learning controllable physical priors for generative motion rather than as an explicit simulator replacement. Its design assumption is that physically meaningful control variables can be injected into a pretrained video diffusion backbone and then sharpened through supervision from a vision-LLM trained to answer physics-oriented questions about generated clips (Narayanan et al., 30 Apr 2026).

Component	Implementation	Function
Dataset	Kubric / PyBullet + Blender	Provides paired motion and physical attributes
Generative model	Cosmos-Predict2-2B + multi-branch ControlNet	Conditions video synthesis on physical property maps
Reward model	Qwen2.5-VL-3B with LoRA fine-tuning	Evaluates generated videos with targeted physics queries

The framework is notable for using continuous control variables rather than only categorical labels. Friction, restitution, deformation parameters, and external force are systematically varied and can be toggled independently or jointly. This suggests that PhyCo is intended to learn a factorized control space in which appearance variation and dynamical variation are separated as much as possible, although the paper formulates this operationally through data generation and conditioning rather than through an explicit disentanglement loss (Narayanan et al., 30 Apr 2026).

2. Physics-rich dataset and parameterization of motion

The PhyCo dataset is built on Kubric / PyBullet + Blender and contains approximately 100,000 photorealistic videos, each four seconds long, rendered at 24 FPS and 432×768 px. The summary identifies eight core dynamic scenarios that isolate fundamental interactions, including block sliding on a plane, ball rebounding off a wall, ball vertical bounce, soft ball drop under gravity, deformable object impacting a soft bed, and multiple balls colliding on a pool table, together with two additional composite setups. Dataset coverage spans purely rigid-body collisions, soft-body deformations, and multi-object contacts (Narayanan et al., 30 Apr 2026).

The physical parameterization is continuous. Friction is represented as the PyBullet Coulomb friction coefficient $\mu \in [0,1]$, sampled uniformly. Restitution is represented as $e \in [0,1]$, also sampled uniformly. Deformation is represented with Neo-Hookean parameters $(\lambda,\mu)$ and damping $\gamma$, varied from near-rigid to highly deformable regimes. External force is represented by normalized magnitude $\|F\| \in [0,1]$ and direction $\phi \in [0,2\pi)$, applied at $t=0$. The paper emphasizes that all four properties can be toggled independently or jointly, yielding continuous, disentangled control (Narayanan et al., 30 Apr 2026).

Appearance factors are randomized separately from dynamics. Each clip varies object color, texture from PolyHaven, HDRI illumination across 50 environments, and camera pose and viewpoint. The dataset cardinality is described as 8 scenarios $\times$ 50 HDRIs $\times$ 20 color-texture combinations $\times$ 25 parameter settings $e \in [0,1]$0K videos, with a split of 90K training and 10K held-out test. This construction is intended to disentangle appearance from dynamics at the data level rather than by architectural constraints alone (Narayanan et al., 30 Apr 2026).

3. Physics-supervised fine-tuning and conditioning interface

The generative backbone is Cosmos-Predict2-2B, described as a DiT variant. Standard inputs are text and the first frame $e \in [0,1]$1. PhyCo augments this backbone with a multi-branch ControlNet that injects pixel-aligned physical property maps $e \in [0,1]$2 into the denoiser at every U-Net block. The property tensor is decomposed into $e \in [0,1]$3 semantic groups, each in $e \in [0,1]$4: $e \in [0,1]$5, $e \in [0,1]$6 for deformation, and $e \in [0,1]$7 for force (Narayanan et al., 30 Apr 2026).

Each group $e \in [0,1]$8 is tokenized by the pretrained Cosmos tokenizer $e \in [0,1]$9, projected by a small adapter $(\lambda,\mu)$0, and fused into the U-Net through learned cross-attention kernels. Only the ControlNet parameters $(\lambda,\mu)$1 are updated; the DiT backbone $(\lambda,\mu)$2 and tokenizer remain frozen. This is an important design choice because it localizes physical adaptation to the control pathway while preserving the pretrained appearance and video prior of the backbone (Narayanan et al., 30 Apr 2026).

The diffusion training objective is standard score matching over noisy latents. For $(\lambda,\mu)$3, with $(\lambda,\mu)$4, the loss is

$(\lambda,\mu)$5

At this stage, there is no separate hand-crafted physics loss beyond the diffusion objective. Physics supervision is therefore implicit in conditioning the denoiser on $(\lambda,\mu)$6 and requiring the model to learn the mapping from physical properties to visible motion. A plausible implication is that physical consistency is first learned as a conditional generative regularity and only later sharpened by explicit reward-based feedback (Narayanan et al., 30 Apr 2026).

4. VLM-guided reward optimization

The second stage uses a vision-LLM to evaluate generated videos with physics-specific prompts. The base VLM is Qwen2.5-VL-3B. Synthetic clips from the PhyCo dataset, paired with ground-truth values for $(\lambda,\mu)$7, $(\lambda,\mu)$8, $(\lambda,\mu)$9, $\gamma$0, $\gamma$1, and $\gamma$2, are matched to a bank of binary Yes/No prompts and scalar JSON prompts. The VLM is fine-tuned with LoRA of rank 64 for 200 steps, yielding approximately 85% QA accuracy on held-out simulation clips (Narayanan et al., 30 Apr 2026).

At inference-time optimization, an $\gamma$3-step rollout, exemplified in the summary with $\gamma$4, produces $\gamma$5 from $\gamma$6. For each binary prompt $\gamma$7, the system extracts VLM logits $\gamma$8 for “Yes” and $\gamma$9 for “No,” and defines the alignment loss

$\|F\| \in [0,1]$0

with $\|F\| \in [0,1]$1 the sigmoid. Gradients $\|F\| \in [0,1]$2 are propagated through the VLM, tokenizer, decoder, and ControlNet, refining $\|F\| \in [0,1]$3 so that generated videos better answer the targeted physics questions (Narayanan et al., 30 Apr 2026).

In practice, the method uses a two-stage pipeline: first $\|F\| \in [0,1]$4, then $\|F\| \in [0,1]$5. The summary also states a joint objective view with typical weighting $\|F\| \in [0,1]$6 and $\|F\| \in [0,1]$7, tuned on a validation set. This makes the reward model a post-diffusion corrective mechanism rather than a replacement for likelihood-style diffusion training (Narayanan et al., 30 Apr 2026).

5. Empirical performance and controllability

Evaluation is reported on the Physics-IQ benchmark, which covers five domains: Solid Mech, Fluid Dyn, Optics, Magnetism, and Thermodynamics. The Physics-IQ score measures alignment of key events, such as fall time, collision, and rebound, to real-world reference. Under 120-frame evaluation, the base Cosmos-Predict2-2B obtains 27.7, adding physics ControlNet raises performance to 35.3, and adding VLM reward produces 36.3. Under matched 57-frame rollout with frame-repeat, the progression is base 30.9, text-only finetune 36.5, ControlNet 38.9, and ControlNet plus VLM 43.6 (Narayanan et al., 30 Apr 2026).

Evaluation setting	Method	Physics-IQ score
120-frame eval	Base Cosmos-Predict2-2B	27.7
120-frame eval	+ControlNet (physics)	35.3
120-frame eval	+ControlNet + VLM reward	36.3
57-frame rollout + frame-repeat	Base	30.9
57-frame rollout + frame-repeat	Text-only finetune	36.5
57-frame rollout + frame-repeat	+ControlNet	38.9
57-frame rollout + frame-repeat	+ControlNet + VLM	43.6

Fréchet Video Motion Distance is also reported to confirm improved temporal coherence. Human evaluation uses a 2AFC protocol with 16 participants and 98 stimulus pairs per method, each pair isolating one attribute among friction, restitution, deformation, and force. In this study, ControlNet+VLM is preferred more than 65% of the time on average for physical realism (Narayanan et al., 30 Apr 2026).

Controllability is assessed on 100 held-out simulation clips using a physics-trained Qwen2.5-VL regressor that estimates $\|F\| \in [0,1]$8, $\|F\| \in [0,1]$9, $\phi \in [0,2\pi)$0, and $\phi \in [0,2\pi)$1 from generated videos. Reported metrics include force magnitude error, friction error, angular deviation, restitution error, and deformation error, all with lower values better. Adding VLM reward reduces mean errors by 20–50% relative to ControlNet alone. Qualitative examples further show smooth variation in sliding speed as $\phi \in [0,2\pi)$2 varies, bounce height scaling with $\phi \in [0,2\pi)$3, squash-and-stretch amplitude responding to $\phi \in [0,2\pi)$4, motion direction steering via $\phi \in [0,2\pi)$5, and compositional control over combinations such as force plus friction or restitution plus deformation (Narayanan et al., 30 Apr 2026).

6. Generalization, limitations, and terminological ambiguity

PhyCo is reported to transfer beyond its synthetic training environment. Although trained purely on synthetic simulation clips, it is said to generalize to real-world Physics-IQ and to artistic or stylized scenes. The summary also reports cross-backbone fine-tuning: applying the PhyCo dataset to Wan2.2 yields a +4.6% Physics-IQ score, which is presented as evidence that the dataset has utility beyond the exact architecture used in the main system (Narayanan et al., 30 Apr 2026).

The reported limitations are explicit. Physics remain approximate because there is no hard enforcement of conservation laws such as energy and momentum. Complex multi-body contacts, fluids, articulations, and frictional spinning are not fully covered. Thin structures can exhibit occasional flicker, with the summary noting improvement from higher-FPS training or stronger temporal backbones. The VLM-based QA stage can fail on very subtle dynamics or under heavy occlusion. Future directions include extension to articulated, fluid-structure, and multi-object scenarios; incorporation of explicit physics constraints such as soft-body FEM losses alongside diffusion; better temporal models such as latent video transformers; and semi-supervised or real-world fine-tuning for tighter sim-to-real alignment (Narayanan et al., 30 Apr 2026).

The label “PhyCo” also appears in a distinct context in the cited literature on photosynthetic light-harvesting. A summary of work on the cryptophyte phycobiliprotein PC645 explicitly labels that complex “PhyCo,” in the title-adjacent description “cryptophyte phycobiliprotein PC645 (‘PhyCo’)” (Blau et al., 2017). This suggests a terminological ambiguity across domains: in generative modeling, PhyCo denotes a controllable physical-prior framework for video generation, whereas in quantum-biology-related discussion the same label may denote PC645.