- The paper introduces PhysGen, a 732M-parameter multimodal world model that jointly predicts continuous video frames and robot actions with diffusion-based decoding and autoregressive rollouts.
- The model achieves 90.8% average success on LIBERO, outperforming OpenVLA and WorldVLA by 13.8 and 8.8 percentage points, while reaching 74% on ManiSkill and 75% on real-world Franka tasks.
- The ablations show that video pretraining, continuous tokenization, autoregressive prediction, and lookahead multi-token prediction each improve performance, supporting video models as data-efficient foundations for robotic control.
PhysGen is a framework that repurposes pretrained autoregressive video generation models as predictive world interaction models for robotic manipulation, built on the NOVA continuous video autoregressive backbone (2603.00110). The central premise is that video generators trained on large-scale visual data encode implicit physical regularities—object permanence, dynamics, temporal coherence—that transfer to control more readily than text-derived knowledge in LLM-based Vision-Language-Action (VLA) models. The paper reports that with no action-specific pretraining whatsoever, PhysGen surpasses OpenVLA and WorldVLA on LIBERO by absolute margins of 13.8% and 8.8%, and matches π₀ on real-world tasks.
Motivation and positioning
The authors argue that VLA architectures built on LLMs suffer from a modality gap: language is symbolic while manipulation demands precise spatial and temporal grounding. Autoregressive video generation models are a more natural substrate because their step-by-step state evolution mirrors sequential decision-making. PhysGen differs from prior joint video-action prediction efforts (WorldVLA, UWM, UniMimic) primarily through its representation: rather than discrete tokenization—which introduces resolution errors that compound into trajectory drift—both frames and actions occupy a shared continuous embedding space, with conditional densities estimated via diffusion. This directly extends MAR-style non-quantized autoregression (Deng et al., 2024) into a unified vision-action setting.
Architecture
PhysGen retains NOVA's frozen components: a Phi LLM tokenizes instructions and a 3D-VAE encodes observations into frame tokens ($360$ per frame). An MLP action tokenizer projects action chunks of length L=8 into the same d-dimensional space. Frame and action tokens are concatenated per timestep into "physical tokens" Pn=[EO,n;EA,n], with a learnable Begin-of-Action token resolving the one-step observation-action offset. A causal Transformer predicts the next physical token conditioned on all preceding tokens; de-tokenization runs a DiT-based denoiser conditioned on the transformer output, with a dedicated lightweight Action-DiT injecting conditioning via cross-attention.
Three design choices support joint modeling:
- Causal masking: chunk-wise full attention within frames, temporal-causal attention within action chunks across timesteps, and unidirectional action-to-frame attention so action planning conditions on future visual states—an implicit inverse-kinematics pathway.
- Lookahead Multi-Token Prediction (L-MTP): three future action tokens are predicted in parallel; only the first executes at inference, with the remainder serving as lookahead context. This combines the longer planning horizon of action-chunking with MTP's convergence benefits.
- Efficiency: fully parallelized teacher-forced training with LoRA fine-tuning preserves the video-pretrained capabilities; KV-caching enables real-time inference. Training fits on a single A100 within 60 GPU hours—a notably modest compute budget.
Empirical results
On LIBERO, evaluated over 500 rollouts per suite with roughly 400 demonstrations each, PhysGen achieves a 90.8% average success rate:
| Method |
Action pretraining |
Avg. SR |
| OpenVLA (7B) |
Yes |
77% |
| Pi0-Fast (3B) |
Yes |
86% |
| WorldVLA (7B) |
— |
82% |
| PhysGen (732M) |
No |
90.8% |
The strongest gain is an 18.8-point improvement over WorldVLA on LIBERO-Long, which the authors attribute to continuous tokenization avoiding quantization-induced drift in long-horizon rollouts. PhysGen trails Pi0-Fast only on LIBERO-Spatial, a limitation ascribed to limited spatial perception of the underlying video backbone. On ManiSkill, PhysGen averages 74%, exceeding ICRT by 12% and π₀ by 5%, though it trails RDT-1B—the paper concedes this directly—and achieves a perfect 100% on PushCube.
Real-world experiments on a Franka Panda (80–100 teleoperated demonstrations per task, 20 trials each) yield a 75% average success rate, matching π₀ despite the absence of any large-scale action pretraining. On Pick Transparency—grasping a transparent cube where refraction produces ambiguous observations—PhysGen exceeds π₀ by 5 points, supporting the claim that video-derived physical intuition specifically aids physically demanding manipulation.
Ablations
Ablations on Libero-Object isolate each component's contribution:
| Variant |
Change |
SR |
| PhysGen-Zero |
No pretraining |
86.4% |
| PhysGen-Discrete |
Discrete action tokens |
94.2% |
| PhysGen-NoAR |
No autoregressive rollout |
95.0% |
| PhysGen-STP |
L-MTP removed |
96.8% |
| PhysGen-Full |
— |
99.6% |
Video generation pretraining contributes the largest margin (+13.2%), followed by continuous over discrete tokenization (+5.4%), the autoregressive formulation (+4.6%), and L-MTP (+3.4%). Qualitative analyses show predicted videos closely aligned with executed trajectories—including correct arm reorientation before grasping—and attention maps concentrating on task-critical regions such as the cube and gripper.
Limitations and open questions
The paper acknowledges that PhysGen underperforms Pi0-Fast on LIBERO-Spatial, indicating that the spatial reasoning of the current video backbone bounds policy performance. Its advantage over π₀ on transparency-related tasks rests on four tabletop tasks with modest trial counts (20 per task), leaving statistical strength of the real-world comparison open. The evaluation also does not test whether gains persist under distribution shift beyond fixed task suites. A concrete open question is how to strengthen spatial perception in video-generative backbones without sacrificing the physical priors that drive the reported improvements.
Conclusion
PhysGen demonstrates that a 732M-parameter model fine-tuned solely from a video-generation prior can outperform substantially larger action-pretrained VLAs on standard benchmarks and match π₀ in real-world execution. The ablations substantiate the two core claims—video pretraining supplies transferable physical knowledge, and continuous joint tokenization outperforms discrete alternatives—with quantified evidence. The result positions pretrained autoregressive video models as a viable and data-efficient alternative foundation for visuomotor policies.