Robot Learning from a Physical World Model (2511.07416v1)

Abstract: We introduce PhysWorld, a framework that enables robot learning from video generation through physical world modeling. Recent video generation models can synthesize photorealistic visual demonstrations from language commands and images, offering a powerful yet underexplored source of training signals for robotics. However, directly retargeting pixel motions from generated videos to robots neglects physics, often resulting in inaccurate manipulations. PhysWorld addresses this limitation by coupling video generation with physical world reconstruction. Given a single image and a task command, our method generates task-conditioned videos and reconstructs the underlying physical world from the videos, and the generated video motions are grounded into physically accurate actions through object-centric residual reinforcement learning with the physical world model. This synergy transforms implicit visual guidance into physically executable robotic trajectories, eliminating the need for real robot data collection and enabling zero-shot generalizable robotic manipulation. Experiments on diverse real-world tasks demonstrate that PhysWorld substantially improves manipulation accuracy compared to previous approaches. Visit \href{https://pointscoder.github.io/PhysWorld_Web/}{the project webpage} for details.

• The paper presents PhysWorld, a framework that transforms task-conditioned video demonstrations into physically feasible robotic actions.
• It integrates 4D spatio-temporal reconstruction and object-centric residual RL to achieve an 82% task success rate in robotic manipulation.
• The approach demonstrates enhanced generalization and robustness over traditional methods, reducing reliance on extensive real-world demonstrations.

Robot Learning from a Physical World Model

Introduction

The research paper titled "Robot Learning from a Physical World Model" introduces PhysWorld, a novel framework that integrates video generation with physical world modeling to enhance robotic learning capabilities. The primary focus is to transform task-conditioned video demonstrations into physically accurate robotic actions without relying on extensive real-world demonstrations. The two-pronged approach leverages the strengths of video generation for visual guidance and a physics-based model to ensure feasible actions. This framework is significant for enabling zero-shot generalizable robotic manipulation by grounding generated visual demonstrations into executable actions suitable for the real world.

Figure 1: PhysWorld pipeline. It integrates video generation and physical world modeling to transform an RGB-D image and a task prompt into feasible robotic actions.

Methodology

Physical World Modeling from Video Generation

PhysWorld begins with video generation from a single RGB-D image and task prompt. The video generation process uses models such as Veo3 to produce task-conditioned visual demonstrations. However, these videos, albeit visually plausible, don't inherently adhere to the physical constraints necessary for robotic tasks. To address this, PhysWorld reconstructs a 4D spatio-temporal representation from the generated video (Figure 2). This involves estimating depth, generating textured meshes, and assembling them into a physically consistent scene by optimizing for gravity alignment and collision constraints.

Figure 2: Qualitative evaluation of physical scene modeling from generated videos.

Object-Centric Learning Using Residual RL

Building upon the physical model, PhysWorld employs object-centric learning to track object motions, which are less susceptible to visual artifacts than embodiment motions. Residual Reinforcement Learning (RL) is utilized, which allows a policy to refine baseline grasping and motion-planning actions, adapting effectively through physical feedback from the model. This synergy between learned residual policies and physics constraints underpins the transformation of visual demonstrations into feasible actions.

Figure 3: Residual RL vs. RL from Scratch.

Experiments

Video Generation and World Modeling

Experimental evaluations demonstrate that PhysWorld's integration of video generation and physics-based modeling significantly enhances manipulation accuracy across diverse tasks. Compared to methods that directly follow visual cues, PhysWorld exhibited superior generalization and robustness, showcasing an 82% average task success rate.

Figure 4: Quantitative evaluation of PhysWorld on real-world manipulation tasks.

Comparative Analysis

PhysWorld's reliance on object-centric tracking outperforms embodiment-centric approaches, particularly where generated videos misrepresent human motion. The framework's ability to build a digital twin of the task environment from minimal data inputs underscores its scalability potential.

Figure 5: Qualitative evaluation of PhysWorld on real-world manipulation tasks.

Implications and Future Directions

The paper highlights the implications of integrating simulation-based learning with generative models for robotics. By eliminating the need for large datasets of real-world demonstrations, PhysWorld sets a precedent for leveraging synthetic training signals. Future research could explore enhanced fidelity in physical simulations and expand multi-modal policy learning to further diminish the sim-to-real gap.

Conclusion

PhysWorld presents a significant advancement in robot learning methodology by effectively coupling video generation with a robust physical modeling framework. It marks a pivotal step towards practical, generalizable robotic manipulation across diverse real-world tasks, opening avenues for further innovations in AI-driven robotics.

In conclusion, while still susceptible to the limitations of physical simulators, PhysWorld illustrates the potential of combining generative models with reinforcement learning to achieve scalable and accurate robotic task execution. The release of code and project resources by the authors is anticipated to stimulate further advancements in this domain.