Efficient Sim-to-Real Transfer of World-Action Models from Synthetic Priors

Abstract: Bridging the sim-to-real gap is a core challenge in deploying learned manipulation policies. Sim-to-real learning is attractive because it can replace expensive real robot demonstrations with scalable synthetic data, yet world-action models have not previously been shown to transfer from simulation to real robotic manipulation. We study whether a world-action model can be trained from synthetic priors and deployed zero-shot in the real world. To this end, we build upon Cosmos Policy, a video diffusion model adapted for visuomotor control. We construct simulation environments with extensive domain randomization and generate demonstrations using the AnyTask motion planning pipeline. We evaluate our approach across object lifting, drawer opening, and pick-and-place tasks using ${\sim}800$ synthetic demonstrations per task and no real demonstrations. When deployed zero-shot on a Franka Robot, our policy attains a 35\% average success rate. To our knowledge, this represents the first successful sim-to-real transfer of a world-action model for robotic manipulation.

• The paper shows that a world-action policy trained solely on synthetic data achieves zero-shot execution in real-world tasks with a 35% success rate.
• The Cosmos Policy framework leverages video diffusion and extensive domain randomization to jointly optimize perception and control in robotic manipulation.
• The approach exhibits strong out-of-distribution generalization, reducing reliance on real-world demonstrations and offering data-efficient robotic skill acquisition.

Efficient Sim-to-Real Transfer of World-Action Models from Synthetic Priors

Introduction

This work addresses the persistent challenge of sim-to-real transfer in robotic manipulation by empirically evaluating the direct deployment of video-based world-action models trained exclusively on synthetic data. The authors employ the Cosmos Policy framework—a video diffusion architecture post-trained for visuomotor control—leveraging fully automated domain-randomized simulations and synthetic expert demonstrations generated via AnyTask. They demonstrate, for the first time, that a world-action policy trained only on synthetic priors is capable of zero-shot execution of real-world robotic tasks with nontrivial success rates, thereby indicating the practical feasibility of such generative policies for data-efficient manipulation.

Methodology

The central component is Cosmos Policy, an adaptation of a pretrained video diffusion model (Cosmos-Predict2) to the visuomotor control setting. This architecture encodes actions, future observations, and value predictions as a sequence of latent video frames, enabling joint optimization over perception and action conditioned on historic inputs.

Synthetic demonstrations are collected using GPU-accelerated simulation (Isaac Gym) with extreme domain randomization across textures, lighting, object placements, and camera parameters. AnyTask is used to automatically generate $\sim$800 demonstrations per task (bananas, bricks, drawers, strawberries), each comprising paired RGB observations and action sequences. No real-world data or teleoperation is involved at any stage.

Training proceeds for 32 epochs on 40 H100 GPUs (about 72 hours), with the policy exposed only to procedurally randomized synthetic data. The policy is then deployed, without adaptation or fine-tuning, on a Franka Research 3 robot in real-world scenarios.

Empirical Findings

A qualitative analysis reveals a strong match between the policy's predicted next-frame RGB views and live camera feeds across various manipulation phases. The predicted states—sampled from the world-action model during real executions—preserve critical spatial and semantic task attributes despite the complete lack of real-world supervision.

Figure 1: Qualitative comparison of predicted and live camera frames during “put strawberry into bowl,” validating perceptual alignment between sim-trained predictions and real-world execution.

Quantitative metrics reveal an average real-world task success rate of 35% across four distinct manipulation tasks, with zero use of real robot demonstrations. Notably, the method outperforms Diffusion Policy models trained on 10 or even 50 real demos per task when considering average success rates under the evaluated setup. These results indicate that, with extensive domain randomization and sufficiently large-scale synthetic priors, world-action models can generalize to the real world under zero-shot conditions.

The model also exhibits generalization to out-of-distribution objects not encountered during training. When presented with a novel bottle, the policy plan and execute the approach, grasp, and lift phases with no additional supervision.

Figure 2: The sim-to-real policy successfully lifts a bottle object absent from the synthetic training set, demonstrating OOD generalization in physical tasks.

Discussion and Implications

This study provides robust evidence that world-action models, embodied as generative video diffusion policies, are amenable to direct sim-to-real transfer with no real data, given sufficiently diverse synthetic training. The demonstrated 35% success rate, while leaving ample margin for further improvement, establishes a lower bound on the attainable accuracy given purely procedural training. Of particular note is the validated OOD generalization, which suggests that such models do not overfit to visual or geometric specifics of training objects, but rather acquire reusable visuomotor representations and control policies.

Practically, the elimination of real-world demonstrations for new manipulation tasks offers large-scale data efficiency and reduces the economic and time costs associated with robot policy training. Theoretically, these results suggest that the world-action modeling paradigm, when paired with large-scale synthetic priors and powerful generative models, is not fundamentally limited by the sim-to-real visual domain gap.

Limitations include the moderate baseline success rates and the need for extensive computational resources (72 hours, 40 H100s). The relative contribution of the joint action-video prediction objective versus domain randomization and pretrained priors remains to be explicitly disentangled via controlled ablation studies, as acknowledged by the authors.

Conclusion

This work demonstrates that world-action models based on video diffusion architectures can, for the first time, be transferred directly from simulation to real-world robotic manipulation without any use of real-world demonstrations. The empirical results—with a 35% average success rate and strong qualitative perceptual alignment—even under out-of-distribution evaluation, solidify the viability of generative sim-to-real policy learning and point to promising directions in data-efficient, scalable robotic skill acquisition. Future research should address optimization of training efficiency, further close the sim-to-real performance gap, and investigate ablation of core components underlying robust transfer.