World-Gymnast: Training Robots with Reinforcement Learning in a World Model

Abstract: Robot learning from interacting with the physical world is fundamentally bottlenecked by the cost of physical interaction. The two alternatives, supervised finetuning (SFT) from expert demonstrations and reinforcement learning (RL) in a software-based simulator, are limited by the amount of expert data available and the sim-to-real gap for manipulation. With the recent emergence of world models learned from real-world video-action data, we ask the question of whether training a policy in a world model can be more effective than supervised learning or software simulation in achieving better real-robot performance. We propose World-Gymnast, which performs RL finetuning of a vision-language-action (VLA) policy by rolling out the policy in an action-conditioned video world model and rewarding the rollouts with a vision-LLM (VLM). On the Bridge robot setup, World-Gymnast outperforms SFT by as much as 18x and outperforms software simulator by as much as 2x. More importantly, World-Gymnast demonstrates intriguing capabilities of RL with a world model, including training on diverse language instructions and novel scenes from the world model, test-time training in a novel scene, and online iterative world model and policy improvement. Our results suggest learning a world model and training robot policies in the cloud could be the key to bridging the gap between robots that work in demonstrations and robots that can work in anyone's household.

• The paper introduces a reinforcement learning framework that trains vision-language-action policies within action-conditioned video world models, achieving up to 18-fold improvement over supervised fine-tuning.
• It leverages pretrained vision-language models for robust reward assignment and data-efficient policy training, enabling effective zero-shot adaptation on real robots.
• The method bridges the sim-to-real gap by iteratively co-training policies and the world model with Dyna-style updates, enhancing generalization to out-of-distribution scenarios.

Reinforcement Learning for Robotic Manipulation in World Models: An Analysis of World-Gymnast

Summary and Core Contributions

"World-Gymnast: Training Robots with Reinforcement Learning in a World Model" (2602.02454) introduces a reinforcement learning (RL) framework for training vision-language-action (VLA) policies in robots using action-conditioned video world models, rather than physical interaction or software simulation. By leveraging pretrained world models and vision-LLMs (VLM) for reward assignment, World-Gymnast enables scalable, data-efficient robot training directly from real-world video-action datasets. The reported experiments demonstrate that RL in such a learned world model outperforms both supervised fine-tuning (SFT) and RL in high-fidelity physics simulators, exhibiting notably higher success rates when policies are deployed on real robots.

Methodological Framework

World-Gymnast fine-tunes VLA policies by rolling them out in a video world model, using an action-conditioned video generation model akin to "WorldGym" [Quevedo et al., 2025]. The RL pipeline involves generating imagined rollouts by executing policy-sampled actions in the world model, where potential success is assessed by a VLM that produces binary task completion rewards. Policy updates use a PPO-style clipped objective with advantages estimated via group normalization.

Distinctive elements of the approach include:

• Group Relative Policy Optimization (GRPO): Utilized to stabilize on-policy gradient estimation, allowing for robust learning in high-variance reward settings.
• VLM-based Reward Assignment: Task success in generated video rollouts is judged using a VLM such as GPT-4o, which is prompted with the initial instruction and the sequence of generated frames.
• Diverse and OOD Training: The use of image augmentation (distractors), language variation, and synthesized novel scenarios during RL fine-tuning augments policy robustness and generalization.
• Test-time Adaptation and Iterative Improvement: Policies can be adapted online at test time from novel scenes with no additional physical data collection, and the world model itself can be continually refined from real-robot interactions via Dyna-style updates.

Empirical Results and Strong Claims

Experiments on the Bridge platform and AutoEval [Zhou et al., 2025] reveal several robust findings:

• Significant Outperformance of Baselines: World-Gymnast achieves up to 18-fold improvement over SFT and up to 2-fold improvement over software RL sim-to-real transfer, notably for the "put eggplant into blue sink" and "put eggplant into yellow basket" tasks.
• Superior Generalization: RL with the world model generalizes more effectively to OOD language and scene variations, showing an increased success rate when training on language-augmented or visually-distracted initial frames.
• Test-time Learning: Zero-shot adaptation from a single novel scene yields dramatic performance improvements (close the drawer: 62%→100% success).
• Iterative Co-training: Continuous data flywheel between policy and world model (via Dyna) reduces the sim-to-real gap and further increases real-world policy reliability.

The authors emphasize that policies trained in the world model not only surpass those trained via SFT or in simulation but also are more robust to visual clutter and instruction diversity. Iterative refinement of the world model using real-robot data leads to further gains, with qualitative rollouts closely tracking real robot behavior and mitigating compounding world model errors.

Theoretical and Practical Implications

From a theoretical standpoint, World-Gymnast validates the utility of action-conditioned video world models—trained directly on real data—as both simulators and policy optimization environments, sidestepping the sim-to-real transfer issue inherent in physics-based simulation. The approach demonstrates that RL-based active exploration, even in a learned model with imperfect physics, leads to more generalizable and robust policies compared to imitation-based or simulator-based methods.

Practically, these findings have significant ramifications for democratizing robotics research. Training and evaluating in a learned model eliminates the need for costly physical rollouts or extensive engineering of digital twins, enabling rapid iteration and policy adaptation in cloud-based settings.

However, some limitations remain: the system's ability to generalize to arbitrarily OOD frames is bounded by the support of the world model's training distribution; reward model hallucinations can lead to suboptimal RL updates, and using dense VLM rewards exacerbates risk of reward hacking. Addressing these points (e.g., via broader world model pretraining or improved reward models such as RoboReward [Lee et al., 2026]) is an explicit direction for future work.

Future Research Directions in AI and Robotics

• Scaling Foundation World Models: Pretraining on broader and more heterogeneous real-robot datasets will be critical for out-of-distribution robustness.
• Improved Reward Modeling: Developing or fine-tuning reward models that are less prone to hallucination will mitigate reward misattribution issues.
• Dense and Structured Reward Integration: Leveraging richer, denser vision-language signals could facilitate more nuanced policy learning and task decomposition.
• Unified Frameworks for Model-Policy Co-Training: Generalizing the iterative Dyna-style procedure to support continual learning for lifelong robot skill adaptation.
• Reliability and Safety Guarantees: Combining learned world model deployment with formal or empirical safety verification before real-world deployment will be essential as these frameworks are scaled in practice.

Conclusion

World-Gymnast (2602.02454) sets a precedent for data-driven reinforcement learning in robotics by leveraging action-conditioned video world models and VLM-based reward assignment. The framework enables practical, scalable, and robust policy optimization that bridges the sim-to-real gap and accelerates research progress towards generalist robots. The results demonstrate strong empirical superiority over SFT and traditional simulation, and open clear avenues for future exploration in world model scalability, reward learning, and closed-loop policy-environment adaptation.