RoboWorld: Fast and Reliable Neural Simulators for Generalist Robot Policy Evaluation

Abstract: Video world models are emerging as a scalable alternative for evaluating generalist robot policies, bypassing the physical constraints and engineering burdens of real-world deployment. However, evaluating policies with video world models remains challenging, as world-model errors can make generated rollouts unreliable and slow inference limits large-scale throughput. We introduce RoboWorld, an automated evaluation pipeline that pairs a fast autoregressive video world model with a task-progress-aware vision-LLM scoring. To enable reliable long-horizon autoregressive world-model rollouts, we propose Step Forcing, which combines anchored and one-step self-forwarded contexts to reduce train--test mismatch while preserving action--observation dynamics. Together, these components enable RoboWorld to align strongly with real-world robot evaluation across tasks and environments, achieving Pearson's r = 0.989 and Spearman's \r{ho} = 0.970.

• The paper introduces Step Forcing, a novel autoregressive training technique that reduces train-test mismatch in video-based neural simulators.
• It integrates fast, action-conditioned video rollouts with a six-level VLM rubric to capture nuanced policy progress and errors.
• Empirical results demonstrate high correlation with real-world evaluations, enabling efficient and scalable robot policy testing in varied settings.

RoboWorld: Fast and Reliable Neural Simulators for Generalist Robot Policy Evaluation

Introduction and Motivation

The evaluation of generalist robot policies, particularly those based on Vision-Language-Action (VLA) models, faces substantial scalability challenges due to the reliance on physical robot deployments and the overhead of environment-specific simulation. RoboWorld, introduced in "RoboWorld: Fast and Reliable Neural Simulators for Generalist Robot Policy Evaluation" (2607.01060), proposes a scalable, automated pipeline that leverages fast autoregressive video world models and a task-progress-aware Vision-LLM (VLM) judge. This framework is designed to enable closed-loop policy evaluation across diverse tasks and environments, aiming to closely approximate real-world evaluation outcomes while significantly reducing the required computational and manual resources.

Step Forcing: Bridging the Train–Test Gap in Video World Models

Long-horizon, action-conditioned video generation for policy rollouts in neural simulators tends to amplify model errors due to compounding inaccuracies across time steps (exposure bias). Prior methods—Teacher Forcing, Diffusion Forcing, Resampling Forcing, Self Forcing—mitigate this but come with cost, dynamics mismatch, or action controllability degradation. RoboWorld introduces Step Forcing, an autoregressive training objective that reduces the train–test mismatch by aligning the denoising schedule between training and inference and mixing one-step self-forwarded priors and anchor steps. This approach allows the model to adapt to its own outputs during generation while preserving accurate action–observation dynamics by periodic anchoring to ground-truth states.

Figure 2: Visual summary of Step Forcing modalities—Diffusion Forcing (red, on noisy ground-truth), Self Forcing (green, on self-generated context), and Step Forcing (combination with one-step self-forwarded priors and anchors).

The primary mechanism of Step Forcing is to use the same discrete denoising schedule both during training and inference. For each frame, the model is trained to predict the clean observation from a context either self-forwarded by one denoising step (simulating inference conditions) or directly from data (anchor), determined by a probability $p$. The stop-gradient operation is employed on the self-forwards to prevent the model from optimizing toward trivial solutions (e.g., making its own output artificially easy to denoise).

Ablations demonstrate that both self-forwarding and periodic anchoring are critical: removing either results in significant degradation in Fréchet Video Distance (FVD) and perceptual quality, particularly for dynamic wrist-camera views in robotics, which are sensitive to small rollout drifts.

RoboWorld Evaluation Pipeline

The full RoboWorld pipeline integrates two core components: the Step Forcing-trained autoregressive video world model, and a VLM-based evaluation rubric that goes beyond binary success/failure metrics to capture nuanced task progress.

Fast Autoregressive World Modeling

RoboWorld adapts large-scale, pretrained bidirectional video diffusion models into causal, autoregressive world simulators. Action conditioning is injected per-frame via a dedicated MLP and cross-attention. Architectural modifications include:

• Causal temporal attention (frame-level) for action-aware prediction.
• Per-frame independent noise scheduling (aligned via Step Forcing).
• Key-Value caching with a sliding window for efficient inference.

This results in an order-of-magnitude speedup in FPS for long-horizon rollouts (up to 15.3 FPS with 4 denoising steps), with competitive or superior quality on LPIPS, SSIM, and FVD compared to far slower non-autoregressive or distillation-based methods.

Figure 1: Quality and throughput of long-horizon, action-conditioned video generation on RoboArena—Step Forcing achieves the best perceptual metrics at maximal generation speed.

Task-Progress-Aware VLM Judging

Instead of collapsing policy evaluation to a single success/failure per trial, RoboWorld employs a six-level rubric scored by a VLM (e.g., GPT-4o). The rubric assigns scores from 0–5 and distinguishes progress, policy errors, and world-model-induced artifacts. A critical distinction is made between observation views: only the stable, external camera views are used for success/failure judgment, while the dynamic wrist view is reserved strictly for world-model failure identification. This prevents artifacts from overly penalizing otherwise correct policies.

Figure 3: The evaluation rubric is designed to primarily reflect task progress, assigning different penalties depending on when and how world-model errors occur.

Empirical Results and Correlation with Real-World Evaluation

The RoboWorld world model is pretrained on DROID and evaluated on RoboArena trajectories. Rollouts generated by the policy in the world model (using the same initial observation as the real-world episode) are scored by the VLM rubric and compared directly to the official RoboArena leaderboard across policies and tasks.

Key Results:

• Pearson $r = 0.989$, Spearman $\rho = 0.970$ correlation with the real-world RoboArena leaderboard across 8 open-sourced policies, demonstrating that RoboWorld preserves the holistic policy ranking obtained via physical trials.
• The correlation increases with the number of evaluation trials, indicating the necessity of environmental/task diversity for robust benchmarking.
• Qualitative analysis confirms that RoboWorld rollouts capture both correct execution and nuanced policy failure modes, and that the evaluation rubric effectively separates policy errors from model artifacts.

  Figure 5: (a) Strong correlation between RoboWorld and RoboArena evaluations; (b) Correlation increases with more diverse trials; (c) Rollout examples for policy-specific success/failure patterns.

Ablations and Rubric Analysis

• Using binary success rates instead of the task-progress-aware rubric reduces ranking correlation ($\rho = 0.922$ vs. $0.970$).
• Incorporating the wrist view into rubric-based success judgments further reduces reliability ($\rho = 0.862$).
• The comparative advantage of the full rubric is robust across different VLM judges (e.g., Gemini-2.5-Flash), indicating method independence from a single LLM implementation.

Extreme Environment Evaluation

RoboWorld is tested for robustness in synthetic, visually altered settings (e.g., disaster sites, spacecraft interiors) constructed from real robot imagery. It maintains a high correlation ($r=0.970$) with the RoboArena leaderboard, supporting the assertion that RoboWorld can facilitate reliable, large-scale policy evaluation in environments where direct physical trial is infeasible.

Figure 6: RoboWorld rollouts and closed-loop policy evaluations in extreme, synthetic environment reconstructions.

Limitations and Future Directions

While RoboWorld sets a new standard for scalable neural robot policy evaluation, several challenges persist:

• Contact-rich object dynamics and dexterous manipulation over long horizons remain failure points for current world models, as evidenced by recurring object morphing/disintegration artifacts during manipulation.
• Training data diversity in large-scale human–object interaction may be a prerequisite for overcoming world model limitations in these domains.
• Improvement of VLM-based evaluators to reduce score inflation and account for subtle task errors observed in manual audits is warranted.

Conclusion

RoboWorld provides a principled and computationally efficient framework for automated policy evaluation in robotics, delivering fast, high-fidelity rollouts with strong alignment to real-world results. By integrating Step Forcing for stable long-horizon video generation and a fine-grained, artifact-aware VLM rubric, it closes several methodological gaps present in prior world-model-based or simulation-based benchmarks. The system has implications for safety-critical deployment, rapid iterative policy development, and simulation-based predeployment risk assessment in arbitrary or inaccessible environments. Future extensions may involve scaling world models and rubrics to address failures in rich contact scenarios, and developing increasingly reliable VLM judges.