Evaluating Gemini Robotics Policies in a Veo World Simulator (2512.10675v1)

Abstract: Generative world models hold significant potential for simulating interactions with visuomotor policies in varied environments. Frontier video models can enable generation of realistic observations and environment interactions in a scalable and general manner. However, the use of video models in robotics has been limited primarily to in-distribution evaluations, i.e., scenarios that are similar to ones used to train the policy or fine-tune the base video model. In this report, we demonstrate that video models can be used for the entire spectrum of policy evaluation use cases in robotics: from assessing nominal performance to out-of-distribution (OOD) generalization, and probing physical and semantic safety. We introduce a generative evaluation system built upon a frontier video foundation model (Veo). The system is optimized to support robot action conditioning and multi-view consistency, while integrating generative image-editing and multi-view completion to synthesize realistic variations of real-world scenes along multiple axes of generalization. We demonstrate that the system preserves the base capabilities of the video model to enable accurate simulation of scenes that have been edited to include novel interaction objects, novel visual backgrounds, and novel distractor objects. This fidelity enables accurately predicting the relative performance of different policies in both nominal and OOD conditions, determining the relative impact of different axes of generalization on policy performance, and performing red teaming of policies to expose behaviors that violate physical or semantic safety constraints. We validate these capabilities through 1600+ real-world evaluations of eight Gemini Robotics policy checkpoints and five tasks for a bimanual manipulator.

• The paper introduces an action-conditioned video modeling framework that ranks robotics policies across nominal, OOD, and safety-critical domains.
• It leverages a Veo simulator with latent diffusion and transformer-based denoising to generate photorealistic, multi-view scene rollouts.
• Real-world trials validate simulation predictions using metrics like MMRV (0.06) and Pearson correlation (0.86), confirming predictive reliability.

Evaluating Gemini Robotics Policies Using Video World Models

Introduction and Motivation

The paper "Evaluating Gemini Robotics Policies in a Veo World Simulator" (2512.10675) presents an extensible, action-conditioned video modeling framework for scalable, rigorous policy evaluation in robotics. It directly addresses a principal challenge in modern robotics: the evaluation of generalist, vision-language-activated manipulation policies across nominal, out-of-distribution (OOD), and safety-critical domains. Unlike physics-based simulators, which require substantial manual engineering and asset creation to provide convincing and general benchmarks, the Veo-based system leverages generative world models for photorealistic, multi-view-consistent scene synthesis and simulation.

This approach also enables systematic policy "red teaming" in semantically rich and safety-critical scenarios, a domain where hardware-based evaluation is frequently intractable due to risk, cost, or practical limitations. The authors complement their simulated evaluations with more than 1600 real-world trials executed across five manipulation tasks using eight policy checkpoints, thereby calibrating and validating the fidelity of their video model evaluator.

System Architecture and Methodology

The evaluation pipeline is centered on the Veo video foundation model, which utilizes a latent diffusion backbone coupled with a transformer-based denoising network. The model is pretrained on an extensive collection of diverse web-sourced videos, filtered for safety and quality, and semantically deduplicated to prevent memorization. For robotics, the system is finetuned using large-scale, multi-task datasets collected via real robot teleoperation (notably with the ALOHA 2 bimanual platform).

Through action conditioning, the model accepts a current multi-camera observation and a sequence of future robot poses, synthesizing future scene observations that respect both action semantics and high visual fidelity. Fine-tuning for multi-view consistency is critical; the method generates temporally and spatially consistent rollouts across all robot camera viewpoints. Generative editing (with Gemini-powered image models) is used for scene augmentation to assess OOD generalization, such as adding novel objects, distractors, or backgrounds.

Policy Evaluation Under Nominal Conditions

The primary objective in nominal settings is to assess if the video-simulated rollouts provide accurate policy ranking and performance prediction. Evaluations span five canonical vision-language manipulation tasks. For each task, policy success is measured over 80 scene-instruction pairs, introducing linguistic and visual variation. Eight vision-language-action policies (GROD variants) are compared.

Strong quantitative alignment is established between real-world performance and video model predictions, particularly in policy ordering. The paper introduces the mean maximum rank violation (MMRV) metric to summarize rank consistency and computes the Pearson correlation coefficient for direct measurement of linear agreement between predicted and empirical policy success rates.

Policy Evaluation for Out-of-Distribution Generalization

Comprehensive evaluation in OOD regimes is designed by manipulating observed scenes along four axes:

1. Background: Changing tablecloth or backdrops (e.g., red, green, or blue).
2. Small Distractor: Introducing novel small plush toys (e.g., axolotl, duck, turtle).
3. Large Distractor: Placement of larger objects (e.g., bighorn sheep plush).
4. Object: Replacing manipulated objects with semantically novel ones (e.g., toy kitchen brush).

Multi-view-consistent observations for these edited configurations are generated with the Veo model, then policies are rolled out in silico using these synthetic observations.

The evaluation reveals fine-grained, axis-specific generalization capacity: for instance, background and distractor shifts minimally impact competent policies, while novel target objects yield substantial performance degradation. The video model's per-axis predictions closely track real-world results: for the main policy, the MMRV is as low as 0.06, and Pearson coefficients for success rate correlation reach 0.86. These results demonstrate the model's ability to both order and relatively quantify generalization-induced failures.

Visualizations of the specific distractors and objects used, such as the axolotl (Figure 1), bighorn sheep (Figure 2), kitchen brush (Figure 3), and red backgrounds (Figure 4), provide direct insight into the OOD axis construction and policy failure modes.

Figure 5: Example of a small distractor object (axolotl) used for OOD robustness evaluation.

Figure 6: Large distractor (bighorn sheep plush) used in OOD scene creation.

Figure 7: Novel object (pink plastic kitchen brush) requiring manipulation under OOD conditions.

Figure 8: Example of a scene with an altered background (red cloth) to evaluate visual robustness.

Safety-Critical Evaluation via Predictive Red Teaming

To surface policy failure modes in safety and semantics, the authors employ red-teaming by generating scenes containing hazards (e.g., sharp tools, liquids, ambiguous destinations, or human actors) and ambiguous user requests. Scenarios are filtered to ensure hazard relevance, multimodal complexity, and ambiguity such that policy rollouts – rather than static inspection – are required for evaluation.

Representative failures synthesized in the video simulator include inadvertent physical human contact, unsafe manipulation near hazards, or incorrect sequential handling (e.g., closing a laptop onto scissors). Crucially, these unsafe behaviors are reliably replicated in corresponding real-world experiments, underscoring the predictive utility of the approach. The core contribution here is scalable, interactive, safety-focused policy assessment before hardware deployment.

Figure 10: Instances of unsafe behavior detected by red-teaming policies in the Veo-based simulator and replicated on hardware.

Limitations and Outlook

While the method achieves high policy ordering fidelity and expressive scenario coverage, certain limitations persist:

• The simulator occasionally exhibits generative artifacts under heavy contact or high-entropy interactions, such as object hallucination or duplication.
• Long-horizon, temporally extended rollouts (beyond 8s) remain an open technical challenge for scene-consistent video generation.
• Full automation of policy evaluation (video scoring) is not realized; human assessors are currently used, although VLM-based automated scorers are proposed as a next step.

Algorithmic and architectural advances in longer-horizon latent action modeling, increased interaction data scale, and enhanced inference efficiency are poised to deepen fidelity and scalability. The framework lays the groundwork for rich, continual, and principled evaluation of generalist robotic manipulation policies independent of manual simulators or real-world trial volume.

Conclusion

This work demonstrates that action-conditioned, frontier video models can function as generalist evaluators for the broad spectrum of robot policies, tightly coupling benchmark flexibility, action-responsiveness, and visual realism. By validating with large-scale hardware experiments, the authors show that rankings and policy failure modes identified in simulation align strongly with real-world outcomes. The implications are significant: scalable, generative simulators permit rigorous, continuous evaluation across nominal, OOD, and safety-critical regimes, accelerating the development and safe deployment of generalist embodied intelligence. Future research should focus on improving simulator fidelity for long-horizon reasoning and fully automating assessment pipelines via vision-LLMs and robust, data-driven scene synthesis.