DreamGen Bench: Robot Video Evaluation

• DreamGen Bench is a diagnostic framework that quantifies the fidelity and physical plausibility of text-to-video models generating robot manipulation videos.
• It evaluates models across diverse generalization axes—objects, behaviors, and environments—using benchmarks like RoboCasa and GR1 Humanoid.
• The framework employs metrics such as Instruction Following and Physics Alignment that strongly predict downstream policy performance in robot learning.

DreamGen Bench is a diagnostic framework designed to evaluate the capability of text-to-video world models to generate robot manipulation videos that comply with high-level instructions and maintain physical plausibility for diverse robotic embodiments and environments. Its primary objective is to quantify the fidelity and utility of synthetic data for robot learning, providing a proxy prediction for downstream policy performance while circumventing the resource constraints and setup complexity of direct robot experimentation (Jang et al., 19 May 2025).

1. Motivation and Definition

DreamGen Bench was developed to address the gap in systematic, low-cost evaluation of world models intended for robot learning. The framework explicitly quantifies, in a repeatable and targeted fashion, how effectively generative video models "dream up" robot experiences that are both instruction-compliant and physically realistic, a prerequisite for their utility in visuomotor policy learning. High DreamGen Bench scores have been empirically shown to correlate strongly with policy performance in the RoboCasa benchmark suite, operationalizing the insight that high-quality synthetic videos lead to robust policy transfer without extensive teleoperation (Jang et al., 19 May 2025).

2. Benchmark Structure and Testbeds

DreamGen Bench encompasses evaluation tasks built around two core datasets:

• RoboCasa (simulation), featuring a Franka Emika Panda manipulator
• GR1 Humanoid (real-world), using the NVIDIA/Gear GR1 robot

Each dataset is partitioned along three generalization axes:

1. Object Generalization: Models generate pick-and-place rollouts involving novel objects never seen during fine-tuning.
2. Behavior Generalization: Videos depict previously unseen verbs (e.g., pour, hammer, swivel) performed in the training environment.
3. Environment Generalization: Behaviors are enacted in ten new room layouts with novel camera angles and increased background clutter.

Across both datasets, evaluation comprises thousands of rollouts, spanning approximately 16–21 unique objects, 10–14 distinct behaviors, and 5–10 different environments, with ∼6,000 and ∼8,000 frames for RoboCasa and GR1 respectively. Models are evaluated in both zero-shot (–zero) and fine-tuned (–sft) variants, yielding eight configurations per base video model.

Dataset	Objects	Behaviors	Environments	Frames (∼)
RoboCasa	16	10	5	6,000
GR1 Humanoid	21	14	10	8,000

3. Synthetic Data Generation: Pipeline Overview

The DreamGen Bench protocol leverages the DreamGen pipeline up to the video generation stage, consisting of:

1. Fine-tuning: State-of-the-art video diffusion models (WAN 2.1, Cosmos, Cog VideoX, HunyuanVideo) are LoRA-adapted using thousands of teleoperated robot trajectories, ensuring precise embodiment adaptation.
2. Rollout: The model receives an initial RGB frame and a natural language instruction (for either a seen or novel task and/or environment), generating a video sequence of the robot performing the task as described.
3. (Optional) Pseudo-action Recovery: Although not included within DreamGen Bench proper, generated videos can be post-processed using inverse-dynamics models (IDM) or a latent-action model (LAPA) to obtain per-frame pseudo-action labels for downstream policy training.
4. Policy Training: The complete DreamGen pipeline involves training policies on these neural trajectories, but DreamGen Bench limits evaluation to the generated video quality.

This protocol isolates the evaluation of the video world model's capacity for reliable generation distinct from the policy learning process itself.

4. Evaluation Metrics

DreamGen Bench utilizes two principal automatic metrics, Instruction Following (IF) and Physics Alignment (PA):

• Instruction Following (IF): For each generated video $v$, a vision-LLM (Qwen-VL 2.5) is provided the prompt: "Does this video complete the task ‘⟨instruction⟩’?" producing a binary output IF$(v)\in\{0,1\}$. The metric is averaged over $N$ videos for each test slice:

$IF = \frac{1}{N} \sum_{i=1}^N IF(v_i)$

• Physics Alignment (PA): Each video receives two scores: one from VideoCon-Physics (identifying physical implausibility) and another from Qwen-VL 2.5 (prompted as "Is the robot's motion physically plausible?"), yielding soft scores in $[0,1]$. The mean defines PA$(v)$ for each video, aggregated as:

$PA = \frac{1}{N}\sum_{i=1}^N PA(v_i)$

• Combined Score: Represents overall video utility:

$\text{Score} = \frac{IF + PA}{2}$

These scalar evaluations serve as proxies for the predictive utility of video models in robot learning.

5. Experimental Setup and Quantitative Results

DreamGen Bench experiments assess four generative video models (WAN 2.1, Cosmos, Cog VideoX, HunyuanVideo), each in zero-shot and fine-tuned ("–sft") variants. For simulation (RoboCasa; 1,200 training demos, ∼6,000 evaluation frames), and real-world (GR1 Humanoid; 2,884 trajectories, ∼8,000 frames), only the split between zero-shot and fine-tuned is considered significant—no additional ablations are performed.

Key quantitative results for fine-tuned ("–sft") variants:

Model	IF (%)	PA (%)	Combined Score (%)
WAN 2.1-sft	77.1	64.9	71.0
Cosmos-sft	79.2	59.4	69.3

Zero-shot models score near 0% for both IF and PA. When these synthetic sequences (∼7,000 per model) are used to train RoboCasa policies, the resulting average success rate $S$ shows a near-linear relationship with DreamGen Bench Score, with a Pearson correlation $r\approx0.92$ ($(v)\in\{0,1\}$0; $(v)\in\{0,1\}$1):

$(v)\in\{0,1\}$2

This confirms that DreamGen Bench scores are reliable predictors of policy learning efficacy for this task distribution.

6. Implications and Limitations

DreamGen Bench functions as a low-cost, hardware-independent diagnostic for evaluating and comparing video world models in the robot learning pipeline. The strong observed correlation between DreamGen Bench metrics and downstream policy success suggests that continued refinement of instruction-following and physics-grounding capabilities in generative video models will translate proportionally into more scalable, robust robot learning frameworks.

Limitations of DreamGen Bench include:

• Dependence on lightweight VLM evaluators, which are susceptible to hallucinations.
• Requirement for manual initial-frame collection in novel environments.
• Restricted focus on short-horizon, single-agent manipulation tasks; it does not currently encompass multi-agent interaction or highly deformable object scenarios.

Addressing these constraints, such as by automating scene initialization or developing more robust evaluators, are important avenues for future work and may further extend generalization axes for robot learning paradigms (Jang et al., 19 May 2025).

7. Prospective Directions

The DreamGen Bench framework lays the groundwork for broader research into scalable robot learning using synthetic data. Promising future directions include:

• Automating the generation of initial scene frames to further streamline the evaluation process.
• Refining physics evaluators for higher reliability and reduced hallucination rates.
• Expanding benchmark coverage to longer-horizon, multi-agent, and highly deformable object tasks.
• Exploring integration of the DreamGen Bench protocol with more diverse robot embodiments and less constrained environmental complexity.

Such developments may enable more generalizable and scalable robot learning, leveraging improvements in world model fidelity as measured by DreamGen Bench metrics.