EWMBench: Embodied World Model Benchmark

• EWMBench is a comprehensive evaluation suite that measures visual scene consistency, motion correctness, and semantic alignment for embodied AI models.
• The benchmark integrates a curated dataset of robotic manipulation scenarios, standardizing inputs and outputs to ensure physical plausibility in video-based tasks.
• It provides fine-grained, open-source tools and multidimensional metrics that underpin reliable assessment of generative models in real-world robotic contexts.

The Embodied World Model Benchmark (EWMBench) is a comprehensive evaluation suite developed to rigorously assess the capability of embodied world models, with a focus on video-based generative models for robotics and physically grounded tasks. EWMBench provides quantitative and qualitative metrics centered on visual scene consistency, motion correctness, and semantic alignment—core criteria that distinguish high-quality embodied world modeling from conventional video generation. Unlike general video evaluation standards, EWMBench targets the unique demands of embodied AI, where physical plausibility and actionable behavior are as critical as perceptual fidelity. The benchmark is complemented by a curated dataset of diverse robotic manipulation scenarios and is supported by a set of fine-grained, open-source tools for standardized assessment (Yue et al., 14 May 2025, Liao et al., 7 Aug 2025).

1. Benchmark Objectives and Conceptual Foundations

EWMBench is designed for the evaluation of embodied world models (EWMs): generative models that, conditioned on task descriptions (e.g., natural language commands) and world initialization (visual and action context), synthesize video sequences depicting physically grounded agent-environment interactions. The motivation for EWMBench stems from the inadequacy of standard generative metrics (such as FID, IS) in embodied settings, where requirements extend to:

• Visual stability of scene elements across temporal spans,
• Correctness of agent motion with respect to task-defined actions,
• Semantic alignment between executed behaviors and intended outcomes,
• Robustness to real-world physics and the dynamics of manipulation,
• Generalizability across diverse, real-world tasks and domains.

Benchmarks such as VBench or general diffusion model metrics do not sufficiently capture scene stability and physical feasibility, nor do they account for the logical concordance between natural language instructions and generated visual behaviors.

2. Dataset Structure and World Initialization Protocol

The EWMBench dataset is curated from Agibot-World, reflecting both household and industrial robotic manipulation contexts. Each sample comprises:

• Initial images: Up to four images depicting the static scene configuration, agent embodiment (e.g., robotic arm type), and object layout.
• Natural language task instructions: Text specifying the intended high-level manipulation (e.g., "Take out toast from the toaster").
• Action trajectory (optional): A ground-truth reference trajectory, encoded as a sequence of 6D end-effector poses and converted into a voxelized representation for analysis.

The world initialization process standardizes the state for evaluation by integrating these modalities:

$$\mathcal{V} = v_{\text{norm}}\left(g_{\text{world}}(f_{\text{proc}}(\mathcal{I}, \mathcal{L}, \mathcal{T}))\right)$$

where $\mathcal{I}$ denotes image input, $\mathcal{L}$ language instruction, and $\mathcal{T}$ trajectory. Pre-processing normalizes input resolution (640×480) and resamples video outputs to 30 frames per second, ensuring methodological consistency (Yue et al., 14 May 2025).

3. Multi-Dimensional Evaluation Metrics

EWMBench employs a multidimensional toolkit, moving beyond global perceptual measures:

A. Visual Scene Consistency

This metric quantifies the preservation of background, objects, and agent configuration throughout the generated video. Patch-wise embeddings are extracted per frame using a fine-tuned DINOv2 encoder. The metric is the cosine similarity (CS) between corresponding patches in initial and subsequent frames:

$$\text{CS}_{i,j} = \cos\left(\text{DINOv2}(\text{patch}_i^{\text{ref}}), \text{DINOv2}(\text{patch}_j^{t})\right)$$

High aggregate similarity indicates stable composition and reduces hallucination of unintended scene changes (Liao et al., 7 Aug 2025).

B. Motion Correctness

Analysis focuses on the robot end-effector trajectory using three complementary metrics:

Metric	Mathematical Definition	Captured Attribute
HSD	$$\text{HSD}_{\text{score}} = 1/d_{\text{symH}}(G, P)$$	Max spatial deviation (Hausdorff)
NDTW	$\text{NDTW}_{\text{score}} = 1/\text{NDTW}(G, P)$	Sequence and timing alignment
DYN	$$\text{DYN}_{\text{score}} = \alpha \cdot VR \cdot 1/W(v) + \beta \cdot AR \cdot 1/W(a)$$	Dynamics (velocity, acceleration)

• $G, P$ are ground-truth and generated trajectories.
• $W(\cdot)$ is Wasserstein distance between velocity/acceleration series.
• $\alpha, \beta$ are empirically set scaling constants (typically 0.007, 0.003) (Yue et al., 14 May 2025, Liao et al., 7 Aug 2025).

C. Semantic Alignment and Diversity

Semantic correctness is measured at global and step levels:

• Global: A vision–LLM (e.g., Qwen2.5-VL-7B-Instruct) captions the entire video; BLEU scores are computed between caption and instruction.
• Stepwise: Key-action steps are extracted (via step-level language prompts and CLIP embedding) and compared for alignment with ground-truth sub-actions.

Diversity is quantified as $1 - \text{CLIP}(\text{feat}_i, \text{feat}_j)$ over video pairs sharing the same instruction, rewarding models that generalize to multiple valid strategies.

4. Benchmarking Protocol and Toolchain

EWMBench toolchain enforces standardized evaluation:

• Detection and Tracking: End-effector positions are localized via fine-tuned YOLO-World, with temporal smoothing provided by BoT-SORT trackers.
• Input/Output Pipeline: All models evaluated must preprocess inputs to defined standards (image size, frame rate) and return temporally synchronized sequences.
• Prompt Suite: MLLM-based models are evaluated with a unified prompt library for task-instruction understanding and logic evaluation, including error taxonomy discovery (hallucinated steps, object disappearances, etc.).
• Open-source Resources: All datasets, scripts, and documentation are publicly available [https://github.com/AgibotTech/EWMBench], facilitating reproducibility and inter-model comparison.

5. Sample Diversity and Task Coverage

EWMBench targets a broad operational envelope by incorporating tasks such as:

• Object manipulation (grasp, move, pour, fetch, insert)
• Sequential assembly/disassembly
• Human-in-the-loop and collaborative manipulation
• Repetitive and long-horizon tasks

Each task is decomposed into 4–10 atomic actions, providing granular control for evaluation and supporting stepwise error analysis (Yue et al., 14 May 2025, Liao et al., 7 Aug 2025). The candidate pool is curated via a greedy algorithm over pose trajectories in voxel space, ensuring action and environment diversity.

6. Empirical Insights and Observed Limitations

Empirical evaluation across general-purpose and domain-adapted video world models—such as EnerVerse, LTX_FT, and Genie Envisioner—reveals:

• Domain adaptation markedly improves motion and semantics scores, as adapted models learn to exploit embodied task logic and physical constraints.
• Fine-grained breakdown exposes distinct error modes: models may display visually perfect scenes but produce incoherent or physically implausible motion sequences.
• Common limitations include the occurrence of “empty grasping,” abrupt scene transitions, or generation of anthropomorphic hands in place of robot arms, indicating the importance of domain alignment and multimodal supervision.

A limitation of the current EWMBench configuration is its focus on fixed viewpoints and end-effector trajectories; future iterations propose richer state information (full-arm kinematics) and dynamic camera perspectives.

7. Extending and Applying EWMBench

EWMBench has become integral to leading platforms for embodied intelligence benchmarking (e.g., Genie Envisioner (Liao et al., 7 Aug 2025)). It provides a rigorous foundation for evaluating:

• Policy learning and closed-loop control directly from synthetic video data,
• Generalization to unseen tasks, robots, and environments,
• Policy and perception robustness in deployment-critical scenarios,
• The impact of architectural variants and domain adaptation methods.

EWMBench is explicitly constructed to promote reproducible research, accelerate innovation in physically grounded video modeling, and provide actionable diagnostics for long-horizon, complex manipulation benchmarks.

---

EWMBench establishes itself as a necessary evaluation standard for embodied world models, targeting the intersection of visual fidelity, physical feasibility, and semantic precision. Its multidimensional metric suite and real-world task diversity fill critical gaps in current model assessment, directly informing advances in embodied AI and foundational robotics research (Yue et al., 14 May 2025, Liao et al., 7 Aug 2025).