Automated, reliable task-success evaluation for robotic manipulation

Develop a fully automated and reliable methodology to assess task success in real-world robotic manipulation videos, robust to diverse failure modes and ambiguous semantics, eliminating reliance on proxy metrics and partial human validation in evaluation suites such as the Embodied World Model Benchmark (EWMBench).

Background

The paper introduces EWMBench, a benchmark for evaluating video-based world models in robotic manipulation, covering visual fidelity, motion consistency, and semantic grounding. Despite its breadth, the current evaluation still depends on proxy metrics and partial human judgments.

The authors explicitly note that creating a fully automated and reliable assessment of task success—especially under diverse failure modes and ambiguous semantics—remains unresolved. Addressing this would enable scalable, consistent, and human-aligned evaluation of manipulation outcomes without manual oversight.