Automated, reliable task-success evaluation for robotic manipulation

Develop a fully automated and reliable methodology to assess task success in real-world robotic manipulation videos, robust to diverse failure modes and ambiguous semantics, eliminating reliance on proxy metrics and partial human validation in evaluation suites such as the Embodied World Model Benchmark (EWMBench).

Background

The paper introduces EWMBench, a benchmark for evaluating video-based world models in robotic manipulation, covering visual fidelity, motion consistency, and semantic grounding. Despite its breadth, the current evaluation still depends on proxy metrics and partial human judgments.

The authors explicitly note that creating a fully automated and reliable assessment of task success—especially under diverse failure modes and ambiguous semantics—remains unresolved. Addressing this would enable scalable, consistent, and human-aligned evaluation of manipulation outcomes without manual oversight.

References

Fully automated and reliable assessment of task success—particularly under diverse failure modes and ambiguous semantics—remains an open challenge.

— Genie Envisioner: A Unified World Foundation Platform for Robotic Manipulation  (2508.05635 - Liao et al., 7 Aug 2025) in Limitations, bullet "Evaluation Methodology"

The open problem is calibration under shift: hidden-state probes lose accuracy under clutter, lighting, novel objects and reworded instructions, and SAFECAST recovers part of it by training the probe on contrast-set perturbations \citep{safecast2026}, while Foresight makes the threshold itself adaptive \citep{foresight2026}.

— FailBench: How Reliable are VLMs at Judging Robot Task Success?  (2609.03611 - Navasardyan et al., 3 Sep 2026) in Section 2, Related Work

These results suggest that robot failure detection remains an open problem even for current general-purpose VLMs.

— FailBench: How Reliable are VLMs at Judging Robot Task Success?  (2609.03611 - Navasardyan et al., 3 Sep 2026) in Section 7, Conclusion

Four questions remain. First, for criteria defined by predicates such as contact or grasp rather than by quantities, what to supervise is undetermined.

— Supervise What Decides Success: Criterion-Aligned Auxiliary Losses for Latent World-Model Planning  (2610.01224 - Hara et al., 1 Oct 2026) in Section 5, paragraph “Future work” (Conclusion and limitations)

Evaluation of model-generated programs under the independent protocol, replication across training seeds, and matched tests of learning-rate and epoch effects remain future work.

— Instruction-Conditioned Motion Control for Robotic Arms: Language-to-Code Decomposition and Reinforcement Learning  (2610.19617 - Huang et al., 14 Oct 2026) in Section IV-B, Evaluation protocol and interpretation of logged rewards, p. 4; Section I, p. 1