Evaluation of embodied intelligence

Develop rigorous, generalizable methodologies and metrics to evaluate embodied intelligent systems, determining how to assess progress and performance across changing tasks and environments while accounting for learning and adaptation.

Background

The paper argues that embodied agents face unique challenges not adequately addressed by conventional machine learning, including non-stationarity, safety, and adaptation. Standard benchmarks and validation approaches often do not capture these complexities. The authors highlight that, unlike passive perception tasks with static datasets, robots act in and alter their environments, complicating evaluation.

This motivates the need for new evaluation methodologies that can measure whether learning agents are making progress and can generalize safely to novel tasks and environments.

References

Finally, an open question in embodied intelligence is how these systems should be evaluated. At heart, how do we as a community know if progress is being made?

— From Machine Learning to Robotics: Challenges and Opportunities for Embodied Intelligence  (2110.15245 - Roy et al., 2021) in Section 1 (Introduction)

The contribution is a traceable engineering design study and evaluation framework for a prototype whose integrated field performance remains to be established.

— A3P5 NEMESIS Integrated Rover Design for Environmental Reconnaissance and Robotic Sampling with Reproducible Mobility Analysis and an External Data Machine Learning Calibration Benchmark  (2609.18245 - Sultan et al., 16 Sep 2026) in Abstract; Section 1, Introduction

These properties may offer computational advantages, but such benefits remain conditional. In vitro networks vary across cultures and over time, and potential advantages such as resilience to perturbations or energy-efficient event-driven processing must be evaluated at the system level; reliable resilience remains to be demonstrated in BHI systems, and any energy advantage must include the costs of culture maintenance, recording, stimulation, and digital control.

— Biological-Hybrid Intelligence: A Conceptual Framework for Distributed Biological--Artificial Computation  (2608.18748 - Barros et al., 19 Aug 2026) in Section 2.1, Living Intelligence

To avoid this, \algname uses standard perceptual metrics common in state-of-the-art video models, leaving improved evaluation metrics as an open area of research.

— CLAP: Cross-Embodiment Video World Models are Zero-Shot Physical Simulators  (2608.27406 - Liu et al., 27 Aug 2026) in Appendix, Section 5, Nuanced Summary, Detailed Q&A, answer to qa:perceptual-vs-object-metrics

What are the minimum set of tasks that an agent must successfully perform right after unboxing for it to be considered reasonably intelligent?

— Deploying Foundation Models for Embodied Navigation  (2609.25666 - Dorbala et al., 22 Sep 2026) in Section “Conclusion {content} Future Directions,” subsection “On Day-One Unboxing”

Public task assets leave unseen-environment generalization open, while source-specific helpers vary the balance between skill selection and low-level control.

— Are Frontier VLM Agents Ready to Be Robot Generalists? An Empirical Study with the Embodied Agent Arena  (2610.00854 - Huang et al., 1 Oct 2026) in Conclusion, Limitations paragraph

Whether policies that carry the robot interface implicitly behave the same way is open, and the axis is well defined for them too.

— The Robot Is Not Its Description: GaugeBench for Representation Robustness in Morphology-Aware Policies  (2610.07597 - Malladi et al., 6 Oct 2026) in Section Discussion and Limitations