---
title: 'EmbodiedEvalKit: Unified Evaluation for Embodied AI'
url: https://www.emergentmind.com/topics/embodiedevalkit
type: topic
---

# EmbodiedEvalKit: Unified Evaluation for Embodied AI

Searching arXiv for "EmbodiedEvalKit" and the cited papers to ground the article in the current literature.
arxiv_search({"query":"EmbodiedEvalKit", "max_results": 10, "sort_by":"submittedDate"})
arxiv_search({"query":"2602.05249", "max_results": 5, "sort_by":"relevance"})
arxiv_search({"query":"\"Automatic Cognitive Task Generation for In-Situ Evaluation of Embodied Agents\"", "max_results": 5, "sort_by":"relevance"})
arxiv_search({"query":"\"Embodied Arena\" embodied evaluation platform", "max_results": 5, "sort_by":"relevance"})
EmbodiedEvalKit is a recurrent label in recent embodied-AI literature for infrastructures that evaluate agents, models, or embodied experience through standardized task definitions, agent interfaces, and metric pipelines. Current usage suggests that the term does not denote a single canonical package; rather, it names several distinct but overlapping systems, including the TEA in-situ evaluation framework for unseen 3D environments, the EmbodiedCity simulator interface for urban embodied tasks, the MFE-ETP automatic benchmark toolkit for multi-modal foundation models, the OmniEAR reasoning harness for tool use and coordination, the Embodied Arena unified leaderboard platform, and the evaluation framework released with Embodied-R1.5 [2602.05249, 2410.09604, 2407.05047, 2508.05614, 2509.15273, 2606.11324]. Across these usages, EmbodiedEvalKit denotes a reproducible layer that binds scene or scenario representation, task formalization, model inference, and metric computation into a common evaluation workflow.

## 1. Multiple uses of the term in the literature

The name “EmbodiedEvalKit” appears in several recent papers with different scopes. In some works it refers to an automatic task-generation system operating inside an unmodified 3D scene; in others it denotes a benchmark SDK, a cross-benchmark orchestration layer, or an evaluation harness for embodied VLMs. This plurality is itself informative: it indicates that embodied-AI evaluation is converging on a toolkit-centric paradigm, but has not yet converged on a single standardized implementation [2602.05249, 2410.09604, 2407.05047, 2508.05614, 2509.15273, 2606.11324].

| Paper | Domain | Defining focus |
|---|---|---|
| TEA [2602.05249] | Unseen household 3D environments | Two-stage Interaction/Evolution task generation |
| EmbodiedCity [2410.09604] | Real-world city environment | Unreal Engine simulator, AirSim-based access, five benchmark tasks |
| MFE-ETP [2407.05047] | Household embodied task planning | Four-capability hierarchy and automatic evaluation platform |
| OmniEAR [2508.05614] | Text-based embodied reasoning | Tool usage, continuous properties, implicit collaboration |
| Embodied Arena [2509.15273] | Unified cross-benchmark evaluation | 22 benchmarks, taxonomy, leaderboards, evolving data |
| Embodied-R1.5 [2606.11324] | Embodied VLM benchmarking | Unified grounding/reasoning API and manipulation evaluation |

A common structural motif is the separation between data or scene representation, model or agent adapters, and metric computation. Another common motif is formal task definition: tasks are not treated merely as natural-language prompts, but as structured objects with explicit state, goal, and success conditions. This suggests a shift away from narrow benchmark scripts toward reusable evaluation substrates.

## 2. Graph-native task and environment representations

One of the most explicit formalizations appears in the TEA framework, where each task \(t\) is defined as a pair of labeled graphs, its Initial-State and Final-State:

$$
\text{Task} = (\text{Initial-State}, \text{Final-State})
$$

with

$$
\text{Initial-State} = (V, E, A(V,E)), \qquad
\text{Final-State} = (V, E, A(V,E)).
$$

Here, \(V\) is a set of vertices, \(E \subseteq V \times V\) is a set of edges, and \(A(V,E)\) assigns attributes such as object color, 3D bounding box, preconditions, and post-effects. The paper’s abstract “object classification” example instantiates this scheme by adding \(\underline{\text{label}(\text{object})}\) in the Final-State; task instances are then grounded to concrete entities such as “Apple A” [2602.05249].

TEA organizes generation as a two-stage loop. In the Interaction stage, an agent explores or executes previously generated tasks, collects RGB, depth, instance masks, and poses, and induces new tasks through task generators. In the Evolution stage, the framework applies Task Reuse and Task Recombination to existing graph-structured tasks. Reuse is triggered when \(\text{subgraph}(t^a) \subseteq \text{graph}(t^b)\), while recombination swaps matching node types across tasks. A diversity filter based on spectral clustering over multi-modal embeddings retains one representative per cluster, with similarity defined as

$$
S_{ij} = \frac{1}{M}\sum_{m=1}^{M}\cos\bigl(f_m(t_i), f_m(t_j)\bigr).
$$

The same paper provides concrete examples such as an “object in-view check,” where the graph includes an edge \((\text{agent} \xrightarrow{\text{sees}} \text{table\_Red})\), and “navigation by label,” where success is formalized as reaching 3D distance \(< 0.5\) m from the target within 10 steps [2602.05249].

A related but distinct graph formalism appears in OmniEAR. Its text-based simulator EAR-Sim represents the environment as \(G_t = (V_t, E_t, A_t)\), where \(V_t\) includes rooms, objects, and agents; \(A_t(v)\) stores continuous attributes such as weight, temperature, material composition, dimensions, and dynamic states; and \(E_t\) includes static containment relations and dynamic proximity. Tasks are defined as \(\mathcal{T} = (S_{\text{init}}, I, G_{\text{goal}}, \mathcal{A}_{\text{task}})\), with success determined by whether the final state after applying an action sequence satisfies all predicates in \(G_{\text{goal}}\) [2508.05614].

OmniEAR extends this representation to dynamic capability acquisition. Agents begin with a basic action set \(\mathcal{A}_{\text{basic}} = \{\text{MOVE}, \text{GRASP}, \text{OPEN}, \dots\}\); when an agent grasps a tool object, its available abilities are augmented by that tool’s `provides_abilities`, and they are removed upon release. Multi-agent collaboration is likewise grounded in physical constraints: when object weight exceeds a single agent’s capacity, the simulator triggers a collaboration protocol that produces a `CORP_GRAB → CORP_GOTO → CORP_PLACE` sequence without explicit instruction. This suggests a broader trend in EmbodiedEvalKit-style systems toward graph-native state, explicit affordances, and executable goal predicates rather than prompt-only evaluation [2508.05614].

## 3. Architectural patterns and system interfaces

At the simulator level, EmbodiedCity presents EmbodiedEvalKit as a benchmark platform built in Unreal Engine 5.3, covering a \(2.8\ \text{km} \times 2.4\ \text{km}\) commercial district in Beijing plus adjacent residential interiors. The environment hosts static geometry as fully textured 3D meshes, with approximately 200 buildings, approximately 100 streets, and more than 6,000 urban assets. Geometry is reconstructed from Baidu and Amap street-view imagery and GIS data; pedestrian and vehicle flows are driven by the Mirage Simulation System using discrete-time kinematics, and access is exposed through a Microsoft AirSim plugin extended with a Python proxy server using HTTP/JSON [2410.09604].

EmbodiedCity’s agent interface is unusually rich. Per time step it can return RGB images, depth maps, semantic segmentation, IMU, GPS, LiDAR, position, velocity, orientation, and vehicle wheel angle. Action spaces are platform-specific: drones receive position or velocity targets, attitude set-points, camera pan/tilt, and takeoff/land commands, whereas ground vehicles receive steering angle, throttle, brake force, gear shift, and camera controls. The Python SDK supports synchronous and asynchronous calls, batched requests, and live video-stream endpoints for up to eight agents in parallel [2410.09604].

MFE-ETP presents a more benchmark-centric software stack. Its EmbodiedEvalKit is organized into `data_loader`, `prompt_engine`, `model_adapters`, `evaluator`, and `metrics`, plus a CLI wrapper. `data_loader` normalizes raw JSON annotations and image frames into `Case(sample_id, images, prompt_text, gt_label)`, `prompt_engine` produces templated inputs, `model_adapters` wrap interfaces such as `GPT-4VAdapter`, `BLIP2Adapter`, and `MiniGPT4Adapter`, `evaluator` includes `QAEvaluator` and `HumanEvaluator`, and `metrics` emits per-case and aggregated JSON/CSV outputs. This is a classical end-to-end benchmark harness: dataset preparation, prompt realization, model invocation, scoring, and report export are all explicit modules [2407.05047].

Embodied Arena generalizes this pattern into a service-oriented architecture. Its components are a Model Adapter Layer with a uniform `predict(input: JSON) → output: JSON` interface, a Data Manager using a standard JSON schema, a Scenario Simulator & Orchestrator for interactive tasks in simulators such as Unity and Habitat, a Metric Engine, and a Leaderboard Builder. Internally, benchmarks are wrapped in a common “Task” class, and evaluation is expressed as a loop over benchmarks, cases, predictions, and metric computation. The framework is therefore oriented not only toward single-benchmark experiments but toward unified, leaderboard-scale comparison across heterogeneous embodied tasks [2509.15273].

The Embodied-R1.5 version of EmbodiedEvalKit pushes further toward standardized embodied VLM evaluation. It defines a four-layer stack: a Data Layer in which each benchmark is preprocessed into a single HuggingFace Parquet file with `prompt`, `gt`, and `meta` fields; an Inference Layer with pluggable backends including vLLM, HuggingFace Transformers, and OpenAI/Gemini APIs; a Parsing Layer that normalizes heterogeneous outputs into a `ModelOutput` dataclass containing text, boxes, points, and trajectories; and an Evaluation Layer implementing each benchmark’s official metric. The framework also includes an `envs/` module for optional simulator or real-robot wrapping [2606.11324].

## 4. Task families and metric systems

The task taxonomies covered by EmbodiedEvalKit-style systems span perception, reasoning, navigation, planning, grounding, trajectory prediction, and manipulation. EmbodiedCity defines five benchmark tasks: First-view Scene Understanding, Embodied Question Answering, Embodied Dialogue, Vision-and-Language Navigation, and Embodied Task Planning. MFE-ETP organizes evaluation into four capabilities—Object Understanding, Spatio-Temporal Perception, Task Understanding, and Embodied Reasoning—each decomposed into concrete sub-tasks such as Type Recognition, Property Recognition, Spatial Relations, Temporal Sequencing, Relevant Object Selection, Step Sequence, Goal Completion, and End-to-end Planning. Embodied Arena systematizes the space into three levels—Perception, Reasoning, and Task Execution—with seven core capabilities and 25 fine-grained dimensions, while Embodied-R1.5 groups benchmarks into Planning & Correction, Pointing & Affordance, Visual Trace, Spatial Cognition, and Manipulation in Simulation [2410.09604, 2407.05047, 2509.15273, 2606.11324].

These frameworks also differ in the granularity and semantics of their metrics. EmbodiedCity uses BLEU-1…4, ROUGE, METEOR, and CIDEr for scene understanding, the same textual metrics plus Sentence-BERT for QA and dialogue, and standard VLN metrics for navigation: Success Rate, Navigation Error, and SPL. Its path-planning discussion explicitly uses A* with \(f(n)=g(n)+h(n)\), and SPL is given as

$$
\mathrm{SPL}=\frac{1}{N}\sum_{i=1}^N S_i \frac{l_i^*}{\max(l_i,l_i^*)}.
$$

This combination reflects a mixed paradigm in which embodied evaluation includes both language generation quality and control performance [2410.09604].

TEA emphasizes diversity and scene coverage in addition to task success. It defines the Maximum Independent Subset under redundancy threshold \(\alpha\),

$$
\mathrm{MIS}(\alpha,\tau)=\arg\max_{\tau' \subseteq \tau}\left\{|\tau'| \mid \forall i \neq j,\ S_{ij}\le \alpha\right\},
$$

and the Maximum Independent Ratio,

$$
\mathrm{MIR}(\alpha,\tau)=|\mathrm{MIS}(\alpha,\tau)|/|\tau|.
$$

It also reports enclosing volume of object positions, per-axis spread, standard deviations, mean and standard deviation of instance volumes, and task-specific accuracy or mIoU. Navigation-specific metrics include Success Rate, Navigation Gain, Steps Taken, Target Neglect Rate, and Lack of 3D Awareness [2602.05249].

MFE-ETP adopts the familiar classification metrics Accuracy, Precision, Recall, and \(F_1\), then adds planning-oriented IoU, Path Efficiency, and Step Efficiency. Embodied Arena uses Exact Matching Accuracy for closed-vocabulary QA, fuzzy matching through CIDEr, BLEU, and ROUGE or LLM-based scorers for open responses, Success Rate and SPL for navigation, and Task Completion Rate for planning; it also defines benchmark-view and capability-view aggregation rules across 22 integrated benchmarks. Embodied-R1.5 further expands the metric repertoire with point-distance partial credit for 2D pointing tasks, trajectory RMSE, semantic-similarity scoring via a learned LLM reward model with BLEU fallback, and a binary format-check reward enforcing output templates such as `<answer>…</answer>` [2407.05047, 2509.15273, 2606.11324].

A plausible implication is that “EmbodiedEvalKit” increasingly denotes not merely a dataset wrapper, but a metric algebra across heterogeneous output spaces: text answers, boxes, points, trajectories, control traces, and full task executions.

## 5. Empirical findings enabled by these evaluation kits

The TEA framework reports that, across 10 unseen scenes, it automatically generated 87,876 tasks in two cycles, and that a 10% random subset reviewed by human annotators was judged 100% physically valid, with 90.8% deemed “helpful in daily life” and 94.4% requiring essential cognitive ability. On 848 sampled tasks, human participants scored \(> 0.80\) accuracy on all tasks, whereas state-of-the-art vision-language models ranged from 0.05 on the worst object localization results to 0.94 on the best relationship detection results. The reported basic perception gaps were approximately \(\Delta \approx 0.70\) in classification and \(\Delta \approx 0.44\) in localization. In navigation, GPT-4o achieved Success Rate \(=0.62\) and Navigation Gain \(=0.48\), while o1 reached Success Rate \(=0.38\) and Navigation Gain \(=0.12\), with humans near 1.0 [2602.05249].

EmbodiedCity evaluates Fuyu-8B, Qwen-VL, Claude 3 Haiku, GPT-4 Turbo, and GPT-4o on large urban splits: 12.2k scene-understanding cases, 50.4k QA questions, 12.6k dialogues, 1.3k VLN routes, and 10.6k planning goals. Claude 3 leads scene understanding with CIDEr approximately 29, GPT-4 Turbo is reported as close second, GPT-4 Turbo outperforms the next-best model on QA by approximately 100% in BLEU/SPL metrics, GPT-4o reaches approximately 77% short-route Success Rate and approximately 76% SPL in VLN with mean SR approximately 35%, and Claude 3 is best on planning with BLEU-4 approximately 4.4 and CIDEr approximately 21.8 [2410.09604].

OmniEAR provides a controlled study of embodied reasoning under constraint-driven planning. Reported single-agent success rates are 85–96% for Direct Command, 56–85% for Tool Use, 42–78% for Attribute Reasoning, and 54–71% for Compound Reasoning. Multi-agent success rates are 88–92% for Explicit Collaboration, 63–85% for Implicit Collaboration, and 28–49% for Compound Collaboration. Fine-tuning Qwen2.5-3B raises single-agent performance from 0.6% to 76.3% on Direct Command, but improves implicit collaboration only from 1.5% to 5.5%. An especially notable result is the “Full-Environment-Information Effect”: complete world-graph information improves Tool Use by up to \(+27.7\%\) and Direct Commands by \(+4\%\) to \(+40\%\), but degrades Implicit Collaboration by 5–23% [2508.05614].

MFE-ETP reports, at the level of its abstract, that several state-of-the-art MFMs significantly lag behind human-level performance on embodied task planning [2407.05047]. Embodied Arena extends this diagnosis to a unified leaderboard setting: it reports that object and spatial perception are the weakest links, with average scores approximately 28–38%, and that embodied capabilities correlate strongly, with \(\rho \approx 0.80, p < 0.0001\), with downstream navigation and planning performance. It also reports that purely voxel or point-cloud methods underperform by 15–25% relative to approaches blending native 3D representations with 2D feature alignment [2509.15273].

The Embodied-R1.5 paper uses its EmbodiedEvalKit to evaluate a single 8B EFM across 24 embodied VLM benchmarks and 4 manipulation suites, reporting state of the art on 16 out of 24 embodied VLM benchmarks and strong zero-shot real-robot results in instruction following, affordance grounding, articulated object manipulation, and long-horizon complex tasks [2606.11324]. In this context, EmbodiedEvalKit functions as the experimental substrate that makes such claims comparable across planning, pointing, spatial cognition, and manipulation.

## 6. Limitations, standardization pressures, and adjacent uses

The literature also makes clear that EmbodiedEvalKit-style infrastructures inherit the limitations of their generators, simulators, and task abstractions. In TEA, reliance on a VLM for semantic tasks introduces hallucination risk and task redundancy; MIR filtering mitigates but does not eliminate redundancy. The framework is evaluated with a simulated Unreal Engine agent rather than a real robot, so physical transfer gaps remain untested, and some high-level tasks such as long-horizon planning are currently out of scope [2602.05249].

Broader unified platforms surface additional tensions. Embodied Arena argues that models overfit to narrow benchmarks and that isolated tasks fail to reveal comprehensive embodied ability, motivating evolving data-generation methods and monthly updated leaderboards [2509.15273]. OmniEAR identifies architectural bottlenecks in attention-based models, arguing that they struggle to filter task-relevant physical constraints from noisy graphs and to maintain working memory for object states, capability sets, and multi-agent plans. The paper’s own implications for EmbodiedEvalKit design include selective constraint filtering modules, persistent state memory or an external graph store, hybrid symbolic-neural architectures for explicit physics reasoning, and modular evaluation kits that expose both abstract planning and continuous constraint reasoning channels [2508.05614].

The term also extends beyond embodied-agent benchmarking into VR embodiment measurement. The Virtual Embodiment Questionnaire defines three factors—ownership, agency, and change in the perceived body schema—validated through confirmatory factor analysis and operationalized as a 12-statement questionnaire on a seven-point Likert scale. The paper concludes that the VEQ’s sensitivity to first-person immersion and sensorimotor disruption makes it a versatile tool for EmbodiedEvalKits across VR therapy, training, social VR, and entertainment [1911.10176]. This is not an embodied-agent benchmark in the same sense as TEA or Embodied Arena, but it shows that the “EmbodiedEvalKit” label can also encompass psychometric evaluation of embodied user experience.

Taken together, the literature presents EmbodiedEvalKit not as a settled product category but as an evolving family of evaluation infrastructures. Some instances emphasize in-situ task generation in unseen scenes; others emphasize simulator access, unified benchmark wrapping, capability taxonomies, leaderboard governance, or multi-format metric engines. The common denominator is methodological: embodied intelligence is being evaluated through formalized tasks, explicit interfaces, and increasingly automated data generation rather than through ad hoc benchmark scripts alone.

Source: https://www.emergentmind.com/topics/embodiedevalkit