Embodied Arena: Unified AI Evaluation

• Embodied Arena is a comprehensive evaluation platform for embodied AI, featuring a hierarchical capability taxonomy and standardized benchmark integration.
• It integrates 22 prominent benchmarks, automated LLM-driven data generation, and rigorous reproducibility tools to enable transparent model comparisons.
• Real-time leaderboards deliver actionable insights by tracking performance across perception, reasoning, and task execution dimensions.

Embodied Arena defines a unified, systematically organized, and continuously evolving evaluation platform for embodied AI. Conceived in direct response to gaps in capability definition, standardized evaluation, and the scalability of embodied data collection, Embodied Arena is engineered to furnish the field with a comprehensive taxonomy of embodied capabilities, an extensible multi-domain infrastructure integrating numerous leading benchmarks, an LLM-driven automated data generation system, and real-time leaderboards for transparent, research-oriented model comparison. These features collectively address bottlenecks in research reproducibility, objective comparison, and the alignment of embodied AI methods with both foundational skill taxonomies and practical challenges (Ni et al., 18 Sep 2025).

1. Systematic Capability Taxonomy

Embodied Arena constructs its evaluation pipeline atop a three-level hierarchical capability taxonomy. This taxonomy organizes embodied competencies as:

• Perception Level: Foundational capabilities including
  • Object Perception: subdivided into object type, property, state, and count.
  • Spatial Perception: spatial relationship, distance, localization, size.
  • Temporal Perception: description and order.
  • Embodied Knowledge: world knowledge and affordance prediction.
• Reasoning Level: Higher-order reasoning decomposed into
  • Object, spatial, temporal, knowledge, and task reasoning.
• Task Execution Level: Goal-driven action skills
  • Embodied navigation (object, location, and instruction-based).
  • Embodied task planning (basic, visual/spatial/temporal/knowledge reference planning).

This results in seven core capabilities and 25 fine-grained dimensions. Existing benchmarks are mapped onto this taxonomy to establish a unified research language; for example, ScanQA is mapped to spatial localization and temporal order, Where2Place to affordance prediction. This design enables granular, cross-benchmark, and longitudinal evaluation within a shared capability space (Ni et al., 18 Sep 2025).

2. Standardized Evaluation Infrastructure

Addressing fragmentation and the lack of comparability across tasks, Embodied Arena delivers a common technical infrastructure for evaluation:

• Unified Input/Output and Execution: The platform supports integration of 22 prominent benchmarks over three domains (2D/3D Embodied Q&A, Navigation, Task Planning), providing a standardized conveyance for task data, response formatting, and environment orchestration.
• Metrics and Scoring Rules: Task performance is measured with standard and specialized metrics:
  • For embodied Q&A: exact match, fuzzy match, CIDEr, BLEU, or LLM-based scoring.
  • For navigation: success rate (SR), Success weighted by Path Length (SPL).
  • For task planning: proportion of tasks completed.
  • Formulas, e.g., for a benchmark $B^n$ with $M$ dimensions: $S^n_{(m)} = (c^n_{(m)} / k^n_{(m)}) \times 100$, and aggregate scores $B_{total} = (1/N) \cdot \sum_{n=1}^N A_{total}^n$.
• Reproducibility Tools: A professional experiment management system logs configurations and performance metrics ensuring transparent, reproducible experiments.
• Flexible Model Integration: The framework allows connection of both closed-source (API), open-source (parameter), and custom-architecture models from over twenty institutes and thirty advanced models (Ni et al., 18 Sep 2025).

3. LLM-driven Automated Data Generation

A critical bottleneck in embodied AI benchmarking is the scalable, diverse, and continuously refreshed generation of scenario and task data. Embodied Arena implements a two-stage LLM- and VLM-guided pipeline:

• Scenario Generation: Floor planning, functional zoning, and layout planning are achieved via hierarchical reasoning, with LLMs ensuring logical spatial connectivity and natural scene semantics through domain randomization and object attribute diversification.
• Capability-Oriented Pipeline: For each capability, the pipeline uses privileged simulator access to extract scene ground-truth (types, locations, attributes) and generates visual-instruction-answer tuples via templates. Curricula along scene, language, and task complexity ("difficulty ladders") are built in, enabling stepwise diagnosis of agent skill acquisition or failure.
• Adaptive Evolution: Regular model performance analysis enables the generation engine to inject adversarial or weakly mastered scenarios, countering overfitting and ensuring the evaluation landscape evolves with research advances.
• Quality Filtering: Combined automated sampling and human curation ensures data diversity and high validity (Ni et al., 18 Sep 2025).

4. Leaderboards, Evaluation Results, and Findings

Real-time leaderboards are integral to Embodied Arena’s dissemination and research guidance objectives:

• Specialized Leaderboards: Individual boards track Embodied Q&A, Navigation, and Task Planning performance.
• Dual View System:
  • Benchmark View: Model-wise ranking on tasks.
  • Capability View: Aggregated scores per capability dimension, clarifying whether models, for example, excel in "spatial reasoning" but lag in "temporal order."
• Nine Key Findings: Based on systematic leaderboard analysis, the platform surfaced the following:
  • Specialized embodied models outperform generic ones at similar scale, but large-scale general models remain competitive due to pretraining.
  • Benchmark-specific overfitting is common, underlining the necessity of unified, multi-dimension evaluation.
  • Basic perception and spatial skills are strongly predictive of downstream task success.
  • Correlation emerges between core capability scores and agentic downstream task performance.
  • Scaling laws diverge from language-only domains; adding more data can be counterproductive without sufficient diversity.
  • Reinforcement finetuning substantially advances reasoning, but its generalization characteristics require further paper.
  • Mixing 2D and 3D representations is technically vital for mastering complex spatial tasks.
  • Architectural choices (end-to-end or modular pipelines) affect long-horizon navigation robustness.
  • The ability to refer and ground (“point”) meaningfully improves not only referential tasks but overall embodied intelligence (Ni et al., 18 Sep 2025).

5. Impact on Embodied AI Research

Embodied Arena advances the field by:

• Unifying Evaluation and Research Veins: Harmonized capability taxonomies and data pipelines reconcile disparate benchmarks, providing a research road map and transparent objective comparison.
• Scalability and Continuous Dataset Evolution: The LLM-driven generation pipeline ensures no static “test set saturation,” and instead continuously challenges models, preventing memorization and encouraging broader skill development.
• Enabling Research Guidance: Leaderboard insights support diagnostic analysis, helping researchers target unsolved challenges (e.g., spatial perception) and refine training paradigms.
• Community Integration: Transparency, professional experiment tracking, and cross-benchmark comparability promote collaborative engagement, accelerate benchmarking cycles, and foster the rapid evolution of embodied agent capabilities (Ni et al., 18 Sep 2025).

6. Context and Broader Significance

Embodied Arena emerges within a landscape marked by the proliferation of isolated benchmarks, ad-hoc evaluation, and a lack of shared research direction in embodied AI. Its synthesis of systematic skill taxonomy, automation in data generation, and capability-oriented evaluation—accompanied by rigorous scoring methodology—marks a substantive infrastructural advance. A plausible implication is that such platforms constitute a necessary precondition for robust progress in embodied intelligence, particularly as the field transitions toward real-world tasks requiring reasoning, perception, action, and adaptability over extended time horizons and environments—a core necessity for eventual artificial general intelligence (Ni et al., 18 Sep 2025).