ESI-Bench: Towards Embodied Spatial Intelligence that Closes the Perception-Action Loop

Abstract: Spatial intelligence unfolds through a perception-action loop: agents act to acquire observations, and reason about how observations vary as a function of action. Rather than passively processing what is seen, they actively uncover what is unseen - occluded structure, dynamics, containment, and functionality that cannot be resolved from passive sensing alone. We move beyond prior formulations of spatial intelligence that assume oracle observations by recasting the observer as an actor. We introduce ESI-BENCH, a comprehensive benchmark for embodied spatial intelligence spanning 10 task categories and 29 subcategories built on OmniGibson, grounded in Spelke's core knowledge systems. Agents must decide what abilities to deploy - perception, locomotion, and manipulation - and how to sequence them to actively accumulate task-relevant evidence. We conduct extensive experiments on state-of-the-art MLLMs and find that active exploration substantially outperforms passive counterparts, with agents spontaneously discovering emergent spatial strategies without explicit instructions, while random multi-view often adds noise rather than signal despite consuming far more images. Most failures stem not from weak perception but from action blindness: poor action choices lead to poor observations, which in turn drive cascading errors. While explicit 3D grounding stabilizes reasoning on depth-sensitive tasks, imperfect 3D representation proves more harmful than 2D baselines by distorting spatial relations. Human studies further reveal that unlike humans who seek falsifying viewpoints and revise beliefs under contradiction, models commit prematurely with high confidence regardless of evidence quality, exposing a metacognitive gap that neither better perception nor more embodied interaction alone can close.

• The paper introduces a diagnostic benchmark that evaluates embodied agents by requiring active evidence gathering to close the perception-action loop.
• The paper demonstrates significant performance gains with active exploration and quantifies the impact of accurate versus flawed 3D scene representations.
• The paper reveals a marked metacognitive gap where models exhibit overconfidence compared to humans, underscoring the need for enhanced epistemic reasoning.

ESI-Bench: Diagnostic Benchmarking of Embodied Spatial Intelligence

Motivation and Background

Spatial intelligence in artificial agents fundamentally involves closing the perception-action loop: the agent must deliberately select and execute actions—locomotion, viewpoint control, manipulation—to obtain informative observations about the environment in order to resolve spatial uncertainty. Unlike conventional benchmarks that provide agents with fixed or exhaustive observations, ESI-Bench explicitly tests the agent’s ability to actively acquire task-relevant information. The benchmark is rigorously grounded in Spelke’s core knowledge systems: object representation, layout/geometry, number representation, and agents' goal-directed actions, operationalized over 10 task categories and 29 subcategories.

Figure 1: ESI-Bench is a comprehensive benchmark for embodied spatial intelligence, spanning 10 task categories and 29 subcategories organized around Spelke’s four core knowledge systems.

This approach targets shortcomings evident in prior spatial benchmarks, which often confound spatial reasoning competence with passive perception, relying on oracle observations, and failing to isolate the agent’s action selection policy as a critical failure point. ESI-Bench aims to disentangle and diagnose errors in perception, action selection, and epistemic calibration within embodied environments simulated using OmniGibson/BEHAVIOR-1K.

Benchmark Structure and Task Taxonomy

The ESI-Bench benchmark comprises 3,081 meticulously constructed tasks, each specified as $(\mathcal{S}, p_0, q, y^*)$, where $\mathcal{S}$ defines the 3D scene, $p_0$ the initial pose, $q$ a natural language question, and $y^*$ the answer. The environment supports a high-level action space including navigation, egocentric viewpoint shift, and manipulation primitives (e.g., pick, place, fill, pour), reflecting realistic autonomy with a 30-step action budget per episode.

Each task category demands different spatial faculties:

• Physical Structure & Dynamics: Revealing occluded contents, predicting stacked item stability, inferring containment under manipulation.
• Perceptual Grounding: Reasoning through partial occlusions, transparency, or reflection, where correct answers are ambiguous unless proper viewpoints are selected.
• Enumerative Perception: Counting or segmenting objects despite occlusion/enclosure, requiring active exploration for disambiguation.
• Cognitive Mapping/Metric Comparison: Planning navigation, comparing sizes/distances where perspective induces ambiguity.
• Temporal Understanding/Action Sequencing: Inferring scene changes or correct orderings based on agent-environment interactions.

Explicit quality control integrates multi-step automated verification (physics-based and semantic validation) and human review for answerability, correctness, and non-triviality. Human verification pass rates exceed 85% even for most challenging tasks.

Figure 2: A dataset example including question, answer, and trajectory, from the Rigid Containment sub-category.

Figure 3: ESI-Bench task categories and definitions, partitioned by core spatial faculties.

Experimental Paradigms and Evaluation

ESI-Bench evaluation encompasses four execution regimes:

1. Passive Single-View: The agent receives only the initial egocentric image.
2. Passive Multi-View: The agent is given 30 randomly sampled views, offering maximal but unstructured scene coverage, with no action selection.
3. Active Exploration: The agent must sequentially choose actions to gather evidence up to the step budget.
4. Ground-Truth Oracle Trajectory: The agent receives the optimal action-observation sequence, isolating deficits in visual recognition from those in action planning.

   Figure 4: Average number of active exploration steps to reach a correct answer for GPT-5 (solid) and Gemini 3.1 (outlined), grouped by Spelke’s four core knowledge systems.

This design isolates the effect of active information-seeking, enables quantification of action versus perception failures, and allows rigorous ablations (e.g., the benefit of 3D representations).

Core Findings and Numerical Results

Active Evidence Acquisition

The benchmarks reveal significant, category-consistent gains from active over passive observation. For key tasks such as View Hallucination and Rigid Containment, active agents see performance improvements from 39.9% to 68.1% and 47.5% to 67.5% (Gemini 3.1), respectively. In contrast, passive multi-view coverage can degrade accuracy or offer negligible benefit, indicating that unstructured image accumulation often dilutes signal with noise and impedes evidence integration.

Action vs. Perception Bottlenecks

Ground-truth trajectory results highlight that the bottleneck for most spatial tasks is on the action-selection axis, not perception, except for inherently ambiguous cases (e.g., Specular Reflection, Geometric Configuration). For Rigid Containment, GPT-5 performance increases from 42.5% (active) to 95.0% (oracle views), showing that access to the correct views is determinative—provided the agent can choose them.

Figure 5: Qualitative Study, showing success and failure modes and reasons behind model behavior.

Error cascades and compounded failures are characteristic: suboptimal actions yield poor observations, constraining further action utility and often rendering recovery infeasible within the step budget.

3D Representation: Promise and Pitfalls

Explicit 3D scene graphs—when accurate—yield large gains (e.g., +43.3 points on Geometric Configuration), resolving ambiguities unsolvable in 2D. However, imperfect 3D reconstructions (e.g., VGGT+Gemini) can substantially undermine performance, with errors in object persistence, misplacement, or hallucination introducing structural artifacts detrimental to downstream reasoning. This problem is especially acute for fine-grained counting and relation classification.

Epistemic Calibration and Metacognitive Gap

A pronounced gap is observed between human and model epistemic behavior. Humans outperform leading models under active exploration by leveraging two metacognitive capacities: epistemic caution (delaying commitment under ambiguity and actively seeking disconfirmatory evidence) and belief updating (revising prior answers in light of new, contradictory observations). Models exhibit high overconfidence and premature, anchoring commitments regardless of evidence uncertainty, rarely seeking falsifying views or adjusting predictions as new data arrives.

Emergent Behaviors and Diversity

Active models demonstrate diverse, non-scripted exploration strategies—e.g., varying between moving for alternative viewpoints and direct interaction (lifting, pouring)—indicating emergent planning but not effective metacognition. Human-annotation studies confirm multiple distinct classes of successful behaviors within subcategories, highlighting the potential for learning-based diversity given suitable feedback.

Figure 6: Additional qualitative examples illustrating emergent agent behaviors and failure modes, including emergent strategies, misleading mirror setups, and counting under merged observations.

Figure 7: Step budget ablation for Gemini 3.1 Active. Performance rises quickly up to 15--20 steps, saturates by 30, and decreases slightly after 40.

Implications and Future Directions

The ESI-Bench suite delivers granular diagnostic signals for embodied agent design:

• Benchmarks must foreground the perception-action coupling; evaluation on fixed or random passive views misses critical bottlenecks and learning targets for spatial skill acquisition.
• 3D scene representations can substantially aid tasks requiring precise spatial inference, but the value is contingent on accurate and uncertainty-aware 3D reconstruction pipelines; naive augmentation can degrade downstream performance.
• Closing the metacognitive gap will require new architectures or agents that can actively reason about epistemic uncertainty, orchestrate information-seeking policies, and implement belief revision—not simply scale up perception modules or enlarge datasets.
• The synthetic simulator environment yields controlled attribution of failure sources, but domain transfer to real-sensor environments, actuation, and domain shift adaptation remains an open problem.

ESI-Bench sets a clear research agenda: advance agents that actively reason about where to look, what to do, and when to decide, using uncertainty as a guide for exploration and learning robust spatial abstractions under partial observability.

Conclusion

ESI-Bench constitutes a rigorous testbed for embodied spatial intelligence that explicitly requires agents to close the perception-action loop. Substantial gains from active exploration over passive observation, the dominance of action-selection failures, the conditional advantage of 3D representations, and the pronounced metacognitive gap between agents and humans, all accentuate the necessity for next-generation models with integrated active perception, epistemic reasoning, and belief updating capabilities. This benchmark provides actionable diagnostics and a rich environment for future research targeting robust, trustworthy, and spatially competent embodied agents (2605.18746).