NavSpace Benchmark: Spatial Navigation Evaluation

• NavSpace Benchmark is a standardized evaluation framework that assesses the spatial reasoning of navigation agents using trajectory–instruction pairs in indoor environments.
• It systematically tests key spatial skills—including vertical perception, precise movement, and relational positioning—across six defined task categories.
• Empirical findings show that current models, including advanced LLM-driven systems, struggle to convert linguistic spatial cues into accurate navigational actions.

The NavSpace Benchmark is a standardized evaluation framework expressly designed to probe the spatial intelligence of embodied navigation agents. Distinct from prior benchmarks concentrating on semantic comprehension, NavSpace introduces a suite of tasks and trajectory–instruction pairs to systematically assess agents’ abilities in spatial reasoning, perception, and decision-making within visually realistic indoor environments. Its six spatially-focused categories and rigorously curated dataset enable granular analysis of agent performance, revealing key limitations in contemporary models and laying the groundwork for future advances.

1. Construction and Composition of the NavSpace Benchmark

NavSpace is constructed through a four-stage pipeline that ensures both fidelity and diversity in the benchmark content:

• Survey and Category Selection: An initial questionnaire survey identified six frequently encountered spatial instruction types reflecting distinct spatial reasoning dimensions. These categories address vertical and horizontal localization, movement precision, viewpoint transformations, relational object positioning, environmental state inference, and structure understanding.
• Photo-Realistic Trajectory Collection: Annotators control autonomous agents within Habitat 3.0’s HM3D scenes, collecting trajectory data at every time step (RGB frame, agent action, and positional coordinates).
• Multimodal Instruction Annotation: For each trajectory, multimodal LLMs (such as GPT-5) generate spatially principled candidate instructions that encapsulate the intended spatial relationship and action sequence.
• Human Cross-Validation: Independent annotators replay collected trajectories to ensure the instruction achieves the prescribed spatial goal, and that the language is unambiguous and executable.

The final benchmark consists of 1,228 trajectory–instruction pairs, each annotated with a detailed spatial instruction and verified for execution success.

2. Task Categories and Instruction Types

NavSpace’s six main task categories are engineered to isolate core spatial reasoning challenges:

Task Category	Instruction Example	Success Radius
Vertical Perception	"Go to the topmost floor"	3.0 m
Precise Movement	"Turn right 180° and move forward 1 m"	1.0 m
Viewpoint Shifting	"Imagine you are the TV...Move toward your front-left"	2.0 m
Spatial Relationship	"Turn at the third door on your left"	2.0 m
Environment State	"If you see the keys, stop, else go to the door"	2.0 m
Space Structure	"Walk around the dining table once"	1.0 m

Each instruction is tightly linked to the trajectory’s spatial semantics, requiring agents to interpret scale, viewpoint transformation, metric relationships, conditional logic, and environment topology. The benchmark operationalizes “success” as reaching a spatial goal within a predefined radius.

3. Comprehensive Evaluation of Navigation Agents

NavSpace enables the head-to-head evaluation of diverse navigation agents, benchmarking 22 distinct models across the six spatial task categories:

• Chance-level Baselines: Random or frequency-based action selection establishes lower bounds (average success rates below 10%).
• Open-source Multimodal LLMs: Models including LLaVA-Video 7B, GLM-4.1V-Thinking 9B, and Qwen2.5-VL, despite leveraging advanced language–vision architectures, exhibit performance near chance, failing to bridge static visual scene understanding and dynamic navigation planning.
• Proprietary MLLMs: Advanced systems such as GPT-4o, GPT-5, and Gemini 2.5 Pro/Flash attain somewhat higher success rates (~20%), but still fall short on complex spatial reasoning requirements.
• Lightweight Navigation Models: Classical approaches (Seq2Seq, CMA, HPN+DN, VLN⟳BERT, Sim2Sim) display limited generalization outside standard VLN paradigms.
• Navigation Large Models: Specialized agents such as NaVid, NaVILA, and StreamVLN demonstrate stronger performance on NavSpace’s spatial dimensions, yet do not achieve robust generalization.

Key empirical findings include widespread failures of LLM-driven agents to translate linguistic spatial cues into precise embodied actions, and the apparent need for models with explicit spatial reasoning modules. Evaluations use metrics such as Navigation Error (NE), Oracle Success Rate (OS), and Success Rate (SR).

A representative agent decision function is:

$$a_{t+1} = \argmax_{a \in A} \, f\Big( p\Big(v(\{I_1, I_2, \ldots, I_t\})\Big), X \Big)$$

where $v(\cdot)$ encodes frames, $p(\cdot)$ projects features, and $f(\cdot)$ integrates with the language instruction $X$.

4. SNav: A Spatially Intelligent Navigation Model

To address the identified deficiencies, the SNav model is introduced with architectures that explicitly target spatial reasoning:

• Core Architecture: SNav is built around three modules—a SigLIP-based vision encoder $v(\cdot)$, a two-layer MLP projector $p(\cdot)$, and a Qwen2 LLM $f(\cdot)$ for auto-regressive navigation action generation.
• Dedicated Fine-tuning Pipelines: SNav’s training regimen spans navigation action prediction, trajectory-based instruction synthesis, multi-modal data recall, and specialized sub-tasks including cross-floor navigation, precise movement, relational parsing, and conditional environment state inference.
• Co-training Strategies: SNav leverages large-scale multinomial trajectory–instruction pairs and targeted data generation pipelines that reinforce learning of complex spatial skills.

A diagram in the source (Fig. SNav) illustrates how raw scene data is transformed through spatially detailed instructions and fed into the agent for planning and decision making.

5. Benchmark Results and Real-world Transfer

Quantitative results demonstrate that SNav outperforms both navigation large models (StreamVLN, NaVid) and proprietary MLLMs (GPT-5, Gemini) on NavSpace:

• Ablation Studies: Pipeline-specific ablations confirm that each spatial data-generation component enhances overall performance.
• Real-world Evaluation: On a physical AgiBot Lingxi D1 quadruped equipped with a monocular RGB camera, SNav attained a 32% average success score across five instruction categories, compared to NaVid (14%) and NaVILA (6%). Visual evidence (Fig. realworld) illustrates qualitative improvements in task execution.

6. Significance and Future Directions

NavSpace establishes a new gold standard for evaluating spatial reasoning in embodied agents. By deconstructing navigation into orthogonal spatial dimensions, the benchmark exposes fundamental limitations in existing models—especially their inability to generalize from language-encoded spatial instructions to concrete navigational actions. The introduction of SNav and its tailored co-training approaches exemplifies future paths to improved spatial intelligence.

Potential extensions suggested in the paper include scaling the benchmark to complex, multi-floor environments, enhancing cross-modal alignment, and developing compositional reasoning modules. A plausible implication is that future navigation benchmarks will adopt similarly principled methodologies to evaluate spatial understanding in conjunction with semantic and pragmatic competence, driving advances in embodied AI.