S-Agent: Spatial Tool-Use Elicits Reasoning for Spatial Intelligence

Abstract: Real-world spatial intelligence requires reasoning over a continuous and evolving 3D world, yet existing VLMs and tool-augmented agents largely remain tied to static, stateless inference from isolated visual observations. We introduce \textbf{\textsc{S-Agent}}, a spatial tool-use agentic paradigm for understanding and reasoning over continuous multi-view images and videos. By formulating spatial reasoning as spatio-temporal evidence accumulation rather than isolated frame-level prediction, \textsc{S-Agent} reshapes spatial perception into scene-centric understanding beyond frame-centric recognition. Specifically, \textsc{S-Agent} casts the VLM as a semantic planner that decides what evidence is needed, while a hierarchy of spatial tools and experts grounds objects in 2D, lifts them into 3D geometric evidence, and aggregates this evidence into high-level spatial knowledge (\textit{e.g.}, counting, measurement, orientation, and relative position). Additionally, a temporal memory mechanism, including Scene Memory for maintaining the evolving scene state and Agent Memory for accumulating reasoning context, enables evidence integration across frames and reasoning steps. Comprehensive experiments on multi-view and video spatial reasoning benchmarks show that \textsc{S-Agent} consistently improves both open-source and closed-source VLMs in a training-free manner. Beyond inference-time augmentation, supervised fine-tuning (SFT) on \textsc{S-Agent}-generated spatial trajectories \textsc{S-300K} yields \textsc{S-Agent-8B}, a compact spatial agent that significantly surpasses similar-scale baselines (e.g., Qwen3-VL-8B) and performs comparably to advanced closed-source models (e.g., GPT-5.4 and Gemini 3).

• The paper introduces S-Agent, a novel framework that shifts spatial reasoning from isolated frame-level processing to evidence-driven, scene-centric intelligence.
• It employs a hierarchical suite of spatial tools along with dual memory architectures to extract 2D visual cues, 3D geometric data, and structured spatial knowledge.
• It demonstrates significant benchmark improvements (e.g., 46.4% on MMSI-Bench) in a zero-shot setting, outperforming state-of-the-art models in dynamic and multi-view scenarios.

Evidence-Driven Spatial Intelligence via S-Agent

Motivation and Problem Setting

Spatial intelligence for VLMs demands robust reasoning over continuous, multi-view, and temporally-evolving 3D worlds, a prerequisite for embodied AI applications in robotics, AR/VR, and autonomous systems. Contrary to human perceptual integration of 3D semantics, conventional VLMs predominantly operate in isolated 2D contexts and generally lack explicit geometric or embodied supervision. This induces a semantic-geometric gap: VLMs are susceptible to lossy semantic priors and unreliable spatial impressiveness, especially when required to generalize across evolving views, occlusions, and temporally persistent object states. Previous agentic approaches, such as VADAR and SpaceTools, augment VLMs with spatial tool APIs and programmatic reasoning, but remain restricted in either static image processing or stateless frame-level inference.

S-Agent Paradigm: Methodological Overview

S-Agent proposes an agentic paradigm that formalizes spatial reasoning as a process of spatio-temporal evidence accumulation, centered on explicit tool-use and persistent memory management rather than end-to-end parameterized spatial capability. The S-Agent pipeline orchestrates hierarchical spatial tools, a VLM-based semantic planner, and dual memory architectures to transition scene understanding beyond frame-centric recognition toward coherent, scene-centric spatial intelligence.

Figure 1: The S-Agent pipeline integrates hierarchical spatial tools, semantic planning, and persistent memory for stateful reasoning across continuous multi-view inputs.

Key design components:

• Hierarchical Spatial Tools: S-Agent decomposes spatial evidence acquisition into three levels:
  1. 2D Visual Evidence: Grounding and localizing query-relevant entities via open-vocabulary detectors and frame selection.
  2. 2D-to-3D Geometric Lifting: Multi-view geometric tools recover metric depth, 3D coordinates, and camera poses from fragmented visual cues.
  3. Spatial Knowledge Aggregation: Specialized experts perform deterministic, task-specific measurements, counting, orientation, and relational computations.

• Temporal Memory: Scene Memory binds grounded entities, visual and geometric facts across frames, preserving object identity and suppressing redundant evidence. Agent Memory records the reasoning trajectory, tool call states, intermediate thoughts, and procedural context.

• VLM Semantic Planner: The planner iteratively issues evidence requests conditioned on evolving scene and agent memory, invoking spatial tools in an adaptive, evidence-driven manner.

Hierarchical and Evidence-Grounded Spatial Reasoning

The agentic reasoning loop unfolds as follows. At each step, the planner issues a request for evidence based on query requirements and current memory states. Depending on the tool level, observations yield either localized visual facts, metric geometry, or high-level structured spatial knowledge. This explicit separation enables the VLM to focus exclusively on semantic planning, with spatial computation performed deterministically or via expert modules. Memory updates ensure persistent context for iterative refinement and suppress duplication across instances.

Zero-Shot and Distillation-Based Performance

Comprehensive experimentation across spatial reasoning benchmarks (MMSI-Bench, ViewSpatial-Bench, ReVSI, and VSI-SUPER) demonstrates that S-Agent effectively enhances both open-source and proprietary VLMs in a training-free, zero-shot manner. S-Agent achieves the highest average score of 46.4% on MMSI-Bench, outperforming Gemini 3 Pro by 1.2% and GPT-5.4 by 4.5%. On ViewSpatial-Bench, S-Agent secures an average score of 60.0%, exceeding GPT-5.4 by 14.4%. The agent exhibits pronounced gains on camera and object motion, multi-step reasoning, perspective-aware localizations, and relative-direction reasoning, underscoring the necessity of temporal evidence integration for robust spatial intelligence.

Figure 2: S-Agent accurately infers 3D relations using hierarchical spatial tools and depth-guided experts, outperforming vanilla VLMs on incomplete cues.

Figure 3: Diverse spatial reasoning scenarios: S-Agent dynamically adapts tool invocation for counting, measurement, multi-step reasoning, and spatial relation tasks.

Beyond inference-time augmentation, S-Agent-generated spatial trajectories are distilled to train a compact spatial agent (S-Agent-8B), using the S-300K dataset. S-Agent-8B outperforms the underlying Qwen3-VL-8B backbone, improving accuracy from 31.1% to 41.6% on MMSI-Bench, and achieves competitive results vis-à-vis advanced closed-source LLMs.

Ablation and Qualitative Insights

Ablative experiments affirm the importance of expert-mediated interpretation and persistent memory modules. Incorporating Level-1 2D evidence markedly improves VLM baselines; however, direct 3D evidence injection has limited benefit unless mediated by experts capable of semantic filtering and relation-specific computation. Scene and agent memories further elevate performance by enabling reusable evidence and trajectory-based planning.

Qualitative examples validate S-Agent's explicit evidence collection and geometric grounding. Cases involving partial occlusion, ambiguous spatial relations, and multi-step aggregation illustrate S-Agent's iterative, tool-grounded trajectory toward accurate answers.

Figure 4: Additional qualitative examples of S-Agent in evidence-driven spatial reasoning.

Figure 5: S-Agent demonstrates persistent, evidence-driven reasoning in complex spatial tasks.

Implications and Future Directions

S-Agent reframes spatial intelligence in multimodal models as an evidence-driven, agentic process, contrasting with the prevailing training-driven, single-shot inference paradigm. Practically, this enables more stable and generalizable spatial reasoning with reduced susceptibility to brittle semantic priors and shortcut exploitation. Theoretically, the paradigm advances scene-centric intelligence, bridging the semantic-geometric gap by persistent accumulation and structured integration of spatial evidence. The approach is conducive to further autonomy in embodied agents, systematic reasoning under uncertainty, and scalable distillation for compact spatial agents.

There remains significant opportunity for extending S-Agent towards continuous exploration, interleaved active perception, learned spatial tool optimization, and reinforcement-driven agentic refinement, especially for tasks requiring long-term spatial persistence, cross-modal association, and planning under partial observability. Integration with advanced 3D reconstruction, spatial calibration, and high-frequency sensory inputs could produce agents with more nuanced, dynamic spatial intelligence.

Conclusion

S-Agent establishes a formal, agentic framework for spatial intelligence by leveraging hierarchical spatial tools, semantic planning, and persistent memory across continuous multi-view observations. It consistently elevates VLM spatial reasoning performance in both zero-shot and supervised distillation regimes, with strong empirical results on motion, multi-step, and perspective-aware tasks. The evidence accumulation paradigm offers a scalable pathway toward robust, grounded spatial intelligence in vision-language agents, presenting new prospects for both theoretical advancements and applied embodied AI (2606.20515).