SpatialClaw: Interactive Spatial Reasoning

• SpatialClaw is a training-free framework that enables agentic spatial reasoning with an interactive Python coding interface, allowing agents to plan, execute, and revise their analysis.
• It overcomes vision–language model limitations by integrating geometric and perception primitives for flexible, multi-step 3D/4D spatial reasoning tasks.
• The system’s persistent Python kernel and five-stage agent loop achieve significant accuracy improvements on spatial benchmarks, supporting compositional reasoning in complex visual scenarios.

SpatialClaw is a training-free framework for agentic spatial reasoning that rethinks the action interface for vision–language agents tasked with open-ended spatial reasoning in 3D and 4D. It addresses the crucial problem of enabling vision–LLMs (VLMs) to perform compositional, flexible analysis of spatial relationships and dynamics in images and video, overcoming the limitations of prior tool-augmented architectures. SpatialClaw leverages a stateful, iterative coding interface, allowing agents to plan, execute, observe, and revise their reasoning process using Python code cells and a suite of geometric and perception primitives, thus supporting multi-step, compositional reasoning workflows not achievable with single-pass or rigid tool-call interfaces (Cho et al., 11 Jun 2026).

1. Motivation and Problem Definition

Spatial reasoning in VLMs encompasses tasks such as inferring object proximity (“Which object is closest to the table?”), motion directionality (“Is the car moving toward the camera?”), and relative positions (“Did the person turn left or right?”). These tasks require integration of depth estimation, camera pose, object trajectories, and geometric computation. Conventional VLMs exhibit two core limitations: (a) they cannot extract reliable metric depth or 3D motion purely from appearance, and (b) their reasoning chain operates in natural language and cannot invoke explicit geometric operations.

Prior tool-augmented agents rely either on single-pass code execution—writing a complete analysis program before observing any intermediate output—or on structured tool calls via a restricted API (e.g., JSON or XML RPC), mapping questions to a fixed set of atomic perception or geometry operations. Both frameworks lack the flexibility for open-ended, multi-step spatial reasoning that adapts to intermediate results and task demands. This motivates the investigation of an action interface in which flexible code, written interactively with intermediate feedback, serves as the mechanism for invoking and composing spatial reasoning primitives.

2. System Architecture and Workflow

SpatialClaw comprises two primary components:

A. Persistent Python Kernel

• The kernel is initialized per question with:
  • InputImages: up to 32 key frames or video frames.
  • Metadata: frame rate, total frames, image vs. video.
  • Pre-imported perception and geometry primitives.
• Tool Entry Points:
  1. tools.Reconstruct(frames): Framewise depth maps, camera intrinsics/extrinsics, dense point clouds.
  2. tools.SAM3.segment_*: Image/video segmentation masks.
  3. tools.Geometry (Euclidean distance, angle_between_vectors, project_point_to_camera, etc.).
  4. tools.Mask (area, bounding_box, centroids, iou, etc.).
  5. tools.Time (frame_to_seconds, seconds_to_frame, etc.).
  6. tools.Graph.plot: Numeric sequence plotting.
  7. tools.Draw.draw_bbox/line/point: PIL overlays.
  8. show(img_or_plot): Registers image/plot for the next agent turn.
  9. vlm.locate, vlm.ask_with_thinking: VLM-backed image query tools. 10. ReturnAnswer(ans): Signals final answer and terminates.

Variables are maintained in persistent scope throughout the interaction, supporting flexible recombination and analysis.

B. Five-Stage Agent Loop

The agent executes the following loop for each question:

1. Planning (one-shot LLM): Receives question, metadata, tool docs; outputs a plan with sub-tasks, evidence checklist, and tool priors.

2. Code Generation (LLM): Receives current context and plan; emits markdown block with purpose, reasoning, next goal, and a single Python code cell.
3. Static Checking & Execution: AST parsed to reject forbidden operations; code cell executed if valid.
4. Feedback Assembly: Captures stdout, error traces, variable type/shape summaries, and visualization outputs; integrates this context into the next agent turn.
5. Answer Submission: If ReturnAnswer observed or step/error budget exceeded, answer is finalized; otherwise, fallback to chain-of-thought.

Pseudocode for this loop:

$$\begin{aligned} \text{for step}=1\ldots N_{\max}: \ &\quad \text{msg} \leftarrow \{\text{question, plan, prior feedback, shown images}\}\ &\quad \text{agentReply} \leftarrow \text{LLM.generate(msg)}\ &\quad (\text{purpose, reasoning, nextGoal, code}) \leftarrow \text{parse(agentReply)}\ &\quad \text{if code contains ReturnAnswer: break}\ &\quad \text{if fails staticCheck(code): feedback} \leftarrow \text{traceback};\; \text{continue}\ &\quad \text{execResult} \leftarrow \text{kernel.exec(code)}\ &\quad \text{feedback} \leftarrow \text{stdout, errors, varSummaries, images}\ &\quad \text{append feedback to context}\ \text{end}\ \text{if no ReturnAnswer, use fallback} \end{aligned}$$

3. Formal and Mathematical Underpinnings

SpatialClaw’s core operations draw heavily on geometric and spatial computation:

• Average accuracy over $N$ benchmarks:

$$\mathrm{Acc}_{\mathrm{avg}} = \frac{1}{N} \sum_{i=1}^N \mathrm{Acc}_i$$

• Framewise 3D data from reconstruction:
  • Depth map: $D_f \in \mathbb{R}^{H\times W}$
  • Camera-to-world matrix: $T_f \in \mathrm{SE}(3)$
  • Point cloud: $P_f \in \mathbb{R}^{H\times W\times 3}$
• Geometry primitives:
  • Euclidean distance: $d(p, q) = \lVert p - q \rVert_2$
  • Angle between vectors: $$\theta = \cos^{-1}\left(\frac{v_1 \cdot v_2}{\|v_1\|\|v_2\|}\right)$$
  • Plane fitting (RANSAC): $(n, \texttt{inlierMask})$

These enable the compositional chains of analysis necessary for nuanced 3D/4D spatial reasoning.

4. Evaluation Protocol and Experimental Regime

Benchmarks span 20 tasks, categorized as follows:

Category	Example Benchmarks
Single-image spatial reasoning	ERQA, Omni3D, SPBench
Multi-view spatial reasoning	MindCube, MMSI, SPAR-Bench
Video (4D) spatial reasoning	MMSI-Video, VSI-Bench-U
General spatial reasoning	BLINK, SpatialTree
General video understanding	CV-Bench, Video-MME

• Up to 1,000 samples per benchmark; random seed fixed.
• Metrics: Per-sample categorical accuracy (MCQ, yes/no), mean relative accuracy (MRA) for numeric questions, and aggregate scores like SPAR-Bench's VCI.
• Six open-source VLM backbones were evaluated: Qwen3.5-397B-A17B, Qwen3.5-122B-A10B, Qwen3.6-35B-A3B, Qwen3.6-27B, Gemma4-31B, Gemma4-26B-A4B, all without fine-tuning, under identical setup ($N_{\max}=30$).

5. Quantitative Results

SpatialClaw attained an average accuracy of 59.9% across the 20 benchmarks, surpassing the previous strongest spatial agent (SpaceTools) by 11.2 percentage points. The improvement is especially pronounced in domains requiring multistep geometric reasoning, such as multi-view and 4D (video) spatial reasoning.

Excerpted results (Gemma4-31B backbone):

Category	No-tool	SpaceTools	SpatialClaw
Multi-view avg.	50.2%	52.9%	63.3%
Video 4D avg.	46.0%	56.0%	68.1%
Overall (20 benches)	48.0%	56.7%	59.9%

SpatialClaw consistently exceeds both the no-tool baseline and prior tool-call/code baselines across all evaluated backbones, with observed gains up to +13 percentage points.

6. Ablation and Error Analysis

Tool-set Ablations (Gemma4-26B-A4B, 500 samples each):

Configuration	Accuracy
No utility wrappers (SAM3/DA3 + NumPy/SciPy only)	56.4%
No perception tools (only code, NumPy, etc.)	51.4%
No-tool baseline	48.7%

This suggests that the iterative code interface itself delivers substantial gains (e.g., +2.7pp above no-tool), independently of perception or convenience wrappers.

Usage and Failure Modes:

• Distance-tasks rely on KD-Tree search and vector norm calculations.
• Directional tasks invoke dot products and angular measures.
• Multi-view reasoning invokes repeated reconstruction, plane fitting, and spatial centroid calculations.
• Error breakdown (1,000 cases): ~45% geometric calculation errors, ~20% perception failures, ~15% tool-selection/coverage, ~10% VLM misinterpretation, ~10% budget/timeout.

A plausible implication is that further improvement may require more robust geometric reasoning primitives and perception modules.

7. Comparative Analysis and Insights

SpatialClaw diverges fundamentally from two prior paradigms:

• Single-pass code execution (e.g., pySpatial, VADAR): Agents must commit to an entire program without access to intermediate results, leading to limited error correction and high one-shot failure rates.
• Structured tool-call interfaces (e.g., SpaceTools): Expose a fixed RPC-based API permitting only predefined compositions, restricting adaptive and open-ended geometric computation at test time.

SpatialClaw’s core innovation is treating code as an interactive, stateful action interface, with each agent turn conditioned on all prior outputs—enabling exploratory, revisable spatial analysis and arbitrary composition with NumPy, SciPy, and Matplotlib. The empirical finding is that this interface yields the single largest zero-shot improvement in open-ended 3D/4D spatial reasoning, eclipsing gains from perception tool improvements or model scaling (Cho et al., 11 Jun 2026).