Spatial Memory Agent: Experience-Grounded Procedure Memory for Spatial Intelligence

Abstract: Spatial intelligence is becoming a foundation for embodied agents, robotic planning, and multimodal assistants. To improve the spatial reasoning ability of VLM agents, existing work has mainly followed two lines. One line uses post-training methods, such as supervised fine-tuning and reinforcement learning. Another line adopts an agentic paradigm in which the model calls external spatial tools, such as depth estimation and 3D reconstruction tools, to gather intermediate spatial evidence. We study a complementary and underexplored route: Can a frozen VLM agent improve its spatial reasoning through \textbf{parameter-update-free self-evolution}, without depending on external expert spatial tools at inference time? We present \textbf{Spatial Memory Agent (SMA)}, an \textbf{experience-grounded runtime framework} that converts verified spatial experience into reusable transferable lessons. In a verifiable spatial environment, SMA queries the frozen VLM, obtains a predicted answer and reward, and uses \textbf{verifier-guided reflection} to distill compact transferable lessons from spatial experience. SMA further assigns each lesson a \textbf{Transfer Reliability Score (TRS)}, which is initialized uniformly and calibrated from later retrieval outcomes as visit evidence of future transfer reliability. During \textbf{read-only deployment}, SMA retrieves lessons by semantic filter and similarity-TRS combined ranking, allowing the retrieved memory to guide frozen model inference. Across five representative spatial benchmarks and four base VLMs, SMA achieves the highest macro average in every base-model block and the best accuracy among the evaluated methods in most of the 20 evaluations, establishing a practical parameter-update-free path for spatial self-evolution across the evaluated frozen model scales and environments.

• The paper introduces SMA, a training-free framework that lets frozen vision-language models improve spatial reasoning by writing verifier-grounded procedural memories and retrieving them according to semantic relevance and transfer reliability.
• SMA achieves the best macro-average results across four Qwen model blocks, improving over the strongest baselines by 1.7–2.9 points and raising Qwen3.6-27B performance by 6.5 points over no-memory inference.
• Ablations show that semantic filtering, structured transferable lessons, and calibrated TRS retrieval are essential, while one-pass memory writing reduces redundancy by 21% and roughly doubles reliability-update coverage compared with continual writing.

Overview

"Spatial Memory Agent: Experience-Grounded Procedure Memory for Spatial Intelligence" (2608.12743) introduces SMA, a training-free, parameter-update-free framework that enables a frozen vision-LLM (VLM) to improve its spatial reasoning by maintaining an external bank of verifier-grounded procedural memories. The work positions itself against two dominant routes for improving spatial intelligence—post-training (e.g., SpatialVLM, RoboSpatial, SpatialEvo) and inference-time spatial tool use (e.g., S-Agent, SpaceTools)—by asking whether a frozen VLM can instead distill verified experience into transferable lessons, calibrate their transfer reliability, and retrieve them to guide future reasoning without weight updates or expert spatial tools.

Method

SMA operates in a verifiable spatial environment of problems $\xi_i = (\mathcal{V}_i, t_i, y_i^\star)$, each with visual inputs, a task, and a verifier signal. Each memory card stores the source task, a rollout summary, a transferable lesson, a visit count, cumulative reward, and a Transfer Reliability Score (TRS). Three components define the pipeline:

Verifier-guided reflection. A reflection model (the same frozen VLM) converts each verifier-scored rollout into a strict-JSON memory containing a summary and a compact transferable lesson ("when <shape>, apply <habit>, avoid <trap>, validate by <check>"), with anti-leakage rules preventing restatement of ground-truth answers.

Two-stage retrieval. A semantic filter over task embeddings (cosine similarity with threshold $\delta$) proposes candidates; a combined score $$S_{ij} = (1-\eta)\,z(\text{rel}_{ij}) + \eta\,z(v_j)$$ then ranks candidates by normalized similarity and TRS, with top-$k$ cards injected into the prompt.

Visit-evidence calibration. TRS is initialized uniformly at $v_0$ regardless of whether the source rollout was correct, then updated as a shrunk empirical mean $v_j \leftarrow (\lambda v_0 + c_j)/(\lambda + n_j)$ whenever the card is retrieved and a reward is observed. This separates future transfer utility from the correctness of the rollout that produced the memory.

A notable design decision is One-Pass Memory Writing: memories are written only on the first pass over the environment split; subsequent passes reuse the fixed bank and update only reliability state. The paper reports that continual writing produces a larger, more redundant bank with lower TRS-update coverage—by the final pass, one-pass writing uses one-tenth as many memories, exhibits 21% less redundancy, and achieves roughly twice the TRS-update coverage. Deployment is strictly read-only: no writes and no value updates occur on the held-out split.

Main results

Across five benchmark slices (RoboSpatial, ERQA, Omni3D, SAT, EmbSpatial) and four frozen Qwen base models, SMA achieves the best macro average in every model block, with gains of 1.7–2.9 points over the strongest baseline:

Base model	SMA avg.	Best baseline avg.	Gain
Qwen3.5-122B-A10B	68.8	66.2 (MemRL-R)	+2.6
Qwen3.6-35B-A3B	66.7	63.8 (MemRL-R)	+2.9
Qwen3.6-27B	69.8	68.1 (MemRL-GT)	+1.7
Qwen3.5-9B	63.5	60.7 (MemP)	+2.8

On Qwen3.6-27B, SMA improves over no-memory inference by 6.5 points, with RoboSpatial rising from 54.1 to 68.5 and Omni3D from 41.6 to 47.6. Extended results on SITE-image and ViewSpatial in the appendix are more mixed: on Qwen3.5-122B-A10B, MemRL-GT exceeds SMA on ViewSpatial (62.8 vs. 59.7) and on the two-benchmark average, so the "best in most of the 20 evaluations" claim does not extend uniformly to the supplementary benchmarks.

Ablations on RoboSpatial (Qwen3.6-27B) attribute the gains to structured writing and calibrated retrieval: removing the semantic filter costs 5.8 points, reward-only reflection costs 5.5, adding raw model output costs 4.4, and removing the transferable lesson or summary costs 3.5 and 3.2 respectively. Sensitivity peaks at TRS weight $\eta = 0.5$ and retrieval depth $k = 3$.

Against the training-based SpatialEvo-7B baseline using the Qwen3.5-9B backbone, SMA reports a macro-average improvement of +16.4 points (63.5 vs. 47.1), higher on every benchmark. The authors frame this as evidence that an external procedure-memory route can be competitive with training-based self-evolution, though the comparison is limited to a single released baseline under their evaluation scope.

Transfer and reliability analysis

Memory banks transfer across models and benchmarks. Model transfer from Qwen3.5-122B-A10B to Qwen3.6-27B yields gains up to +9.4 on RoboSpatial and +5.7 on SAT; benchmark transfer probes (e.g., ERQA→RoboSpatial, +7.6) are positive on every selected probe, with the paper conceding that magnitudes depend on source–target similarity.

The TRS signal itself is validated empirically: binned deployment accuracy rises monotonically from 19.3% in the $[0.2, 0.3)$ TRS bin to 97.3% in $[0.9, 1.0]$, though the authors acknowledge this pooled trend may partly reflect benchmark difficulty and question composition. Retrieval composition matters: when all three retrieved memories come from successful source questions, accuracy is 93.0%, versus 39.0% when all come from failures. A similarity–accuracy analysis against MemP shows SMA reduces macro-average retrieval similarity from 0.792 to 0.698 while raising accuracy from 66.8% to 69.8%—supporting the paper's claim that the best memory is not always the nearest memory, and that TRS converts retrieval from semantic matching into evidence-weighted procedure selection.

At the atomic-ability level (ten post-hoc diagnostic capabilities), SMA improves all ten on average, with the largest gains on Correspondence (+11.2 pp), Attribute (+8.0 pp), and Object motion (+7.6 pp). By contrast, MemP shows negative gains on Tracking (−3.0 pp) and Affordance (−1.9 pp), suggesting unfiltered procedural memory can actively hurt when retrieved experience does not transfer. Qualitative case studies attribute failures to two sources: benchmark-side ambiguity (underspecified questions, hidden geometry) and genuine visual-grounding limitations of the base model, where even high-TRS retrieved memories cannot compensate for misread visual evidence.

Limitations and open questions

The paper is explicit about two gaps. First, credit assignment: task-level verification feedback cannot determine whether a successful or failed outcome should be attributed to memory writing, reflection, retrieval, filtering, or the model's final use of retrieved memory—a limitation it connects to recent attribution-based memory work (AttriMem, Memory-R2, MemQ). Second, memory lifecycle maintenance: SMA has no explicit policies for deletion, merging, compression, expiration, or trusted revision under storage or latency budgets; it relies solely on retrieval-time scoring to downweight unreliable memories. Additionally, the main-table protocol reports the best checkpoint from a 10-pass run with per-benchmark tuned similarity thresholds, which introduces a mild selection advantage over baselines; and the TRS-accuracy correlation is observational rather than causal. Whether attribution-guided process feedback or lifecycle policies can be integrated into parameter-update-free spatial self-evolution remains open.

Conclusion

SMA demonstrates that a frozen VLM can achieve measurable spatial self-evolution through an external, verifier-grounded procedure memory with calibrated transfer reliability, achieving the best average among evaluated memory baselines across four base models and improving over a training-based self-evolving baseline by a wide margin in the tested comparison. The evidence supports reliability-aware retrieval over pure semantic similarity, and one-pass writing over continual rewriting. The framework's dependence on verifier signals, its unresolved credit-assignment problem, and its lack of memory lifecycle management define the boundaries of the current result.