Spatial Memory Bank: Design & Applications

• Spatial Memory Bank is a design pattern that stores spatially organized representations, enabling retrieval based on geometric support rather than mere temporal order.
• They utilize diverse substrates—from deep feature maps and 3D Gaussian indices to voxel grids and semantic landmarks—to support tasks like segmentation, detection, and navigation.
• Applications in computer vision, robotics, and neuroscience demonstrate a balance between fidelity and compression, addressing challenges like occlusion, dynamic updates, and multi-agent coordination.

A spatial memory bank is a memory structure that stores and reuses spatially organized representations across time, viewpoints, agents, or cognitive states. In current research, the stored substrate ranges from deep feature maps at specific encoder stages, to sets of 3D Gaussian indices, static point clouds with 2D and 3D descriptors, voxelized feature grids, object- and landmark-centered semantic records, and topological codes defined over neuronal assemblies. What unifies these otherwise heterogeneous systems is that memory is not treated as an unstructured temporal cache: it is tied to spatial support, queried by spatial or geometric criteria, and updated so that past observations remain actionable for segmentation, detection, reasoning, navigation, or map formation (Zhang et al., 17 Dec 2025, Lyu et al., 2024, Wang et al., 13 Jan 2026, Babichev et al., 2016).

1. Representational scope and stored entities

The most immediate form of a spatial memory bank is a store of dense spatial features. In MWNet, the “spatial–temporal memory bank” is attached after the deepest encoder stage and stores deep spatial feature maps from previous frames, with long-term entries $L_i$ and short-term entries $S_j$ satisfying $L_i, S_j \in \mathbb{R}^{C \times H' \times W'}$. Its purpose is to preserve fine spatial details over long temporal horizons and to complement the short-window ConvLSTM fusion in the encoder (Zhang et al., 17 Dec 2025). A related but more class-structured design appears in semi-supervised semantic segmentation, where the memory bank is class-wise, pixel-level, and global across training: for each semantic class it stores a FIFO queue of projected 256-D teacher features, with per-class capacity $\psi = 256$ by default (Alonso et al., 2021).

A second family stores explicitly geometric entities rather than dense CNN features. Gaga’s 3D-aware memory bank stores groups of Gaussian indices, $\mathcal{M}=\{\mathcal{G}_1,\dots,\mathcal{G}_K\}$, where each $\mathcal{G}_i\subset\{1,\dots,N_G\}$ represents one 3D object instance in Gaussian Splatting. The memory entry is therefore a spatial support in world coordinates, not a learned key-value embedding (Lyu et al., 2024). CogniMap3D likewise treats memory as scene-level static geometry: each memory stores a static point cloud $$P_{\text{static}}\in\mathbb{R}^{N_{\text{pts}}\times 3}$$ together with associated 2D and 3D features, and excludes dynamic geometry by construction (Wang et al., 13 Jan 2026).

Embodied systems extend this idea toward hybrid symbolic–geometric stores. BSC-Nav separates a sparse landmark memory, $\mathcal{M}_{\text{landmark}}=\{L_k\}_{k=1}^N$ with $L_k=\{c_k,\theta_k,\rho_k,\mathcal{T}_k\}$, from a dense allocentric voxelized cognitive map $$\mathcal{M}_{\text{cog}}=\{\mathcal{F}_{\mathbf v}\}_{\mathbf v\in\mathcal V}$$, where each voxel keeps a bounded buffer of visual features (Ruan et al., 24 Aug 2025). Meta-Memory stores high-density semantic-spatial entries $S_j$0: caption, caption embedding, four-frame composite image, and metric 2D position, all indexed in a vector database (Mao et al., 25 Sep 2025). In robotic control, mindmap turns a TSDF-based metric-semantic reconstruction into a feature mesh $S_j$1, whose vertices and features act as reconstruction tokens and thus as a scene-centric spatial memory bank for a 3D diffusion policy (Steiner et al., 24 Sep 2025).

This diversity suggests that “spatial memory bank” is best understood as a design pattern rather than a single architecture. The common invariant is that memory entries preserve spatial support—pixels, proposals, voxels, points, Gaussians, landmarks, or topological regions—and are therefore retrievable by more than semantic similarity alone.

2. Read, write, and compression mechanisms

Read operations vary from attention to hard geometric matching. MWNet reads memory through pixel-level cross-attention: the current deep feature $S_j$2 produces a query $S_j$3, while keys and values are derived separately from concatenated long-term and short-term memories. The two attention outputs are concatenated, aligned with a $S_j$4 convolution, and passed through an FFN before decoding. The bank is then updated every frame, with the short-term memory acting as a queue and the long-term memory pruned by similarity-based compression under fixed lengths $S_j$5 and $S_j$6 in the reported setting (Zhang et al., 17 Dec 2025).

Q-GeoMem makes the write policy explicitly question-dependent. Its Semantic-Geometric Evidence Bank stores compact pooled entries $S_j$7 together with relevance $S_j$8, novelty $S_j$9, and evidence score $L_i, S_j \in \mathbb{R}^{C \times H' \times W'}$0. When the bank is full, the candidate with the smallest $L_i, S_j \in \mathbb{R}^{C \times H' \times W'}$1 is evicted. Reading uses these stored evidence scores as attention modulation, while a separate Fine-Grained Context Bank keeps the most recent $L_i, S_j \in \mathbb{R}^{C \times H' \times W'}$2 frames in a sliding window (Gao et al., 26 May 2026). MAMBA introduces yet another axis: instead of concatenating all stored features during video object detection, it samples a light-weight key set from a much larger memory bank and updates memory feature-wise rather than frame-wise. This decouples memory capacity from attention cost and allows the bank to preserve informative spatial features across many more frames (Sun et al., 2024).

In systems grounded directly in geometry, read and write become set operations. Gaga writes a new 2D mask into memory by computing the set of corresponding Gaussians $L_i, S_j \in \mathbb{R}^{C \times H' \times W'}$3, measuring overlap with every existing group using

$L_i, S_j \in \mathbb{R}^{C \times H' \times W'}$4

and either merging into the maximally overlapping group when the overlap exceeds $L_i, S_j \in \mathbb{R}^{C \times H' \times W'}$5 or creating a new group otherwise (Lyu et al., 2024). CogniMap3D retrieves candidate scenes through approximate nearest-neighbor search in a global feature table of 2D descriptors, then verifies the match with ICP on static point clouds using inlier count and RMSE thresholds, and finally fuses the aligned static points back into memory (Wang et al., 13 Jan 2026). In Shared Spatial Memory Through Predictive Coding, read and write are mediated by a variational information bottleneck: the sender compresses its internal spatial state into a latent message $L_i, S_j \in \mathbb{R}^{C \times H' \times W'}$6, and the receiver decodes from $L_i, S_j \in \mathbb{R}^{C \times H' \times W'}$7 plus its own state so as to minimize future prediction error while paying a KL rate penalty (Fang et al., 6 Nov 2025).

A recurring design tension is therefore between expressivity and compactness. Dense feature banks require sampling or compression; geometric banks require hard update rules to prevent duplication; distributed multi-agent banks require messages that preserve predictive value under severe bit budgets.

3. Metric anchoring, geometry, and retrieval

A spatial memory bank becomes qualitatively different from a generic cache once entries are metrically anchored. Gaga is explicit about this point: a memory entry is 3D-aware because group membership is defined over Gaussians anchored in world coordinates by positions $L_i, S_j \in \mathbb{R}^{C \times H' \times W'}$8, and masks from arbitrary views are associated by overlap in Gaussian index space rather than by temporal continuity or 2D similarity (Lyu et al., 2024). CogniMap3D uses the same principle at scene scale: static points reconstructed from depth and pose are stored in octrees with adaptive voxel downsampling, indexed by DINOv2/VGGT global descriptors for retrieval, and verified geometrically with ICP before relocalization and factor-graph refinement (Wang et al., 13 Jan 2026).

Embodied QA and navigation systems exploit metric anchors to support query-conditioned retrieval. Meta-Memory stores each segment as $L_i, S_j \in \mathbb{R}^{C \times H' \times W'}$9 and combines Semantic-Similarity Retrieval with Spatial-Range Retrieval. The latter returns

$\psi = 256$0

after which an LLM filters candidate memories and a Memory-Integration tool builds a cognitive map using landmark positions and Dijkstra paths before returning an $\psi = 256$1 answer (Mao et al., 25 Sep 2025). BSC-Nav similarly projects detections and DINOv2 patch features into world coordinates, fuses overlapping landmarks, and maintains a voxelized allocentric map whose buffers are updated only when a new feature’s neighborhood surprise exceeds a threshold $\psi = 256$2 (Ruan et al., 24 Aug 2025).

The same emphasis on geometry appears in language-agent memory-palace systems. There, each memory is anchored by a world coordinate, and visibility is treated as a separate perception predicate over stored geometry rather than as a property of retrieval. A ray-versus-voxel digital differential analyzer, applied from observer to target, supplies the required line-of-sight test; in controlled behind-wall targets, text and live FoV cone both score $\psi = 256$3 while cone-plus-DDA reaches $\psi = 256$4, and the live pre-registered run reduces false-visible on 96 behind-wall targets from $\psi = 256$5 to $\psi = 256$6 (Kwon et al., 9 Jun 2026). This result is narrower than a general theory of memory, but it isolates a decisive point: coordinates alone enable spatial recall, whereas occlusion requires both coordinates and stored geometry.

4. Operational roles across application domains

Across application areas, spatial memory banks serve different immediate functions, but those functions are structurally similar: maintain latent spatial consistency, retrieve information that is absent from the current view, and support downstream inference under partial observability.

Domain	Memory form	Operational role
Ultrasound video segmentation	Long/short-term deep feature maps	Preserve boundaries, track small low-contrast lesions (Zhang et al., 17 Dec 2025)
Semi-supervised segmentation	Class-wise FIFO of pixel features	Pull labeled and unlabeled pixels toward same-class global distributions (Alonso et al., 2021)
Video object detection	Pixel- and proposal-level feature banks	Aggregate long-range spatial-temporal evidence at manageable cost (Sun et al., 2024)
Open-world 3D segmentation	Gaussian index groups	Enforce cross-view instance consistency by 3D overlap (Lyu et al., 2024)
Video spatial reasoning	FGCB + SGEB	Retain recent dense context and compact long-range question-relevant evidence (Gao et al., 26 May 2026)
Robotic manipulation	TSDF feature mesh tokens	Condition 3D diffusion policies on scene-centric memory outside the current FoV (Steiner et al., 24 Sep 2025)
Navigation and SLQA	Landmark lists, cognitive maps, semantic-spatial memory entries	Answer location queries, retrieve targets, and plan routes (Mao et al., 25 Sep 2025)
Embodied navigation	Landmark memory + voxel cognitive map	Support zero-shot object, instance, and long-horizon navigation (Ruan et al., 24 Aug 2025)

The same pattern extends to shared multi-agent settings. In Shared Spatial Memory Through Predictive Coding, each agent maintains a BEV map and a grid-cell-like metric for self-localization, while communication transmits only the bottlenecked information most predictive of a partner’s future observation. On Memory-Maze, success degrades gracefully from $\psi = 256$7 to $\psi = 256$8 as bandwidth shrinks from 128 to 4 bits/step, whereas a full-broadcast baseline drops from $\psi = 256$9 to $\mathcal{M}=\{\mathcal{G}_1,\dots,\mathcal{G}_K\}$0 (Fang et al., 6 Nov 2025). In other words, the bank need not be centralized to be shared: it can emerge as a synchronized set of local memories coupled by uncertainty-reducing messages.

A plausible implication is that application-specific differences mostly change the stored substrate and the read/write controller, not the underlying rationale. Whether the downstream task is segmentation, detection, question answering, navigation, or control, the bank is valuable when instantaneous sensory evidence is insufficient.

5. Biological and topological conceptions

In computational neuroscience, spatial memory bank concepts are often expressed without key–value language but with equivalent structural intent. One line of work models hippocampal memory as a flickering simplicial complex generated by transient place-cell assemblies. Even though assemblies appear and disappear rapidly, the large-scale topology of the coactivity complex—captured by Betti numbers—can remain stable and match the topology of the environment. In the canonical arena with a central hole, this topological barcode is $\mathcal{M}=\{\mathcal{G}_1,\dots,\mathcal{G}_K\}$1, and persistent spatial memory refers to persistence of that topological structure rather than persistence of individual synapses (Babichev et al., 2016). A closely related model with transient connections reaches the same conclusion via zigzag persistent homology: the cognitive map’s stability is an emergent property of large-scale topological invariants, and weakening of synaptic connectivity can be compensated by simulating neuronal activity (Babichev et al., 2017).

A more abstract generalization treats hippocampal memory spaces as finite topological spaces induced by simplicial or relational schemas. In that framework, a spatial cognitive map is a special case of a broader memory space $\mathcal{M}=\{\mathcal{G}_1,\dots,\mathcal{G}_K\}$2, and memory consolidation corresponds to topological reduction toward a compact core that the paper links to Morris’ schemas (Babichev et al., 2017). Another attractor-based formulation replaces explicit topology with localized persistent states in a 2D manifold. There, a local random network with global normalization stores spatial locations as discrete attractors; for $\mathcal{M}=\{\mathcal{G}_1,\dots,\mathcal{G}_K\}$3, the information-theoretic estimate based on $\mathcal{M}=\{\mathcal{G}_1,\dots,\mathcal{G}_K\}$4 bits yields $\mathcal{M}=\{\mathcal{G}_1,\dots,\mathcal{G}_K\}$5 distinguishable locations (Natale et al., 2019).

Recent work further reframes capacity in geometric terms. Comparing food-caching chickadees to zebra finches, one study reports a transition from a disorganized “mist” to a “crystalline” hippocampal code, with Shesha geometric stability $\mathcal{M}=\{\mathcal{G}_1,\dots,\mathcal{G}_K\}$6 versus $\mathcal{M}=\{\mathcal{G}_1,\dots,\mathcal{G}_K\}$7, temporal coherence $\mathcal{M}=\{\mathcal{G}_1,\dots,\mathcal{G}_K\}$8 versus $\mathcal{M}=\{\mathcal{G}_1,\dots,\mathcal{G}_K\}$9, and a modeled capacity jump from failure below $\mathcal{G}_i\subset\{1,\dots,N_G\}$0 to high-fidelity readout beyond $\mathcal{G}_i\subset\{1,\dots,N_G\}$1, at the cost of a 169-fold representational redundancy (Raju, 16 May 2026). These models do not store memories as explicit data structures, but they embody the same central idea: spatial memory is stabilized by a structured manifold, topological invariant, or attractor geometry that preserves relational information despite local change.

6. Distinctions, limitations, and open tensions

Several recurrent misconceptions are corrected by the literature itself. First, a spatial memory bank is not synonymous with a generic temporal cache. Q-GeoMem argues that generic caches store redundant or irrelevant geometry and therefore weaken long-horizon reasoning; what matters is question-guided evidence selection and adaptive fusion, not mere history length (Gao et al., 26 May 2026). Second, a memory bank need not be learned. Gaga’s bank is explicitly “not a learned neural module”; it is a set of Gaussian index sets built online and read by geometric overlap (Lyu et al., 2024). Third, spatial recall and visibility are distinct operations: recall can and often should be occlusion-blind, whereas visibility is a perception predicate over stored geometry that must be computed separately (Kwon et al., 9 Jun 2026).

The limitations are equally consistent. MWNet improves ultrasound segmentation but incurs computational and memory overhead, with prediction speed and GPU consumption worse than single-frame methods (Zhang et al., 17 Dec 2025). The class-wise pixel memory of semi-supervised segmentation does not explicitly model spatial relationships and may face scalability issues with very large class sets (Alonso et al., 2021). CogniMap3D depends on the quality of static scene capture and can fail under scene mismatch, registration error, or larger-scale memory growth (Wang et al., 13 Jan 2026). Meta-Memory assumes static environments during deployment, and its LLM-driven reasoning remains exposed to hallucinations and retrieval errors (Mao et al., 25 Sep 2025). In embodied systems more generally, fixed capacities, sparse communication, and geometry errors can cause important long-range evidence to be dropped or misaligned (Fang et al., 6 Nov 2025, Ruan et al., 24 Aug 2025).

A broader tension concerns what the bank should optimize for: fidelity, compression, or actionability. Dense banks preserve detail but are expensive; compressed banks scale but risk losing rare evidence; object- or geometry-centric banks are interpretable but may underrepresent appearance variation; biologically inspired topological banks preserve invariants but discard metric detail. Current work suggests no single optimum. Instead, spatial memory banks are increasingly designed around the task’s irreducible query structure—boundary preservation in ultrasound, 3D overlap in open-world segmentation, metric visibility in language agents, or uncertainty reduction in multi-agent coordination.