LongSpace: Exploring Long-Horizon Spatial Memory from Perception to Recall in Video

Abstract: Multimodal LLMs (MLLMs) have advanced image and video understanding and can increasingly handle longer visual inputs. Long-horizon tasks such as autonomous driving and robotic navigation require more than recognizing the current view, as models must remember and retrieve previously observed spatial layouts, routes, viewpoint changes, and object states. To evaluate this capability, we introduce LongSpace-Bench, a room-tour video benchmark for long-horizon spatial memory, covering scene perception, spatial relations, and spatial memory. In this work, we further propose LongSpace, a memory framework for long-video spatial reasoning. LongSpace models long videos as sequential chunks, incorporates 3D structural cues into early decoder layers, and constructs layer-aware memory for question-guided retrieval. Experiments on multiple spatial reasoning benchmarks show that LongSpace improves long-video spatial understanding, further demonstrating explicit spatial memory as a key capability for long-horizon video MLLMs.

• The paper introduces LongSpace, highlighting the integration of geometry-aware spatial perception and hierarchical KV memory to tackle long-horizon reasoning in video data.
• The methodology fuses 3D visual cues with transformer features and compresses long video contexts using temporally ordered memory banks, achieving state-of-the-art accuracy on spatial memory tasks.
• The implications extend to advancing autonomous navigation and embodied AI by providing a robust blueprint for handling extended temporal and spatial processing in multimodal applications.

LongSpace: Evaluation and Modeling of Long-Horizon Spatial Memory in Video-LLMs

Introduction

Addressing the need for robust long-horizon spatial memory in Multimodal LLMs (MLLMs), "LongSpace: Exploring Long-Horizon Spatial Memory from Perception to Recall in Video" (2606.05677) introduces both a comprehensive benchmark and a novel model architecture for spatial reasoning over extended video contexts. The study fills the empirical gap in existing benchmarks and methods, which primarily target either short-horizon or static spatial understanding and overlook the requirements for memory, recall, and structured spatial organization during prolonged observation. The work's key focus is evaluating and improving the capacity of MLLMs to encode, retain, and query spatial evidence spanning from seconds to hours in realistic room-tour videos.

Figure 1: LongSpace-Bench spans temporal horizons from seconds to hours, requiring spatial evidence to be retained and retrieved across dynamic observations and scene evolution.

LongSpace-Bench: Benchmarking Long-Horizon Spatial Reasoning

LongSpace-Bench is constructed from continuous indoor room-tour videos, facilitating assessment over diverse temporal scales and a full spectrum of spatial reasoning subtasks. The dataset's taxonomy adheres to three ability groups:

• Scene Perception (e.g., Object Counting, Scene Classification, Scene Consistency): Assessment of stable semantic understanding over large time intervals.
• Spatial Relations (e.g., Relative Distance, Relative Orientation): Geometric configuration inference under changing viewpoints.
• Spatial Memory (e.g., Appearance Order, State Change, Egocentric Reasoning, Route Planning, Route Recall): Memory-dependent tasks requiring evidence retrieval from non-local and distantly observed events.

Figure 2: The distribution of question types and the coverage of video durations underscore the benchmark's coverage over spatial reasoning and memory, including challenging long-horizon segments.

Grounding each question in annotated video evidence, the benchmark contains 445 videos with 159 hours of footage and 4073 carefully verified QA pairs. The annotation pipeline assures that only questions requiring true spatial-temporal retrieval and reasoning are retained, minimizing bias from language priors or static contexts.

Figure 3: Construction pipeline of LongSpace-Bench: collecting, filtering, and taxonomy-guided annotation ensures spatial memory and relational focus, excluding clips irrelevant to spatial reasoning.

The suite of QA types further operationalizes the taxonomy, offering dedicated probes for both short-term perception and truly long-range recall, such as memory of appearance order, navigation routes, or cross-room transitions.

Model: LongSpace Framework

Geometry-Aware Spatial Structure Perception

LongSpace augments standard MLLMs by explicitly integrating 3D geometry priors into the visual modality. A dedicated geometry encoder (e.g., $\pi^3$) processes local 3D cues (such as depth and layout) from sampled video frames, aligning them with vision transformer features and injecting the fused representation into the early decoder layers. This ensures that not only appearance but spatial structure is encoded throughout the model's computation.

Hierarchical KV Memory for Long-Video Organization

A central innovation is the Hierarchical KV Memory (HKM) subsystem, which enables scalable memory organization for long videos. Rather than maintaining all frame tokens in context (which is budget- and efficiency-prohibitive), LongSpace processes videos in temporally ordered chunks, compressing the token states into hierarchical layer-wise memory banks. These banks encapsulate evidence at multiple semantic levels:

• Sensory layers: Fine-grained spatial-visual token memory.
• Working layers: Short-term segment memory with binding for objects and local relations.
• Long-memory layers: Temporal anchors and global scene cues, retaining context across distant video segments.

The memory update mechanism employs a priority scoring scheme (feature norm, state change, temporal anchoring, recency), and comprises candidate selection, compression under budget constraints, and reordering by temporal position.

Figure 4: Overview of LongSpace: spatial structure perception fuses 2D appearance with 3D tokens, while hierarchical KV memory organizes evidence into multi-level roles for long-horizon reasoning.

Memory Retrieval and Decoding

For QA, the model encodes the question and performs a multi-level memory read. Sensory and working memory rely on top-$k$ sparse retrieval guided by question-relevance scoring; long-memory retrieval utilizes segment-level prototype matching for coarse-to-fine evidence localization. Retrieved KV states are appended as a frozen prefix during autoregressive decoding, thus avoiding recomputation and enabling efficient evidence utilization.

Figure 5: LongSpace visualizes relevant evidence localization—heatmaps highlight how question-relevant information is extracted from vast temporal windows.

Experimental Results and Analysis

Quantitative Performance

On LongSpace-Bench, LongSpace-9B achieves a state-of-the-art overall accuracy of 49.2 (compared to 46.5 for Qwen3-VL-32B and 45.3 for Gemini-3-Pro), with particularly notable strength in memory-centric tasks—Appearance Order (52.5), State Change (44.9), and Route Recall (51.8). In contrast, baseline MLLMs and geometry-aware competitors retain a marked performance deficit, particularly as video horizon increases. Even proprietary models like GPT-5 and Gemini-3-Pro, with substantially larger scaling, yield lower accuracy on explicitly long-horizon spatial memory tasks.

Figure 6: Comparative effect of different inference settings (uniform sampling, recent-window, long-memory) on accuracy across short, medium, and long video horizons.

Ablation studies show the necessity of both moderate-depth geometry injection (with optimal performance at 8 early decoder layers) and hierarchical/role-specific memory organization. Reducing memory capacity or omitting hierarchical roles rapidly degrades accuracy, demonstrating that memory structure and prioritization, not raw storage capacity, underpins long-range spatial recall.

Qualitative Task Coverage

LongSpace-Bench includes realistic QA instances spanning the full task taxonomy. The following figures illustrate key task categories:

Figure 7: Object Counting—quantitative perception with ambiguous and cluttered backgrounds.

Figure 8: Relative Orientation—understanding geometric relationships despite viewpoint changes.

Figure 9: Appearance Order—recall requiring retrieval of temporally dispersed events.

Figure 10: Route Planning—reasoning over multi-room navigation based on previously observed layouts.

Implications for Model Development and Future Directions

The findings indicate that explicit, hierarchical spatial memory markedly improves MLLM performance on long-horizon tasks. Merely increasing context length or relying on dense token representations is insufficient for robust spatial reasoning: structured memory organization, geometric feature fusion, and task-aware evidence prioritization are critical components. This framework sets a new benchmark for evaluating not only visual understanding but also memory and planning capabilities that are central for downstream applications such as autonomous navigation, embodied agents, and embodied QA systems operating in real and simulated environments.

Nevertheless, the research does have limitations. LongSpace-Bench is primarily constructed from indoor environments and passive room-tour videos, excluding the complexities of outdoor navigation, active manipulation, and dynamic multi-agent interaction. Future work should thus extend benchmarks to a broader domain of embodied experiences (e.g., outdoor navigation, active exploration) and refine memory architectures for even larger video scales and open-world generalization.

Conclusion

"LongSpace: Exploring Long-Horizon Spatial Memory from Perception to Recall in Video" (2606.05677) introduces the first unified framework and benchmark for evaluating long-horizon spatial memory in video-LLMs. The combination of explicit geometry-aware perception, hierarchical KV memory, and task-diverse QA evaluation substantiates the necessity of long-horizon spatial memory organization for next-generation MLLMs. This work provides a rigorous empirical foundation and methodological blueprint for future research at the intersection of perception, reasoning, and long-term memory in multimodal systems.