SpaceEra++: A Unified Framework Towards 3D Spatial Reasoning in Video

Abstract: Visual-spatial understanding, defined as the ability to infer object relationships and scene layouts from visual inputs, is fundamental to downstream tasks such as robotic navigation and embodied interaction. However, pre-trained vision-LLMs (VLMs) remain constrained by spatial uncertainty stemming from inherently 2D observations and by the scarcity of data for 3D spatial understanding. To address these limitations, we proposed a novel framework, SpaceEra, in the NeurIPS 2025 Spotlight paper. Although it achieved significant performance gains, we further observed that its effectiveness is hindered by insufficient input from scanning videos and weak reasoning constraints. To tackle these newly emerged challenges, we extend the original framework into a comprehensive system, termed SpaceEra++, which spans data construction, model design, training optimization, and prompting inference. Specifically, to alleviate input insufficiency, we introduce ScenePick, a frame sampling strategy that balances spatial coverage with object semantics to produce compact yet comprehensive scene representations. In addition, to enhance spatial reasoning, we develop SpaceAlign, which enforces pairwise object constraints by jointly exploiting absolute coordinates and relative spatial relations, thereby aligning optimization with spatial accuracy. Extensive experiments across multiple benchmarks demonstrate consistent improvements over strong baselines, while ablation studies validate both the individual and joint contributions of each component, and further analyses provide guidance for future research.

• The paper presents a unified framework integrating synthetic data, geometry-aware frame sampling, reinforcement learning, and stepwise prompting to improve 3D spatial reasoning in video.
• It leverages the large-scale ScanForgeQA dataset and ScenePick sampling to overcome data insufficiency and spatial ambiguity in multi-view video inputs.
• Empirical results demonstrate 4–8% accuracy gains across benchmarks while maintaining efficient inference, underscoring its potential for robotics, AR, and indoor navigation.

SpaceEra++: A Unified Framework Towards 3D Spatial Reasoning in Video

Introduction and Motivation

Visual-spatial reasoning in intelligent systems is essential for applications in robotics, autonomous navigation, and embodied AI. Human spatial understanding arises from inferring complex 3D relationships with limited 2D perceptual input; however, most Vision-LLMs (VLMs) trained on 2D imagery struggle with the inherent ambiguities and uncertainty when tasked with 3D spatial reasoning. While point-cloud-based methods provide geometric fidelity, their practical use is constrained by data collection difficulties and computational overhead. Vision-only solutions that operate on scanning videos or multi-view imagery present a scalable alternative but introduce challenges of spatial uncertainty and insufficient data diversity.

SpaceEra++ is introduced as a unified, end-to-end framework that addresses these challenges by integrating contributions at the data, model, optimization, and inference levels. It builds on the earlier SpaceEra framework, incorporating new mechanisms for frame selection and reinforcement learning, specifically targeting input sufficiency and biased spatial reasoning.

Framework Overview

The SpaceEra++ architecture comprises four synergistic components:

1. ScanForgeQA dataset – a large-scale, procedurally generated dataset for supervised training in 3D spatial QA tasks.
2. ScenePick sampling – an adaptive, geometry- and semantics-aware frame selection mechanism that maximizes both spatial coverage and question relevance.
3. SpaceAlign RL – a spatially constrained group reinforcement learning strategy enforcing both absolute and relative coordinate consistency.
4. SpatialMind prompting – a hierarchical inference decomposition to guide VLMs into stepwise 3D reasoning.

   Figure 1: Overview of the SpaceEra++ framework, including new content in data construction, input sampling, RL optimization, and prompting inference.

ScanForgeQA: Synthetic QA Dataset for 3D Spatial Reasoning

SpaceEra++ tackles data scarcity with the ScanForgeQA dataset, which contains over 34,000 single-room simulated scenes and nearly a million QA pairs, covering diverse categories such as attribute estimation, spatial reasoning, and hypothesis analysis. Scenes are sourced via both disassembly of richly annotated multi-room public datasets and large-scale LLM-driven synthesis, ensuring diversity and realism.

Scans are generated using two paradigms: orbit (360° panoramic sweeps) and navigation (egocentric traversals with procedural object-based starts and endpoints), emulating the human scene exploration process critical for spatial reasoning tasks.

Figure 2: The pipeline of ScanForgeQA data construction, encompassing scene construction, scan synthesis, and automated QA pair generation.

ScenePick: Geometry- and Semantics-Driven Frame Sampling

Given impractically long video sequences in scanning datasets and hard memory constraints on VLMs, uniform random sampling discards many spatial cues. ScenePick uses a learned scene-to-frame mapping: leveraging a visual geometry foundation model (VGGT) to reconstruct dense 3D point clouds and semantic grounding tools to score frame relevance for question-specific objects. The sampling algorithm selects frames that maximize both voxel-wise spatial coverage and focused representation of question-relevant objects, cast as a weighted maximum coverage optimization.

Figure 3: Illustration of the ScenePick frame sampling strategy, balancing spatial coverage and object relevance.

This strategy yields inputs with richer, less redundant spatial information and demonstrably bridges the input insufficiency gap for VLMs on spatial reasoning tasks.

Figure 4: Scene comparisons using different frame sampling methods; ScenePick yields more complete and detailed reconstructions, emphasizing essential objects.

SpaceAlign: Reinforcement Learning with Absolute and Relative Spatial Constraints

To reduce biased reasoning optimization, SpaceEra++ introduces SpaceAlign, a group RL algorithm extending GRPO. Prior RL approaches in VLMs largely train for discrete answer correctness or absolute coordinate estimation, largely ignoring the structured relationships among objects—for spatial reasoning, these relative inter-object cues are critical.

SpaceAlign incorporates multiple composite rewards:

• Format and task rewards ensure QA consistency and type correctness.
• Absolute coordinate reward targets direct object localization accuracy via grid maps.
• Relative relation reward measures pairwise geometric consistency—combining directional (cosine-based) and distance-ratio similarities.

This composite reward schema tightly couples per-object accuracy with overall scene layout preservation, resulting in policies that internalize both location and topology for more faithful spatial reasoning.

Figure 5: Reward curves demonstrating faster, more stable convergence and higher final rewards for SpaceAlign-trained models.

SpatialMind Prompting: Hierarchical Decomposition for Inference

SpatialMind enables structured, interpretable reasoning by decomposing both the scene and the query. Scene decomposition involves:

1. Extraction of per-frame object instances and their local coordinates.
2. Accumulation into a global 3D map via projection and transformation.
3. Generation of alternate scene representations (2D grids, textual descriptions).

Question decomposition instantiates stepwise reasoning templates—distinct for categories like relative distance, directionality, and containment—defined via LLM (e.g., GPT-4o) guidance or open-source alternatives.

Figure 6: The SpatialMind prompting strategy: scene decomposition and question decomposition facilitate interpretable, chain-of-thought spatial reasoning.

Empirical Evaluation

Extensive experiments are reported on multiple spatial benchmarks, including VSI-Bench, OpenEQA, ScanQA, and SQA3D, across proprietary and open-source VLM backbones. Strong numerical evidence is provided:

• Consistent improvements in average benchmark accuracy (+4–8%) across all model scales with ScenePick and SpaceAlign.
• Ablations demonstrate that combining both geometric (space coverage, absolute/relative rewards) and semantic (object-centric) cues yields maximal gains.
• Robustness to random seeds and prompting architectures is empirically verified.
• Inference efficiency is maintained or improved using techniques such as Flash Attention.

Qualitative analyses highlight correct spatial reasoning in complex, egocentric queries where baseline models fail.

Figure 7: Parameter analysis showing the sensitivity of framework performance to hyperparameters $\lambda$ (voxel size) and $M$ (grid size).

Figure 8: Example of the SpaceAlign reasoning process compared with previous methods, with accurate spatial layout and query response.

Implications and Future Directions

SpaceEra++ unifies advances at every stage—synthetic data design, input representation, multimodal learning, and inference guidance—to produce VLMs with enhanced 3D spatial understanding capabilities in video. The demonstrated improvements have immediate practical ramifications for indoor robotics, digital twins, augmented reality, and egocentric reasoning systems, especially in scenarios where 3D priors or real-world scans are inaccessible.

The integration of semantic-aware frame selection with RL-driven absolute and relational rewards supplies a generalizable blueprint for future frameworks requiring structural reasoning from sequential or partial observation. Promising future research includes generalizing the SpaceEra++ pipeline to multi-room, large-scale, or outdoor domains, and integrating richer interaction signals (e.g., agent action, language-based feedback) for embodied AI.

Conclusion

SpaceEra++ provides a comprehensive, modular pipeline for 3D spatial reasoning in video, resolving critical shortcomings in existing approaches through principled advances in data creation, input sampling, RL optimization, and inference decomposition. Experimental results validate that each component—ScenePick and SpaceAlign—delivers both independent and synergistic gains to VLMs' spatial perception capabilities, without compromising inference efficiency or reproducibility.

The modular, unified approach of SpaceEra++ will likely inform successive work on spatial and embodied AI, underpinning the next generation of VLMs capable of reliable, human-comparable spatial understanding from vision alone.