SpatialSV: Internalizing Interpretable 3D Spatial Awareness in MLLMs via Task-Oriented Visual Supervision

Abstract: Unlocking the spatial intelligence of multimodal LLM (MLLMs) is crucial for understanding and interacting with the 3D world. Prevailing approaches typically inject spatial priors via external tools, which impose significant inference overhead, or rely on latent feature distillation, which remains uninterpretable and lacks fine-grained geometric constraints. To address these issues, we propose SpatialSV, a framework designed to internalize robust 3D spatial awareness within MLLMs while simultaneously offering inherent interpretability. Deviating from passive feature imitation, SpatialSV employs task-oriented visual supervision, compelling the model to actively lift its 2D visual features into explicit 3D representations, including depth maps, camera poses, and point clouds. Crucially, this 2D-to-3D lifting process provides a transparent window into the model's representations: the resulting 3D reconstructions serve as an intuitive proxy for visualizing and diagnosing the quality of the model's intrinsic spatial knowledge. Extensive experiments across multiple models and benchmarks demonstrate the effectiveness of SpatialSV in enhancing and interpreting MLLMs' spatial intelligence. Furthermore, the framework exhibits strong generalization in semi-supervised settings, validating its potential to leverage unlabeled visual data for scalable, interpretable spatial representation learning.

• The paper presents a framework that internalizes interpretable 3D spatial awareness in MLLMs through task-oriented visual supervision, enhancing spatial reasoning.
• It employs a multi-layer 2D-to-3D lifting pipeline with decoupled projectors and task-specific modules to align hidden features with explicit 3D spatial annotations.
• Empirical results show significant improvements in spatial tasks and model interpretability, validated by ablation studies and benchmark analyses.

SpatialSV: Internalizing Interpretable 3D Spatial Awareness in MLLMs via Task-Oriented Visual Supervision

Motivation and Context

Spatial intelligence in Multimodal LLMs (MLLMs) underpins applications requiring understanding and reasoning about 3D physical environments. Existing paradigms predominantly augment MLLMs with external spatial priors—such as depth maps, point clouds, or spatial prompts generated by auxiliary models—to compensate for their lack of intrinsic spatial representations. However, these approaches are hindered by dependence on upstream errors, increased inference latency, and limited interpretability of internal representations. Feature-level distillation from 3D vision foundation models (VFMs) remains coarse-grained, resulting in suboptimal spatial mental modeling.

SpatialSV directly addresses these deficits, proposing a framework for internalizing robust, interpretable 3D spatial awareness into MLLMs via explicit, task-oriented visual supervision. By aligning multi-layer hidden visual features with explicit 3D spatial representations, SpatialSV enables fine-grained spatial constraints and transparent diagnosis of spatial knowledge embedded in the MLLM.

Figure 1: (a) Conventional methods depend on external spatial priors, inducing inefficiency and fragility. (b) SpatialSV integrates interpretable 3D spatial awareness intrinsically into MLLMs via task-oriented visual supervision.

Methodology

SpatialSV establishes a multi-task, multi-layer 2D-to-3D lifting pipeline. The framework extracts visual features from multiple layers of the MLLM, projects them into a shared 3D latent space using decoupled projectors, and applies task-specific DPT modules to predict depth maps, ray maps (encoding camera pose and ray directions), and point cloud maps (encoding 3D spatial coordinates). This process is supervised by explicit 3D annotations sourced from off-the-shelf VFMs, ensuring fine-grained geometric alignment.

The training objective is a composite of autoregressive text prediction, feature distillation, and three task-oriented 3D spatial losses (depth, ray, and point cloud), reinforcing both coarse-level guidance and precise geometric constraints:

Figure 2: SpatialSV schematic: Multi-layer visual feature lifting and alignment to explicit 3D spatial tasks enable robust spatial representation learning and interpretability.

SpatialSV's design exploits layer-wise semantic complementarity, capturing both low-level visual and high-level cross-modal features. Ablation studies confirm that combining supervision across multiple layers with decoupled projectors maximizes performance, underscoring the necessity of both spatial and semantic diversity in internal representation learning.

Spatial Representation Probing and Interpretability

A systematic probe-based analysis quantifies the correlation between internal spatial representation quality and spatial intelligence. MLLMs, post fine-tuning with SpatialSV, exhibit depth estimation maps with reduced RMSE and increased geometric fidelity compared to feature-distillation or pure text-supervised variants, directly reflecting higher spatial reasoning accuracy across benchmarks.

Figure 3: Quantitative and qualitative depth probing: Greater spatial intelligence correlates with improved internal representation quality and more faithful 3D reconstructions.

Moreover, the intrinsic interpretability of SpatialSV derives from its task-oriented supervision. The resulting 3D reconstructions serve as intuitive proxies for the spatial knowledge encoded in the model, enabling transparent diagnosis and capability boundary assessment. This interpretability is empirically validated by strong correlations between depth prediction quality and both intra-model spatial intelligence and model-specific sample difficulty.

Figure 4: Correlation between lifted 3D representation quality and spatial intelligence (VQA accuracy) within each MLLM.

Figure 5: Sample binning by 3D lifting quality reveals higher QA accuracy for samples with faithful spatial representations.

Experimental Results

SpatialSV consistently advances spatial intelligence across model families and datasets. On MindCube-Tiny and VSI-Bench, SpatialSV achieves absolute gains of $3.42\%$–$12.66\%$ over text-supervised variants and $2.97\%$–$7.81\%$ over distillation-based variants, with pronounced improvements in spatial reasoning tasks such as route planning. The framework demonstrates robust cross-model and cross-dataset generalization, with superior performance on Ego3D-Bench, Spatial457, ViewSpatial-Bench, 3DSR-Bench, SP-Bench, TopViewRS, CVBench, and MMBench.

Qualitative evaluations on MindCube-Tiny confirm that SpatialSV enabled MLLMs successfully capture essential spatial semantics (e.g., occlusions, object locations), leading to correct answers on complex spatial questions.

Figure 6: MindCube-Tiny qualitative: Successful retrieval of spatial object relationships in the predicted maps.

Figure 7: Example 1: SpatialSV MLLM correctly reconstructs scene geometry and answers.

Figure 8: Example 2: Absent spatial features cause failure; presence yields correct reasoning.

Figure 9: Example 3: Model's spatial ignorance leads to incorrect answers.

Semi-supervised experiments show that SpatialSV leverages unlabeled visual data effectively: with $50\%$ textual annotations, performance approaches fully supervised settings ($53.9$ vs $55.3$ overall accuracy on MindCube-Tiny), validating the scalability of spatial representation learning via visual supervision alone.

Implications and Future Directions

SpatialSV robustly enables interpretable 3D spatial awareness in MLLMs without reliance on external spatial priors or costly inference pipelines, thus supporting deployment in real-time, resource-constrained, or embodied AI scenarios. The explicit, interpretable spatial representations facilitate diagnosis, sample difficulty estimation, and model capability boundary assessment.

The fine-grained, task-oriented supervision paradigm introduced by SpatialSV is theoretically significant, advancing the understanding of spatial mental modeling in autoregressive multimodal architectures. Practically, the framework's generalization and semi-supervised capacity open avenues for large-scale spatial intelligence pretraining with minimal annotation overhead.

Future studies may explore extending SpatialSV to broader spatial-temporal modeling, integrating reinforcement or active learning for dynamic spatial reasoning, and deploying in embodied agents with online adaptation to novel environments.

Conclusion

SpatialSV provides a principled framework for internalizing robust and interpretable 3D spatial awareness in MLLMs via task-oriented visual supervision. It establishes a clear correlation between spatial representation quality and reasoning performance, delivers strong empirical improvements across diverse benchmarks, and introduces a transparent, scalable methodology for spatial intelligence modeling in multimodal systems.