Spatial 3D-LLM: 3D-Aware Language Models
- Spatial 3D-LLM is a multimodal language model that integrates explicit 3D perception to produce geometry-aware outputs grounded in spatial relations.
- It employs diverse input modalities, including direct point clouds, multi-view images, and 2D data with geometric inference, to bolster spatial representation.
- Architectural innovations, specialized training strategies, and curated datasets drive improvements in tasks such as 3D captioning, QA, and spatial grounding.
Spatial 3D-LLM denotes a multimodal LLM endowed with explicit 3D perception and reasoning ability: it consumes 3D signals or 2D observations endowed with geometric grounding, learns spatial representations, and generates geometry-aware outputs that respect physical layout, object relations, and embodied context. In the literature, related formulations appear as “3D-LLM,” “Spatial 3D-LLM,” “SpatialLLM,” and “Spatial-MLLM,” reflecting a shared objective—injecting the 3D world into LLMs—while differing in input modality, alignment strategy, and task emphasis (Hong et al., 2023, Wang et al., 22 Jul 2025, Ma et al., 1 May 2025, Wu et al., 29 May 2025).
1. Conceptual scope and emergence
The 2023 paper “3D-LLM: Injecting the 3D World into LLMs” defines a family of models that can take 3D point clouds and their features as input and perform captioning, dense captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, and navigation (Hong et al., 2023). Its central claim is that LLMs and VLMs are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics, and layout, and that these capabilities can be introduced by aligning 3D features to language-space backbones.
Subsequent work makes the spatial emphasis more explicit. “Spatial 3D-LLM: Exploring Spatial Awareness in 3D Vision-LLMs” defines a 3D multimodal LLM designed to improve spatial awareness in complex 3D scenes by enriching the scene’s spatial embeddings and feeding them as visual prompts to an LLM backbone (Wang et al., 22 Jul 2025). In parallel, “SpatialLLM” studies the impact of 3D-informed data, architecture, and training setups for spatially-intelligent large multimodal models operating on real images, while “Spatial-MLLM” targets visual-based spatial reasoning from purely 2D observations (Ma et al., 1 May 2025, Wu et al., 29 May 2025).
A recurrent misconception is that Spatial 3D-LLM denotes only a point-cloud model. The survey “How to Enable LLM with 3D Capacity? A Survey of Spatial Reasoning in LLM” instead proposes a taxonomy that categorizes existing methods into three branches: image-based methods deriving 3D understanding from 2D visual data, point cloud-based methods working directly with 3D representations, and hybrid modality-based methods combining multiple data streams (Zha et al., 8 Apr 2025). This suggests that the field is best understood as a design space rather than a single architecture.
2. Spatial representations and input modalities
Spatial 3D-LLMs operate over several representational regimes. Point-cloud systems process unordered 3D points directly; multi-view systems render or observe a scene from several cameras and lift features into a global frame; video-based systems treat a 3D scene as a dynamic video stream augmented with coordinates; situated systems bind all relations to an agent-centric pose; and 2D-only systems attempt to recover 3D structure through geometric priors (Zha et al., 8 Apr 2025, Zheng et al., 2024, Zhang et al., 2024).
| Branch | Representative formulation | Spatial signal |
|---|---|---|
| Image-based | Multi-view, RGB-D, monocular | 3D inferred from 2D visual data |
| Point cloud-based | Direct 3D representations | Native geometry from points or objects |
| Hybrid modality-based | Multiple data streams | Combined spatial and semantic evidence |
A canonical geometric relation across these systems is the pinhole camera model. “3D-LLM” writes the projection of a homogeneous 3D point to the image plane as
while “Video-3D LLM” back-projects RGB-D pixels into camera and then global coordinates before encoding them as position-aware video tokens (Hong et al., 2023, Zheng et al., 2024). This is not a mere preprocessing detail: it determines whether patch, object, or scene tokens are indexed in a shared world frame or remain tied to view-local image coordinates.
Situated variants replace world-centric reasoning with agent-centric reasoning. “SPARTUN3D” defines the agent’s situation as , and maps global points into the local frame by
The resulting scene graph stores agent-centric directions, distances, angle bins, and pass-by objects, allowing left/right/front/backward to change with the agent’s position and orientation rather than with a fixed global frame (Zhang et al., 2024).
The choice of input modality has direct consequences for what kind of spatial reasoning is natural. Point-cloud systems preserve metric geometry; RGB-D and multi-view systems recover scene layout through calibrated projection and fusion; monocular and 2D-only systems trade direct geometry for learned structure priors and inference-time heuristics. This suggests that “spatial awareness” is not a single capability but a compound property arising from coordinate choice, tokenization, and cross-view consistency.
3. Architectural mechanisms for spatial awareness
A common architectural pattern is encoder–projector–LLM alignment. “3D-LLM” renders multi-view images, extracts language-aligned visual features, constructs 3D features, and feeds arbitrarily many 3D tokens into a Perceiver/Resampler or Q-Former, while adding a 3D localization mechanism based on position embeddings and location tokens representing axis-aligned bounding boxes (Hong et al., 2023). The localization component is central: it allows the model to output discrete 3D location tokens rather than only descriptive language.
“Spatial 3D-LLM” introduces a progressive spatial awareness scheme that expands the perception field across three stages: intra-referent, inter-referent, and referent–scene contextual interactions. Intra-Referent uses cluster abstraction and FFN-based voting to refine proto-object centers; Inter-Referent applies GCN-based message passing over visual referents; Contextual Interactions apply stacked self-attention and cross-attention layers between referents and the global scene, followed by Refine-Location with center and pairwise spatial constraint losses (Wang et al., 22 Jul 2025). The architectural emphasis is not on a single spatial token but on progressively mining local, relational, and contextual structure.
By contrast, “SpatialLLM” is explicit that it does not introduce custom 3D tokenizers or positional embeddings. Instead, it uses mixed visual encoders—CLIP and DINOv2—to form hybrid feature maps, a standard MLP/projector, and standard LLaVA-style cross-attention, while injecting depth, azimuth/elevation, and pairwise distance supervision into multimodal alignment and instruction tuning (Ma et al., 1 May 2025). This design argues that substantial 3D spatial reasoning can emerge from 3D-informed supervision without explicit geometric modules.
“Video-3D LLM” places the spatial signal directly into visual tokens through additive 3D positional encoding: . Its 3D-PE is computed from pooled patch-level global coordinates and yields a position-aware video representation aligned across viewpoints and time (Zheng et al., 2024). “3D-SLIM” targets a different bottleneck: decoder masking. It replaces the causal mask with a Geometry-adaptive Mask that constrains object-object attention based on spatial density and an Instruction-aware Mask that allows object tokens to directly access instruction context, while adding no extra parameters (Jeon et al., 2 Dec 2025).
A separate line of work adds explicit latent reasoning structure. “SSR3D-LLM” introduces latent spatial reasoning steps and memory tokens for grounding, and a geometry-aware scorer reads these latent steps in order to refine candidate rankings step by step with step-length masking (Li et al., 27 May 2026). This suggests a more general architectural split within the field: some models inject geometry into token representations, some into attention topology, and some into the inference interface itself.
4. Data construction and training strategies
The field is strongly shaped by dataset engineering. “3D-LLM” collects over 300k 3D-language data using three prompting mechanisms: Boxes–demonstration–instruction prompting, ChatCaptioner-based prompting, and revision-based prompting, spanning captioning, dense captioning, QA, grounding, dialog, navigation, and task decomposition (Hong et al., 2023). The model is then pretrained on mixed held-in task datasets and finetuned for held-out benchmarks such as ScanQA, 3DMV-VQA, ScanRefer, and navigation.
“SpatialLLM” makes data design the central variable. It introduces 3DI-Pb-IN166K, 3DI-Pb-OI1M, and 3DI-Ft-1M, alongside SpatialVQA with 1,323 questions. Training follows a staged pipeline: optional Stage 0 pre-pretraining of the vision encoder, Stage 1 3D-informed multimodal alignment using CC558K plus 3DI-Pb-IN166K, and Stage 2 3D-informed instruction tuning on Mix665K plus 3DI-Ft-1M (Ma et al., 1 May 2025). The reported ablations attribute the largest single gain to Stage 2 instruction tuning, with further gain from Stage 1 alignment and a negative result for Stage 0 LoRA-based vision pre-pretraining.
“Spatial 3D-LLM” adds new task-specific supervision through the MODLE dataset, which contains 263K VL annotations across ScanNet-derived scenes and supports 3D object distance measurement, 3D object movement, and 3D object placement (Wang et al., 22 Jul 2025). Its joint loss combines the LLM instruction-following objective with pairwise spatial constraint loss and center loss, explicitly coupling language supervision to referent localization quality.
Other datasets alter the problem framing rather than only scale. “SPARTUN3D” contributes approximately 133k examples for situated captioning and situated QA, with Spa-prompt substantially improving valid outputs over Cord-prompt in human validation (Zhang et al., 2024). “MM-Spatial” introduces CA-VQA with ~10M QA pairs over 220K frames from 2K videos for supervised fine-tuning and an evaluation benchmark with ~62K QA pairs over 2.6K frames from 265 videos, using blind filtering to reduce language priors (Daxberger et al., 17 Mar 2025). “Spatial-MLLM” constructs Spatial-MLLM-120k and trains with supervised fine-tuning and GRPO for long chain-of-thought spatial reasoning (Wu et al., 29 May 2025).
Across these systems, a consistent pattern emerges: high-quality spatial behavior is usually tied to synthetic or semi-automatic 3D-language corpora, staged alignment, and explicit task mixtures rather than to a single generic instruction dataset.
5. Tasks, benchmarks, empirical results, and applications
The task envelope of Spatial 3D-LLM is broad. Core tasks include 3D captioning, dense captioning, 3D QA, task decomposition, 3D grounding, 3D-assisted dialog, navigation, embodied planning, situated QA, open-vocabulary grounding, metric distance estimation, and 3D layout editing (Hong et al., 2023, Wang et al., 22 Jul 2025, Zhang et al., 2024, Liu et al., 9 Jul 2025). A more stringent formulation is contextual object localization and disambiguation, where the generated instruction must be target-exclusive rather than merely descriptive (Chang et al., 2024).
Representative benchmarks show that gains are usually largest on tasks that explicitly require geometry. “SpatialLLM” reports 62.7% average accuracy on SpatialVQA, compared with GPT-4o at 54.0 and SpatialVLM-13B at 52.2, with particularly strong gains on 3D distance-based reasoning (Ma et al., 1 May 2025). “Spatial 3D-LLM” reports Scan2Cap CIDEr 72.2, ScanQA CIDEr 82.5, SQA3D EM@1 46.2, ScanRefer [email protected]/0.5 of 44.3/37.2, and Multi3DRefer [email protected]/0.5 of 48.3/41.2 (Wang et al., 22 Jul 2025). “Video-3D LLM” reports ScanRefer [email protected]/0.5 of 58.12%/51.72%, Multi3DRefer [email protected]/0.5 of 58.0%/52.7%, Scan2Cap [email protected] of 83.77, ScanQA CIDEr 102.06 with EM 30.10, and SQA3D EM 58.6 (Zheng et al., 2024).
Evaluation itself is contested. “3D Spatial Understanding in MLLMs: Disambiguation and Evaluation” argues that n-gram metrics can rate “surface similarity” but fail to test disambiguation or spatial correctness, and formalizes exclusivity through a grounding scorer that must uniquely select the target with a margin (Chang et al., 2024). “3D-SLIM” further shows that decoder masking alone can shift performance across grounding, captioning, and QA, with gains such as ScanRefer [email protected] 55.5 → 59.6 and Multi3DRefer [email protected] 57.1 → 63.7 for Chat-Scene under the proposed mask (Jeon et al., 2 Dec 2025). These results imply that benchmark scores are sensitive not only to data and encoders, but also to representational assumptions embedded in decoding.
The application surface extends beyond understanding benchmarks. “SceneCraft” models a scene graph as a blueprint detailing the spatial relationships among assets in a scene and converts text descriptions into Blender-executable Python scripts (Hu et al., 2024). “Intelligent Co-Design” uses Reference, Spatial, Interactive, and Grader agents to turn natural-language intent and reference images into structured spatial rules and optimized 3D indoor design (Lim et al., 16 Mar 2026). “ShapeCraft” represents 3D assets as a Graph-based Procedural Shape and uses LLM agents to generate structured, textured, and interactive 3D assets (Zhang et al., 20 Oct 2025). “3DrawAgent” makes a frozen LLM a training-free 3D drawing agent that sequentially draws 3D Bézier curves under geometric feedback (Xiao et al., 9 Apr 2026). This suggests that Spatial 3D-LLM is becoming as much a programmatic interface for geometry as a question-answering model.
6. Limitations, controversies, and future directions
Despite rapid progress, several limitations recur across the literature. Data scarcity and computational challenges remain central in both surveys (Zha et al., 8 Apr 2025, Ma et al., 2024). “3D-LLM” notes rendering dependency, reliance on camera parameters, and degradation under extremely occluded or sparse point clouds (Hong et al., 2023). “SpatialLLM” reports failure modes for heavily occluded or non-rigid objects, rare poses, and edge cases where monocular calibration is unreliable (Ma et al., 1 May 2025). “Video-3D LLM” assumes correct intrinsics/extrinsics and static scenes, and identifies multi-target disambiguation as a remaining difficulty (Zheng et al., 2024). “Spatial 3D-LLM” is limited primarily to point clouds and does not model editing realism constraints such as physics, collision, or ergonomics (Wang et al., 22 Jul 2025).
A second controversy concerns what should count as genuine spatial reasoning. Some systems rely on explicit geometric modules, scene graphs, or discrete location tokens; others report strong results by keeping the architecture close to standard multimodal LLM design and moving the 3D signal into supervision or masking (Ma et al., 1 May 2025, Jeon et al., 2 Dec 2025). This suggests that the field has not converged on a single necessary inductive bias. It has, however, converged on the view that naive 2D-biased pipelines are insufficient for tasks requiring 3D distance, orientation, compound spatial relations, or situated viewpoint changes.
Future directions are unusually consistent across papers. “SpatialLLM” identifies opportunities in explicit geometric modules, 3D positional encodings, multi-view consistency, sensor fusion, real-time inference, and continuous 3D scene memory (Ma et al., 1 May 2025). “Spatial 3D-LLM” points to broader modality coverage, dynamic scenes, physics-aware editing, open-vocabulary spatial reasoning, and real-time performance improvements for robotics and AR (Wang et al., 22 Jul 2025). The surveys emphasize better 3D tokenization, uncertainty-aware tokens, spatially aware prompting and tool-use, simulation-driven pretraining, stronger standardized benchmarks, and safety- and uncertainty-aware embodied deployment (Zha et al., 8 Apr 2025, Ma et al., 2024).
Taken together, these works define Spatial 3D-LLM not as a narrow model class but as an active research program: aligning LLMs with geometry, viewpoint, and interaction so that spatial understanding can be expressed as grounded language, structured coordinates, executable code, or embodied action.