VLM-3D: Vision-Language Models in 3D

• VLM-3D is a research field that integrates vision-language models with explicit 3D representations and geometric priors to solve spatial tasks beyond single-image semantics.
• It encompasses diverse applications including 3D visual grounding, semantic scene understanding, text-to-3D generation, robotic manipulation, and volumetric medical imaging.
• Methodologies range from multi-view calibrations with panoramic renderings and voxel-space fusion to closed-loop agentic controls that iteratively refine spatial predictions.

Searching arXiv for papers on VLM-3D and closely related terminology.

VLM-3D is an umbrella designation for research that couples vision-LLMs with explicit 3D representations, geometric priors, or multi-view calibration in order to solve tasks that require spatial reasoning beyond single-image semantics. In the cited literature, the term spans 3D visual grounding, semantic occupancy prediction, scene-centric and object-centric 3D scene understanding, text-to-3D generation, agentic scene generation, robotic manipulation, and volumetric medical imaging. This suggests that VLM-3D denotes a family of interfaces between pretrained VLMs and 3D substrates—panoramic equirectangular renderings, voxel grids, Gaussian splats, NeRF fields, point clouds, and calibrated multi-view image sets—rather than a single model class (Jung et al., 24 Dec 2025, Doruk et al., 3 Mar 2026, Halacheva et al., 1 Jul 2025, Petit et al., 20 Nov 2025).

1. Scope and conceptualization

The literature uses VLM-3D in several closely related senses. In 3D visual grounding, the emphasis is on transferring pretrained 2D VLM reasoning to 3D localization by introducing an explicit 2D–3D interface. PanoGrounder, for example, renders 3D scenes into multi-modal panoramas so that a pretrained CogVLM-17B can ground a query per panorama and lift detections back into a 3D box (Jung et al., 24 Dec 2025). SeqVLM treats zero-shot 3D grounding as proposal-guided reasoning over real multi-view image sequences, using Mask3D proposals, CLIP-based semantic filtering, and iterative VLM selection (Lin et al., 28 Aug 2025).

A second usage centers on scene representations that are already 3D-native. GaussianVLM associates language with each Gaussian primitive in a 3D Gaussian splat scene, then sparsifies the resulting dense language-aligned representation into task-aware global and local tokens for a frozen LLM with LoRA adapters (Halacheva et al., 1 Jul 2025). LLaVA$^3$ instead builds an intermediate NeRF and distills LLaVA-aligned and SAM–CLIP feature fields into object-centric omnidirectional token sets, which are then consumed by a frozen LLaVA variant without fine-tuning (Petit et al., 20 Nov 2025).

A third usage treats VLM-3D as a claim about generic multimodal models themselves. VLM3 argues that vision-LLMs are “native 3D learners” once focal length unification, text-based pixel reference, and data mixture and scaling are introduced, and it applies plain next-token supervision to metric depth, pixel correspondence, camera pose estimation, and object-level 3D understanding (Cai et al., 28 May 2026). This claim contrasts with other works that retain stronger geometric scaffolding, but it remains part of the same research trajectory.

2. Representational interfaces between language and 3D structure

A central design question in VLM-3D is how to expose 3D structure to models that were largely pretrained on 2D image–text corpora. PanoGrounder uses equirectangular panoramic renderings covering $360^\circ$ horizontally and $180^\circ$ vertically, augmented with range/depth and multi-view fused semantic features. Its multi-modal feature adapter injects geometry into mid-level VLM layers and semantics into late layers through zero-initialized residual modules, preserving pretrained behavior at initialization (Jung et al., 24 Dec 2025). SeqVLM preserves real-scene appearance rather than rendering synthetic views: it projects semantically filtered 3D proposals onto real image sequences, validates visibility with depth consistency $|D(u,v)-Z_c|\le \tau$, expands the projected box by $\alpha=0.25$, and stitches the selected views into a sequence for VLM reasoning (Lin et al., 28 Aug 2025).

Voxel-space methods expose geometry differently. VLMFusionOcc3D projects multi-view images and LiDAR point clouds into a unified voxel space, injects LoRA-adapted CLIP text embeddings into camera and LiDAR voxel volumes through gated cross-attention, and adaptively fuses them with weather-conditioned gating (Doruk et al., 3 Mar 2026). WeatherOcc3D likewise uses a CLIP-based environmental embedding to generate channel-wise gating masks and a global fusion scalar $w_{env}$, with prompts such as “clear day” and “rainy night” selecting the environmental regime (Doruk et al., 15 May 2026). VISA takes a different position: it does not use a VLM at inference at all, but uses offline instance audits from a VLM to supervise voxel semantics during training through taxonomy, attribute, and audit-graph losses (Xian et al., 11 Jun 2026).

Scene-centric 3D representations occupy another branch of the design space. GaussianVLM relies on SceneSplat to predict a SigLIP2 language feature for every Gaussian primitive and then compresses tens of thousands of per-splat features into 128 task-selected scene tokens and 4 ROI tokens (Halacheva et al., 1 Jul 2025). LLaVA$^3$ learns view-independent and view-dependent token fields in a NeRF, segments the scene into an object/part/subpart hierarchy, and packages per-object token sets as “virtual images” ordered by a radar sweep (Petit et al., 20 Nov 2025). Agentic 3D scene generation with spatially contextualized VLMs introduces a different interface again: a scene portrait, a semantically labeled point cloud, and a scene hypergraph together serve as a continually updated geometry-aware working memory for a GPT-4o-based agent (Liu et al., 26 May 2025).

3. Major task families and representative results

3D visual grounding remains one of the most prominent VLM-3D tasks. PanoGrounder defines the problem as predicting a 3D bounding box $B=(c,s,R)$ from a renderable 3D scene and a free-form text query, and reports state-of-the-art performance on ScanRefer and Nr3D, including 61.0 [email protected] IoU on ScanRefer and 74.6 Top-1 on Nr3D under single-dataset training; mixed ScanRefer+ReferIt3D training raises these to 62.0 and 76.1, respectively (Jung et al., 24 Dec 2025). SeqVLM addresses the same task in a zero-shot setting and reports 55.6 [email protected] and 49.6 [email protected] on ScanRefer, plus 53.2 [email protected] on Nr3D (Lin et al., 28 Aug 2025).

Dense scene prediction is another major family. VLMFusionOcc3D reports improvements over strong voxel baselines on nuScenes/OpenOccupancy and SemanticKITTI, including OccMamba mIoU increasing from 25.2% to 26.6% on nuScenes and from 24.6% to 26.4% on SemanticKITTI; rainy and night conditions show especially large gains (Doruk et al., 3 Mar 2026). WeatherOcc3D targets the same modality-trust problem and reports mIoU 26.3 on OccMamba and 21.1 on M-CONet, with rainy and night improvements of +3.2 and +3.9 mIoU for OccMamba (Doruk et al., 15 May 2026). VISA focuses on closed-set occupancy world models and improves OccWorld from 19.06 to 20.05 mIoU and GaussianWorld from 21.36 to 21.91 mIoU, while raising GaussianWorld rare-class mIoU from 15.60 to 16.79 (Xian et al., 11 Jun 2026).

Spatial reasoning and 3D scene understanding form a broader cluster. GaussianVLM improves scene-centric and object-centric performance under LL3DA and LEO protocols, including strong out-of-domain gains on ScanNet++ object-count QA, where accuracy rises from 4.2% for LL3DA to 24.1% (Halacheva et al., 1 Jul 2025). LLaVA$^3$ reports zero-shot ScanQA CIDEr 77.69 and EM@1 26.00, and improves 3D grounding over LeRF, OpenNeRF, and ConceptGraph on Sr3D+ and Nr3D (Petit et al., 20 Nov 2025). SpatialStack-5B reports 67.5 average on VSI-Bench and 85.5 average on CV-Bench, while preserving general multimodal capability on MMBench, Video-MME, and TempCompass (Zhang et al., 28 Mar 2026). VLM3 extends the task family further, reaching $\delta_1=0.904$ for depth, average AUC@30° of 94.0 for camera pose estimation, and 91.35% qualitative accuracy on SpatialRGPT-Bench object-level 3D understanding (Cai et al., 28 May 2026).

Generation tasks show a parallel development. VLM3D introduces differentiable semantic and spatial rewards inside score distillation sampling for text-to-3D generation and obtains the highest GPTeval3D Elo scores on all six reported metrics, including Alignment 1365.5 and Overall 1268.6 (Bai et al., 19 Sep 2025). Agentic 3D scene generation with spatially contextualized VLMs uses a scene portrait, point cloud, and hypergraph to generate, verify, and ergonomically optimize environments, reporting CLIP 0.385, BLIP 0.737, LPIPS 0.571, and top-ranked aesthetic quality and functional plausibility in GPT-4o evaluation (Liu et al., 26 May 2025).

Robotics and domain-specific perception also fall within the same umbrella. A monocular HRI system based on LLaVA-v1.5 7B, QLoRA, and a custom regression head predicts object positions in the robot base frame with median MAE of 13 mm and median Euclidean position error of 27 mm (Wahl et al., 1 Mar 2026). A zero-shot dexterous manipulation pipeline uses multi-view VLM reasoning, triangulation, reference-view ray voting, affordance-region generation, and closed-loop replanning to improve 3D grounding accuracy and execution reliability over single-view RGB-D grounding and fine-tuned VLA baselines (Kim et al., 17 Jun 2026). In medical imaging, Med3DVLM reaches 61.00% R@1 on 2,000-sample image-text retrieval, 36.42% METEOR on report generation, and 79.95% accuracy on closed-ended VQA for 3D scans (Xin et al., 25 Mar 2025).

4. Training regimes, supervision strategies, and system control

VLM-3D systems vary sharply in how much of the underlying model stack is trained. PanoGrounder fine-tunes CogVLM with LoRA and lightweight geometry/semantic adapters, avoiding any dedicated 3D detector or 3D box regression head; coordinate prediction is autoregressive over normalized digits and supervised by digit-wise cross-entropy plus an Earth Mover’s Distance loss that encodes numerical distance (Jung et al., 24 Dec 2025). VLMFusionOcc3D keeps the CLIP image tower frozen, inserts LoRA into the CLIP text encoder, and supervises the fused occupancy model with cross-entropy, Lovász-Softmax, and the DAGA alignment loss (Doruk et al., 3 Mar 2026). WeatherOcc3D similarly keeps the CLIP text encoder frozen except for LoRA adapters and trains only the gating/fusion layers and weather heads (Doruk et al., 15 May 2026).

Other works minimize architectural intervention. VLM3 explicitly avoids architecture changes, regression losses, and heavy augmentations, using standard next-token cross-entropy for depth numbers, correspondence coordinates, unit vectors, and Euler angles (Cai et al., 28 May 2026). VISA keeps the occupancy world model unchanged at inference and treats VLM outputs as training-time audits rather than persistent model inputs, which is a distinct form of distillation (Xian et al., 11 Jun 2026). The 3D object annotation pipeline of “Leveraging VLM-Based Pipelines to Annotate 3D Objects” does not train a new 3D model at all; instead, it aggregates per-view joint image–text likelihoods with log-sum-exp, producing calibrated distributions over open-vocabulary object labels and material classes for 763,844 Objaverse objects (Kabra et al., 2023).

Closed-loop agentic control appears when the output must remain physically executable or structurally coherent. SpatialAct isolates this issue experimentally, defining Multi-turn Interactive Refinement and Single-step Error Detection and Fix in simulator-grounded 3D scenes, and shows that models that perform well on isolated spatial tasks still struggle to maintain coherent spatial beliefs across multi-turn feedback (Liu et al., 29 May 2026). The zero-shot dexterous manipulation system operationalizes a corrective loop: after each primitive, a VLM decides whether to continue, retry, replan, or terminate based on fresh multi-view observations (Kim et al., 17 Jun 2026). Agentic scene generation uses an analogous generate–render–analyze–refine loop over Blender code and a scene hypergraph, with automatic verification before ergonomic adjustment (Liu et al., 26 May 2025). This suggests that a substantial portion of recent VLM-3D work is moving from static prediction toward iterative state updating and execution monitoring.

5. Evaluation, benchmarks, and validity

Evaluation has become a research problem in its own right. ReVSI argues that many prior spatial-intelligence benchmarks are invalid under modern VLM settings because point-cloud-derived annotations may drift from what is visible in video, and because models are often evaluated on question–answer pairs that are unanswerable under sparse frame budgets. ReVSI re-annotates 381 scenes from ScanNetv2, ScanNet++, ARKitScenes, 3RScan, and MultiScan; regenerates QA pairs under frame budgets of 16, 32, 64, and all frames; and formalizes answerability through object visibility metadata (Zhang et al., 27 Apr 2026). The benchmark’s dummy-video stress tests further expose hallucination and prior-driven behavior in general and specialized VLMs.

SpatialAct provides a complementary diagnosis for action-conditioned reasoning. It contains 333 scenes and 4,355 QA pairs across Abstract Geometric, Urban Architectural, and Indoor Scenes, and reports that the best model, Gemini-3.1 Pro, reaches Repair Rate 0.411 and Scene Success Rate 0.206 in Multi-turn Interactive Refinement, whereas humans reach 0.911 and 0.763 (Liu et al., 29 May 2026). The benchmark identifies a reasoning-to-action gap: good scores on object meaning, spatial relation, spatial orientation, mental rotation, and spatial visualization do not translate into reliable multi-turn repair.

Evaluation challenges are equally visible in 3D generation. A cross-model VLM-judge protocol for single-image mesh quality fixes a 24-view render rig, uses Qwen2.5-VL-7B-Instruct and InternVL3-8B as independent judges, and applies a mandatory position-bias correction by querying both A/B orders and keeping only order-consistent verdicts. Under this protocol, cross-model agreement reaches 0.83 with Cohen’s $360^\circ$0, while geometry validity is only 0.62 and render-space CLIP is 0.48, i.e., chance-level (Asaria et al., 16 Jun 2026). The paper’s conclusion is not that all proxies fail universally, but that the cheap geometry and render-CLIP proxies tested there do not substitute for the VLM-judge protocol under the stated conditions.

6. Limitations, misconceptions, and future directions

A recurring misconception is that stronger language grounding automatically produces stronger 3D reasoning. Several papers argue against that simplification. VISA shows that improving text-space similarity between 3D features and caption embeddings does not reliably improve closed-set occupancy mIoU, and concludes that VLMs are better suited to closed-set occupancy as reliability-aware semantic auditors than as generic caption-embedding targets (Xian et al., 11 Jun 2026). ReVSI shows that apparent spatial competence can be inflated or obscured by annotation noise and answerability mismatches rather than by genuine reasoning gains (Zhang et al., 27 Apr 2026). SpatialAct shows that strong observation-conditioned scores do not imply stable multi-turn state tracking or dependable action selection (Liu et al., 29 May 2026).

Another misconception is that a single representation solves all 3D tasks. Panoramas retain long-range relations but suffer from ERP distortion and dependence on rendering quality; PanoGrounder explicitly notes sensitivity to polar distortion, near-surface placements, noisy depth, and 3DGS artifacts (Jung et al., 24 Dec 2025). GaussianVLM and LLaVA$360^\circ$1 rely on high-quality Gaussian splat or NeRF reconstructions and are correspondingly sensitive to incomplete reconstructions, textureless surfaces, and per-scene preprocessing overhead (Halacheva et al., 1 Jul 2025, Petit et al., 20 Nov 2025). Med3DVLM addresses volumetric efficiency, but its own qualitative analysis still reports hallucinations and imperfect fine-grained lesion localization (Xin et al., 25 Mar 2025).

Prompt dependence, domain shift, and rare classes remain unresolved across multiple subfields. VLMFusionOcc3D notes that VLM priors can mislead under rare regional layouts or unusual vehicle morphologies, and that prompt construction matters (Doruk et al., 3 Mar 2026). WeatherOcc3D observes that binary visibility and illumination heads limit weather granularity and may misguide fusion when environment prediction is wrong (Doruk et al., 15 May 2026). The monocular HRI position-estimation system reports higher errors for vertically shaped objects, unusual designs, wider objects, and unusual lighting, with depth remaining the hardest coordinate (Wahl et al., 1 Mar 2026).

Future directions named in the literature are correspondingly diverse but coherent. PanoGrounder points to “no target” and “multiple targets,” broader panorama-driven scene understanding, and improved ERP-aware feature processing (Jung et al., 24 Dec 2025). VLMFusionOcc3D calls for open-vocabulary occupancy, few-shot domain adaptation, alternative VLMs, 4D temporal priors, and uncertainty-aware fusion (Doruk et al., 3 Mar 2026). SpatialAct recommends explicit spatial belief states, constraint-aware planners, and persistent memory (Liu et al., 29 May 2026). VLM3 argues that broader intrinsics normalization, textual multi-view constraints, and emergent 3D reconstruction through autoregressive scene graphs are promising directions (Cai et al., 28 May 2026). Taken together, these proposals suggest that VLM-3D is evolving toward systems that combine pretrained multimodal reasoning with explicit geometric state, reliability modeling, and evaluation protocols that better separate genuine 3D competence from dataset artifacts.