3D Occupancy Grounding

• 3D occupancy grounding is a family of methods that use voxel-based and sparse representations to integrate geometry, semantics, and language into a unified 3D scene understanding framework.
• It encompasses diverse approaches including voxel-level visual grounding, open-vocabulary voxel embedding, and occupancy-derived spatial reasoning, mainly applied in autonomous driving, robotics, and indoor navigation.
• Evaluation reveals challenges in spatial resolution, calibration, and standardization while demonstrating improvements in detection, segmentation, and planning through occupancy-based reasoning.

Searching arXiv for papers on 3D occupancy grounding and closely related occupancy-grounded scene understanding. 3D occupancy grounding denotes a family of methods in which the grounded object of inference is occupied support in 3D space rather than only a 2D box, a 3D cuboid, or a class label. Across the literature, the term is used in several non-identical ways: visual grounding of a referred object into a voxel mask, open-vocabulary occupancy prediction in which voxels are aligned to language, occupancy-grounded vision-language reasoning, and occupancy-grounded higher-level scene structure such as room nodes in scene graphs. The shared premise is that a voxelized or occupancy-derived representation provides a more explicit substrate for attaching geometry, semantics, and, in some systems, language to physical space (Sima et al., 2023, Dong et al., 2023, Vobecky et al., 2024, Li et al., 18 Jun 2026).

1. Conceptual scope

The immediate background is camera-based 3D occupancy estimation, which treats the scene as a voxelized geometry field rather than as a set of boxes. “A Simple Framework for 3D Occupancy Estimation in Autonomous Driving” models occupancy as a renderable 3D geometry field and evaluates it with a discrete ray-depth protocol when only sparse LiDAR is available, while “Scene as Occupancy” argues that a traffic scene should be represented as a dense, geometry-aware, semantically labeled 3D occupancy field and uses that field as a general scene descriptor for detection, segmentation, and planning (Gan et al., 2023, Sima et al., 2023). Within that setting, grounding emerges when occupancy is no longer only reconstructed, but used as the locus where object identity, language, or spatial hierarchy is resolved.

Formulation	Grounded output	Representative work
Voxel-level visual grounding	3D occupancy instance mask	OG (Dong et al., 2023)
Open-vocabulary occupancy grounding	Text-aligned voxel embeddings or voxel retrieval	POP-3D (Vobecky et al., 2024), AGO (Li et al., 14 Apr 2025)
Referred-object occupancy grounding	Ternary voxel map with target voxels	GroundingOcc (Shi et al., 2 Aug 2025)
Occupancy-grounded VLM reasoning	3D-grounded foreground tokens	Occ-VLM (Li et al., 18 Jun 2026)
Occupancy-grounded spatial hierarchy	Room polygons in a 3DSG	Occupancy-Grounded Room Segmentation (Zumaya et al., 11 Jun 2026)

A useful precursor, although not a language-grounding system, is “OccuSeg: Occupancy-aware 3D Instance Segmentation,” which defines “3D occupancy size” as the number of voxels occupied by each instance and uses that instance-level volumetric extent as a clustering prior. That formulation is directly relevant because it treats occupancy as an object-specific geometric attribute rather than only a scene-level binary field (Han et al., 2020).

2. Representational substrates

The dominant substrate is the dense voxel grid. POP-3D predicts a voxel feature grid

$$V = 3D(\mathbf{I}) \in \mathbb{R}^{H_\text{V}\times W_\text{V}\times D_\text{V}\times C_\text{V}}$$

and, in the reported implementation, uses a $100\times100\times8$ grid over $$[-51.2\text{m},+51.2\text{m}] \times [-51.2\text{m},+51.2\text{m}] \times [-5\text{m},+3\text{m}]$$, with a 512-dimensional language-aligned output per voxel (Vobecky et al., 2024). GroundingOcc defines the task-level target as

$\mathcal{O} \in \{0,1,2\}^{H\times W\times Z},$

where $0$ denotes free space, $1$ occupied space from non-target objects, and $2$ occupied space from the referred object; on Talk2Occ the scene range is $x,y\in[-40,40]$ m and $z\in[-1,5.4]$ m, with 0.4 m voxel size (Shi et al., 2 Aug 2025). Occ-VLM likewise reconstructs semantic occupancy from posed RGB images, then uses the occupied grid centers

$$\mathcal{G} = \{\mathbf{a}_i \mid \mathbf{O}_i > 0\}$$

as the spatial anchors that connect 2D tokens to 3D reasoning (Li et al., 18 Jun 2026).

Not all work keeps the final representation as a dense voxel tensor. ODG replaces dense grids with a hierarchical dual sparse Gaussian representation, separating static and dynamic queries and refining Gaussian means and semantic scores in a coarse-to-fine transformer. In that formulation, occupied support is represented by Gaussian primitives rather than per-voxel logits, but the task target remains semantic occupancy in 3D space (Shi et al., 11 Jun 2025). At a still higher abstraction level, “Occupancy-Grounded Room Segmentation for Hierarchical 3D Scene Graphs” derives a 2D free-space layer from a 3D occupancy map using vertical-clearance analysis, decomposes that free space into polygons, and anchors one room node per tracked region. Here occupancy is not the final reasoning object; it is the geometric substrate from which room-scale structure is extracted (Zumaya et al., 11 Jun 2026).

These variants indicate that 3D occupancy grounding is not tied to one representation class. It includes direct voxel fields, sparse 3D Gaussian primitives, occupancy-derived polygons, and occupancy-filtered token sets, provided that occupied structure remains the object to which semantics or higher-level reasoning are attached (Shi et al., 11 Jun 2025, Zumaya et al., 11 Jun 2026, Li et al., 18 Jun 2026).

3. Grounding mechanisms

One line of work performs direct object grounding in occupancy space. OG extends a semantic occupancy backbone with a 3-channel affinity field $100\times100\times8$0, clusters occupied voxels into instances with DBSCAN, and associates a Grounded-SAM 2D mask to one clustered 3D occupancy instance through camera-ray “scan-line voxels,” semantic filtering, and a nearest-cluster heuristic. The output is an instance-specific 3D voxel mask rather than a box (Dong et al., 2023).

A second line of work grounds voxels to language embeddings. POP-3D predicts two heads on the same 3D feature volume: an occupancy head and a 3D-language head, the latter producing a 512-dimensional embedding per voxel aligned to a pre-trained vision-language space. Training is tri-modal—images, language, and LiDAR—and uses LiDAR both for occupancy supervision and for lifting dense 2D MaskCLIP+ features into sparse 3D targets. At inference, a free-form text query is encoded and compared directly against occupied voxel embeddings, producing zero-shot 3D semantic segmentation, 3D heatmap grounding, or retrieval (Vobecky et al., 2024).

AGO formalizes this idea further as adaptive grounding. It first computes voxel-text similarity

$100\times100\times8$1

and optimizes cross-entropy and Lovász losses against 3D pseudo-labels, while a separate modality adapter maps 3D embeddings into a space aligned with VLM-derived image embeddings. An inference-time identifier then selects, per voxel, the lower-entropy prediction between the grounded branch and the adapted branch. The central claim is that known categories are handled more reliably by text-grounded 3D learning, whereas unknown categories benefit from image-embedding transfer (Li et al., 14 Apr 2025).

GroundingOcc makes the occupancy field itself the explicit target of referring expression resolution. It combines a multimodal encoder, an occupancy head, a grounding head, a 2D grounding head, and a depth estimation module. The architecture is coarse-to-fine: 2D grounding and box prediction provide coarse localization priors, while the occupancy head predicts a dense ternary voxel map for free space, general occupied space, and the referred object (Shi et al., 2 Aug 2025). Occ-VLM uses a related but broader strategy for indoor vision-language reasoning: it predicts semantic occupancy from posed RGB images, projects occupied 3D locations back into the image planes, samples the corresponding final-layer 2D tokens, adds 3D positional encoding, and feeds those occupancy-grounded foreground tokens to a LLM (Li et al., 18 Jun 2026).

4. Supervision, benchmarks, and evaluation

The supervision regimes are highly heterogeneous because dense 3D language annotations are rare. POP-3D avoids manual 3D language annotations entirely: the occupancy branch is supervised from unlabeled LiDAR point clouds, while the language branch is trained by distilling dense 2D image-language features to 3D point locations. It also introduces a language-driven 3D grounding and retrieval benchmark on top of nuScenes with 105 samples split into 42/28/35 train/val/test, evaluates on the concatenated val+test set of 63 samples, and reports 18.4 mAP, exceeding a MaskCLIP+ baseline at 14.9 mAP (Vobecky et al., 2024).

GroundingOcc introduces Talk2Occ by combining Talk2Car language with Occ3D-nuScenes occupancy labels. After filtering, the benchmark contains 8,949 training samples and 976 validation samples, and uses occupancy-set IoU,

$100\times100\times8$2

together with [email protected] and [email protected]. On this benchmark, GroundingOcc-Refine reports 32.68 overall [email protected] and 9.01 overall [email protected], compared with 21.10 and 2.46 for the strongest constructed multimodal baseline (Shi et al., 2 Aug 2025).

AGO frames evaluation as both closed-world self-supervised occupancy and open-world transfer. On Occ3D-nuScenes it reports 19.23 mIoU in the closed-world self-supervised setting, a 4.09 mIoU improvement over VEON, and then evaluates pretraining, zero-shot, and few-shot known/unknown mIoU under a label-expansion protocol. In the few-shot setting, AGO reports 38.15 known mIoU, 8.50 unknown mIoU, and 14.43 total mIoU, outperforming reproduced alignment-based baselines (Li et al., 14 Apr 2025).

Evaluation can also target occupancy-grounded spatial structure rather than referred-object masks. Occupancy-grounded room segmentation evaluates predicted room polygons on 12 Matterport3D scenes by Hungarian matching to annotated room footprints with valid matches at $100\times100\times8$3. Averaged over scenes, the occupancy-grounded method reports Recall 0.379, Precision 0.518, F1 0.427, mIoU 0.364, and boundary F1 0.224, while the Hydra place-connectivity baseline reports 0.152, 0.703, 0.241, 0.323, and 0.113 respectively (Zumaya et al., 11 Jun 2026).

The field does not yet have one universal benchmark convention. OG, for example, demonstrates 3D occupancy grounding mainly through qualitative instance-segmentation and interactive grounding results and does not define a comprehensive quantitative grounding metric in the available description (Dong et al., 2023).

5. Application regimes

Autonomous driving is the dominant application domain. “Scene as Occupancy” treats semantic occupancy as a general scene descriptor and reports that, in planning, occupancy prediction reduces collision rate by 15%–58% relative to compared representations. Collaborative Perceiver uses local-density-aware occupancy as an auxiliary structural signal for camera-only 3D detection, and ODG uses static/dynamic Gaussian queries, motion compensation, and rendering supervision to improve occupancy prediction on Occ3D-nuScenes and Occ3D-Waymo (Sima et al., 2023, Yuan et al., 28 Jul 2025, Shi et al., 11 Jun 2025). In this domain, “grounding” often means grounding scene understanding in occupied 3D structure rather than only in boxes or BEV features.

A related strand uses grounding to denote stronger geometric attribution inside the occupancy pipeline itself. “Semantic Causality-Aware Vision-Based 3D Occupancy Prediction” argues that a 3D semantic voxel should be causally attributable to the corresponding 2D evidence, introduces class-specific gradient supervision on the 2D-to-3D lifting path, and reports substantially improved robustness to camera perturbation in LSS-style pipelines (Chen et al., 10 Sep 2025). VG3S likewise uses a frozen geometry-grounded vision foundation model plus a hierarchical geometric feature adapter to inject cross-view geometric priors into Gaussian-based semantic occupancy prediction, improving GaussianFormer-2 from 30.56/20.02 to 34.41/21.52 IoU/mIoU on the nuScenes occupancy benchmark (Yan et al., 6 Mar 2026). These works broaden the term beyond language grounding to vision-to-volume or geometry-to-volume grounding.

Indoor embodied reasoning introduces a different use case. Occ-VLM reconstructs semantic occupancy from posed RGB images only and then uses occupancy as the grounding space for 3D VQA and dense captioning, reporting 18.41 mIoU on EmbodiedScan occupancy while remaining competitive with RGB-D or point-cloud VLMs on ScanQA, SQA3D, and Scan2Cap (Li et al., 18 Jun 2026). Occupancy-grounded room segmentation uses occupancy-derived free-space polygons to connect object, navigation, and room reasoning in a hierarchical 3DSG, turning room nodes into explicit geometric entities rather than latent topological clusters (Zumaya et al., 11 Jun 2026).

Sidewalk robotics extends the idea to monocular, robot-centric perception. WalkOCC predicts a $100\times100\times8$4 semantic occupancy volume from a single front-facing RGB image over $100\times100\times8$5 m, $100\times100\times8$6 m, and $100\times100\times8$7 m, and couples LiDAR-RGB paired geometry with larger 2D-only monocular data. On Sidewalk3D it improves over FlashOCC from 27.17 to 30.02 occ IoU and from 14.23 to 16.46 mIoU, while also improving fine-grained classes such as curb and gutter (Ma et al., 17 Jun 2026).

6. Recurrent limitations and unresolved questions

A persistent source of confusion is definitional. A common misconception is that 3D occupancy grounding is synonymous with language-conditioned referring expression resolution. The cited literature does not support that restriction: some papers use the term for open-vocabulary voxel-language alignment, some for occupancy-grounded VLM tokenization, and others for the geometric grounding of occupancy predictors in cross-view structure or 2D semantic causality (Chen et al., 10 Sep 2025, Yan et al., 6 Mar 2026). The field therefore lacks a single canonical task boundary.

Resolution and representation fidelity remain fundamental bottlenecks. POP-3D explicitly notes that its $100\times100\times8$8 voxel grid hurts small-object discovery and fine localization, GroundingOcc’s best reported [email protected] is 9.01 on Talk2Occ, and OG relies on a relatively heavy external Grounded-SAM module while providing mainly qualitative grounding evidence (Vobecky et al., 2024, Shi et al., 2 Aug 2025, Dong et al., 2023). These results indicate that voxel-level grounding is more demanding than box localization and that both geometric precision and standardized evaluation are still immature.

Many current systems also depend strongly on calibration, pseudo-label quality, or auxiliary geometric modules. WalkOCC requires accurate intrinsics, extrinsics, LiDAR-camera synchronization, and pose estimation for temporal accumulation; AGO depends on the quality of Grounded-SAM pseudo-labels and projection from 2D to 3D; Occ-VLM requires posed RGB inputs and does not define an explicit unknown state in its occupancy representation (Ma et al., 17 Jun 2026, Li et al., 14 Apr 2025, Li et al., 18 Jun 2026). This makes the empirical performance of occupancy grounding tightly coupled to upstream geometric fidelity.

At larger spatial scales, occupancy-derived structure still suffers from boundary ambiguity. Occupancy-grounded room segmentation improves room-instance recovery over place-connectivity clustering, but the paper explicitly states that wall-accurate room boundaries remain an open problem, especially under map defects, open-plan layouts, and local free-space mergers (Zumaya et al., 11 Jun 2026). Similar issues recur at the object level whenever thin structures, occlusion, or coarse voxelization make occupied support geometrically ambiguous.

This suggests that future progress in 3D occupancy grounding will depend less on introducing occupancy as a representation—already well established—and more on improving how occupancy is aligned to language, semantics, visibility, and higher-level scene structure across scales.