---
title: 'GroundingOcc: Voxel-Based 3D Occupancy Grounding'
url: https://www.emergentmind.com/topics/groundingocc
type: topic
---

# GroundingOcc: Voxel-Based 3D Occupancy Grounding

GroundingOcc denotes an end-to-end formulation of **3D occupancy grounding** in which a natural-language expression is grounded not by a 2D or 3D bounding box, but by a voxelized occupancy field over the scene. In the formulation introduced for autonomous driving, the inputs are a natural language prompt \(T=\{w_1,\dots,w_L\}\), a point cloud \(P=\{p_1,\dots,p_N\}\), and multi-view images \(I=\{I_1,\dots,I_K\}\); the output is a voxel grid \(\mathcal{O}\in\{0,1,2\}^{H\times W\times Z}\) whose labels denote free space, voxels occupied by any object, and voxels occupied specifically by the referred object [2508.01197]. This replaces box-level grounding with voxel-level grounding, and thereby makes object extent, shape irregularity, and partial occupancy explicit. In a broader research sense, “GroundingOcc” also names a family of grounding problems in which occluded, cluttered, or spatially structured regions are constrained by occupancy, geometry, interaction, or scene structure rather than treated as unconstrained completion [2508.01197][2307.05873][2604.09100].

## 1. Task definition and representational scope

GroundingOcc, in its canonical form, is defined as **multi-modality 3D occupancy grounding** for autonomous driving [2508.01197]. The referred object is specified by language, while geometry is supplied by LiDAR and surround-view imagery. The target representation is not a box but a voxel field,
\[
\mathcal{O} \in \{0,1,2\}^{H\times W\times Z},
\]
where class \(0\) denotes free space, class \(1\) denotes voxels occupied by any object, and class \(2\) denotes voxels occupied by the referred object [2508.01197].

This formulation differs from conventional grounding in two ways. First, it distinguishes general scene occupancy from referent-specific occupancy in a single output space. Second, it evaluates grounding through voxel overlap rather than box overlap. The ground-truth referred-object occupancy is defined as
\[
\mathcal{O}_{\text{gt}} = \{ v_i \mid v_i \in \mathbf{V}_{\text{occ}} \cap \mathcal{B},\ \ell(v_i) \neq \text{free} \},
\]
where \(\mathbf{V}_{\text{occ}}\) are occupied voxels, \(\mathcal{B}\) is the ground-truth 3D bounding box of the referred object, and \(\ell(v_i)\) is the voxel semantic label [2508.01197]. Prediction quality is then measured by voxel IoU and thresholded as Acc@0.25 and Acc@0.5:
\[
\text{IoU} = \frac{|\mathcal{O}_{\text{gt}} \cap \mathcal{O}_{\text{pred}}|}{|\mathcal{O}_{\text{gt}} \cup \mathcal{O}_{\text{pred}}|}, \qquad
\text{Acc}@\theta = \frac{1}{N}\sum_{i=1}^{N}\left(\text{IoU}_i > \theta\right).
\]
These definitions make the task sensitive to the actual occupied subset of a referent volume rather than to the enclosing rectangular extent alone [2508.01197].

A plausible implication is that GroundingOcc shifts grounding from a localization problem to a joint **localization-plus-structure** problem. That interpretation is reinforced by related occupancy-grounding work in indoor scenes, where voxel-wise grounding is also performed at the level of 3D occupancy instances rather than boxes [2307.05873].

## 2. Benchmark construction and data regime

The benchmark associated with GroundingOcc is **Talk2Occ**, constructed by linking **Talk2Car** language annotations and **Occ3D-nuScenes** occupancy labels through shared nuScenes sample tokens [2508.01197]. For each sample, the benchmark provides surround-view camera images, LiDAR, 3D bounding boxes, voxel-level occupancy labels, and a referring expression inherited from Talk2Car [2508.01197].

To ensure sample quality, objects are retained only when their 3D bounding-box centers satisfy
\[
x \in [-40\,\text{m},40\,\text{m}],\quad
y \in [-40\,\text{m},40\,\text{m}],\quad
z \in [-1\,\text{m},5.4\,\text{m}],
\]
and only if the boxes contain occupied voxels [2508.01197]. All annotations are transformed into a common ego-vehicle coordinate system [2508.01197]. The 3D region represented in experiments spans \(x,y\in[-40.0\,\text{m},40.0\,\text{m}]\) and \(z\in[-1.0\,\text{m},5.4\,\text{m}]\), with voxel size \((0.4\,\text{m},0.4\,\text{m},0.4\,\text{m})\) [2508.01197]. The model uses an initial voxel-query resolution of \(50\times 50\times 16\), upsampled to \(200\times 200\times 32\) for supervision [2508.01197].

Talk2Occ reuses **11,959** natural-language prompts from Talk2Car, each tied to one referent object [2508.01197]. After filtering, **9,925** objects remain, split into **8,949** training samples and **976** validation samples [2508.01197]. The dataset also distinguishes **Unique** scenes, in which the referred category appears once, from **Multiple** scenes, in which several instances of that category are present [2508.01197]. Table 1 in the source paper reports, for example, **4,515** car instances with about **180** occupied voxels per object, **2,603** pedestrian instances with about **32** voxels, **981** trucks with about **446** voxels, and **344** buses with about **871** voxels [2508.01197].

This benchmark design places GroundingOcc in direct contrast with box-based visual grounding. The data construction also suggests that the task is particularly sensitive to long or irregular objects, partially occupied volumes, and same-class disambiguation in dense traffic scenes [2508.01197].

## 3. Model architecture and coarse-to-fine grounding

GroundingOcc is a **single-stage**, end-to-end model that fuses multi-view RGB, LiDAR, and language [2508.01197]. Its architecture consists of a multimodal encoder, auxiliary 2D grounding and depth modules, an occupancy head for voxel-wise prediction, and a grounding head for 3D box refinement [2508.01197].

The visual backbone is **ResNet101-DCN**, producing multi-scale features \(C_3,C_4,C_5,C_6\) at \(1/8,1/16,1/32,\) and \(1/64\) resolution [2508.01197]. The text encoder is **RoBERTa-base**, yielding token embeddings
\[
F^\mathcal{T}\in\mathbb{R}^{L\times d}
\]
used throughout the model [2508.01197]. LiDAR is voxelized through parameterized voxelization and 3D sparse convolutions to produce voxel features \(F^\mathcal{L}\), while camera voxel features \(F^\mathcal{C}\) are obtained by a 3D voxel-query mechanism conditioned on text embeddings [2508.01197]. These two streams are adaptively fused into fused voxel features \(F^\mathcal{F}\) [2508.01197].

A **Vision-Language PAN**, inspired by YOLO-World, conditions image features on text by computing
\[
X_l' = X_l + X_l \cdot \sigma\left(\max_{j\in\{1,\ldots,L\}}(X_l F^\mathcal{T}_j{}^\top)\right)^\top
\]
at each scale \(l\) [2508.01197]. This produces language-aware feature pyramids that feed both the 2D grounding branch and the voxel-query pipeline [2508.01197].

The occupancy head upsamples fused coarse voxel features
\[
F^\mathcal{F}\in\mathbb{R}^{\frac{H}{S}\times \frac{W}{S}\times \frac{Z}{S}\times D}
\]
with 3D deconvolutions to high-resolution voxel features
\[
F^\mathcal{O}\in\mathbb{R}^{H\times W\times Z\times D'},
\]
then predicts occupancy grounding logits \(O^\mathcal{OG}\) and a binary occupancy mask \(O^\mathcal{M}\) [2508.01197]. The grounding head separately performs BEV pooling over \(F^\mathcal{O}\), then uses a Deformable-DETR-like decoder to regress referred-object box parameters \((l,h,w)\), \((x_{3D},y_{3D},z_{3D})\), and yaw \(\psi\) [2508.01197].

The model is explicitly described as **coarse-to-fine** in three senses: coarse voxel queries are refined into a dense occupancy grid; coarse 2D and depth cues inform fine 3D voxel prediction; and coarse 3D box localization can optionally refine fine voxel occupancy in the **GroundingOcc-Refine** variant [2508.01197].

## 4. Auxiliary modules, geometric supervision, and training objectives

The 2D grounding module is intended to provide an easier intermediate grounding signal [2508.01197]. A transformer decoder over language-conditioned image features computes
\[
X_{2d} = \text{FFN}(\text{MHCA}(\text{MHSA}(X_l'), F^\mathcal{T})),
\]
and convolutional heads then predict box offsets \((l,r,t,b)\), 2D center \((x_{2D},y_{2D})\), and centerness \(d\) [2508.01197]. The resulting 2D spatial information is fused into 3D grounding queries through
\[
Q_{\text{fusion}} = \text{Fusion}(Q_g,\text{TopK}(F),\text{PE}(p),Q_{pos}),
\]
where \(Q_g\) are grounding-query contents and \(\text{PE}(p)\) encodes normalized 3D coordinates [2508.01197].

The depth-estimation module uses occupancy-derived ray-casting instead of sparse projected LiDAR for supervision [2508.01197]. For each pixel,
\[
[X_c,Y_c,1]^T = K^{-1}[u,v,1]^T
\]
defines the ray in camera coordinates, which is transformed to ego coordinates and voxelized. The depth target is the smallest sampled distance \(d\) whose voxel is not free:
\[
D(u,v)=\min\{d \mid O(V_{\text{index}})\neq l_{\text{free}}\}.
\]
This makes depth supervision geometrically consistent with the occupancy field itself [2508.01197].

The occupancy objective is
\[
\mathcal{L}_{\text{occ}} =
\lambda_7\mathcal{L}_{\text{ce}} +
\lambda_8\mathcal{L}_{\text{mask}} +
\lambda_9\mathcal{L}_{\text{Lovasz}} +
\lambda_{10}\mathcal{L}_{\text{scal}}^{\text{geo}} +
\lambda_{11}\mathcal{L}_{\text{scal}}^{\text{sem}},
\]
with \(\lambda_{7\sim 11}=(10,2,1,1,1)\) [2508.01197]. Here \(\mathcal{L}_{\text{ce}}\) is voxel cross-entropy, \(\mathcal{L}_{\text{mask}}\) is focal loss on \(O^\mathcal{M}\), \(\mathcal{L}_{\text{Lovasz}}\) is Lovasz-Softmax, and the final two terms are geometric and semantic scene-class affinity losses from MonoScene [2508.01197]. The 2D grounding loss is
\[
\mathcal{L}_{\text{2D}}=
\lambda_1\mathcal{L}_{\text{lrtb}}+
\lambda_2\mathcal{L}_{\text{xy2D}}+
\lambda_3\mathcal{L}_{\text{centerness}}+
\lambda_4\mathcal{L}_{\text{GIoU}},
\]
with \(\lambda_{1\sim4}=(5,10,1,2)\), and the 3D loss is
\[
\mathcal{L}_{\text{3D}}=\lambda_5\mathcal{L}_{\text{cls}}+\lambda_6\mathcal{L}_{\text{bbox}},
\]
with \(\lambda_{5\sim6}=(2,0.25)\) [2508.01197]. The overall objective is
\[
\mathcal{L}_{\text{ours}}=
\mathcal{L}_{\text{2D}}+
\mathcal{L}_{\text{3D}}+
\mathcal{L}_{\text{dmap}}+
\mathcal{L}_{\text{occ}}.
\]

Training uses **4 NVIDIA RTX 4090 GPUs**, batch size **1** per GPU, **AdamW**, initial learning rate \(2\times 10^{-4}\), weight decay **0.01**, cosine annealing, and data augmentation including image cropping, color distortion, and GridMask [2508.01197]. The image resolution is \(320\times 800\) [2508.01197].

A plausible interpretation is that GroundingOcc treats 2D grounding and depth not as auxiliary outputs for their own sake, but as geometric regularizers for voxel grounding. That reading is consistent with the reported ablations [2508.01197].

## 5. Empirical performance and ablation findings

GroundingOcc is evaluated against random and two-stage baselines: **GT-Rand**, **Box-Rand**, **L-baseline**, **C-baseline**, **M-baseline**, **GroundingOcc**, and **GroundingOcc-Refine** [2508.01197]. The overall results on Talk2Occ are as follows.

| Method | Acc@0.25 | Acc@0.5 |
|---|---:|---:|
| GT-Rand | 4.81 | 3.48 |
| Box-Rand | 5.22 | 5.02 |
| L-baseline | 11.06 | 1.43 |
| C-baseline | 15.98 | 1.74 |
| M-baseline | 21.10 | 2.46 |
| GroundingOcc | 27.35 | 7.47 |
| GroundingOcc-Refine | 32.68 | 9.01 |

GroundingOcc therefore improves over the multi-modal two-stage baseline by **+6.25** at Acc@0.25 and **+5.01** at Acc@0.5, while GroundingOcc-Refine adds a further **+5.33** and **+1.54** over the single-stage model [2508.01197]. In the harder **Multiple** split, the gains remain pronounced: M-baseline reports **22.48 / 2.59**, GroundingOcc **28.58 / 7.68**, and GroundingOcc-Refine **34.01 / 9.49** [2508.01197]. The source paper attributes this to improved language-based disambiguation among same-class objects, aided by depth and 2D grounding [2508.01197].

Module ablations isolate the contribution of successive components. Starting from M-baseline at **21.10 / 2.46**, adding multi-frame fusion yields **21.90 / 3.07**, adding the depth predictor yields **24.46 / 5.40**, and adding the 2D grounding head yields **26.94 / 5.94** [2508.01197]. Occupancy-loss ablations further show that semantic supervision alone gives **26.94 / 5.94**, adding geometric supervision gives **29.30 / 8.07**, adding \(\mathcal{L}_{\text{scal}}^{\text{geo}}\) gives **31.67 / 8.68**, and adding \(\mathcal{L}_{\text{scal}}^{\text{sem}}\) reaches **32.68 / 9.01** [2508.01197].

These ablations indicate that geometric supervision is not merely auxiliary but materially changes voxel-grounding quality [2508.01197]. The qualitative analysis in the source text also reports that the multi-modal baseline tends to produce “rough” occupancy and poor depth alignment, whereas GroundingOcc-Refine yields tighter referred-object occupancy, with remaining failure modes in ambiguous referring expressions and scenes containing multiple similar objects [2508.01197].

## 6. Relation to occupancy grounding, occlusion grounding, and physically grounded completion

GroundingOcc belongs to a broader line of work that replaces coarse localization with structured grounding. In indoor 3D occupancy prediction, **Occupancy Grounding (OG)** equips semantic occupancy with instance segmentation by predicting a 3D affinity field and associating 2D Grounded-SAM masks to 3D occupancy instances through pixel-to-voxel transformation and clustering [2307.05873]. OG is trained with
\[
\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{ori}}+\lambda \mathcal{L}_{\text{aff}},
\]
where the affinity field regresses voxel offsets to instance centers and supports voxel-wise grounding after DBSCAN clustering [2307.05873]. Relative to OG, GroundingOcc moves from indoor single-image occupancy plus external 2D grounding to an end-to-end, multi-modal, language-conditioned driving setting with direct referred-object occupancy supervision [2508.01197][2307.05873].

A second adjacent direction appears in **“Physically Grounded 3D Generative Reconstruction under Hand Occlusion using Proprioception and Multi-Contact Touch”**, which frames occlusion grounding as metric-scale amodal object reconstruction under severe hand occlusion [2604.09100]. There, the invisible geometry is constrained by a camera-aligned signed distance field, proprioceptive hand geometry, touch-derived contact voxels, and non-interpenetration objectives [2604.09100]. This work does not define GroundingOcc as a benchmark name, but it uses the term in the sense of **grounding occluded regions** by embedding hand and object into a common metric grid and constraining completion with physical interaction [2604.09100]. This suggests a broader research meaning of GroundingOcc: not only grounding language in occupancy, but grounding missing or ambiguous spatial structure through physically valid occupancy constraints.

A third relevant direction is **AnchorDiff**, a training-free concept-grounding method for MM-DiTs that decouples semantic anchoring from structural refinement through graph propagation on image self-attention [2605.26460]. Although it operates in 2D image-token space rather than voxel space, its core principle—using semantics only to choose an anchor and using structure to control propagation—addresses concept leakage in cluttered scenes [2605.26460]. A plausible implication is that similar anchor-and-propagate schemes could be adapted to occupancy grounding where several same-class objects coexist.

The autonomous-driving GroundingOcc paper also identifies clear limitations: dependence on nuScenes-derived data and sensor configuration, the resolution trade-off imposed by \(0.4\text{ m}\) voxels and a \(200\times 200\times 32\) grid, limited linguistic diversity inherited from Talk2Car, possible annotation noise from automatically derived occupancy labels, and errors in ambiguous scenes where language does not uniquely identify an object [2508.01197]. The source text highlights future directions including more diverse benchmarks, open-vocabulary grounding, stronger language understanding, and continuous geometric representations such as NeRF or implicit fields [2508.01197].

Taken together, these works position GroundingOcc as a technically specific model and benchmark for **language-guided 3D occupancy grounding** in autonomous driving [2508.01197], and as a broader paradigm in which grounding is performed over structured spatial fields—occupancy grids, SDFs, or instance-aware voxel volumes—rather than over boxes alone [2307.05873][2604.09100].

Source: https://www.emergentmind.com/topics/groundingocc