GroundingOcc: Voxel-Based 3D Occupancy Grounding

• GroundingOcc is an end-to-end framework for 3D occupancy grounding that assigns each voxel a label to distinguish free space, general objects, and a specific referent.
• It replaces box-level grounding with voxel-level prediction, enabling precise shape delineation and disambiguation in cluttered, complex driving scenes with metrics such as voxel IoU.
• The approach employs a coarse-to-fine strategy that fuses multi-modal inputs—including 2D, depth, and language cues—to significantly improve grounding accuracy over traditional methods.

GroundingOcc denotes an end-to-end formulation of 3D occupancy grounding in which a natural-language expression is grounded not by a 2D or 3D bounding box, but by a voxelized occupancy field over the scene. In the formulation introduced for autonomous driving, the inputs are a natural language prompt $T=\{w_1,\dots,w_L\}$, a point cloud $P=\{p_1,\dots,p_N\}$, and multi-view images $I=\{I_1,\dots,I_K\}$; the output is a voxel grid $\mathcal{O}\in\{0,1,2\}^{H\times W\times Z}$ whose labels denote free space, voxels occupied by any object, and voxels occupied specifically by the referred object (Shi et al., 2 Aug 2025). This replaces box-level grounding with voxel-level grounding, and thereby makes object extent, shape irregularity, and partial occupancy explicit. In a broader research sense, “GroundingOcc” also names a family of grounding problems in which occluded, cluttered, or spatially structured regions are constrained by occupancy, geometry, interaction, or scene structure rather than treated as unconstrained completion (Shi et al., 2 Aug 2025, Dong et al., 2023, Caddeo et al., 10 Apr 2026).

1. Task definition and representational scope

GroundingOcc, in its canonical form, is defined as multi-modality 3D occupancy grounding for autonomous driving (Shi et al., 2 Aug 2025). The referred object is specified by language, while geometry is supplied by LiDAR and surround-view imagery. The target representation is not a box but a voxel field,

$\mathcal{O} \in \{0,1,2\}^{H\times W\times Z},$

where class $0$ denotes free space, class $1$ denotes voxels occupied by any object, and class $2$ denotes voxels occupied by the referred object (Shi et al., 2 Aug 2025).

This formulation differs from conventional grounding in two ways. First, it distinguishes general scene occupancy from referent-specific occupancy in a single output space. Second, it evaluates grounding through voxel overlap rather than box overlap. The ground-truth referred-object occupancy is defined as

$$\mathcal{O}_{\text{gt}} = \{ v_i \mid v_i \in \mathbf{V}_{\text{occ}} \cap \mathcal{B},\ \ell(v_i) \neq \text{free} \},$$

where $\mathbf{V}_{\text{occ}}$ are occupied voxels, $P=\{p_1,\dots,p_N\}$0 is the ground-truth 3D bounding box of the referred object, and $P=\{p_1,\dots,p_N\}$1 is the voxel semantic label (Shi et al., 2 Aug 2025). Prediction quality is then measured by voxel IoU and thresholded as [email protected] and [email protected]: $P=\{p_1,\dots,p_N\}$2 These definitions make the task sensitive to the actual occupied subset of a referent volume rather than to the enclosing rectangular extent alone (Shi et al., 2 Aug 2025).

A plausible implication is that GroundingOcc shifts grounding from a localization problem to a joint localization-plus-structure problem. That interpretation is reinforced by related occupancy-grounding work in indoor scenes, where voxel-wise grounding is also performed at the level of 3D occupancy instances rather than boxes (Dong et al., 2023).

2. Benchmark construction and data regime

The benchmark associated with GroundingOcc is Talk2Occ, constructed by linking Talk2Car language annotations and Occ3D-nuScenes occupancy labels through shared nuScenes sample tokens (Shi et al., 2 Aug 2025). For each sample, the benchmark provides surround-view camera images, LiDAR, 3D bounding boxes, voxel-level occupancy labels, and a referring expression inherited from Talk2Car (Shi et al., 2 Aug 2025).

To ensure sample quality, objects are retained only when their 3D bounding-box centers satisfy

$P=\{p_1,\dots,p_N\}$3

and only if the boxes contain occupied voxels (Shi et al., 2 Aug 2025). All annotations are transformed into a common ego-vehicle coordinate system (Shi et al., 2 Aug 2025). The 3D region represented in experiments spans $P=\{p_1,\dots,p_N\}$4 and $P=\{p_1,\dots,p_N\}$5, with voxel size $P=\{p_1,\dots,p_N\}$6 (Shi et al., 2 Aug 2025). The model uses an initial voxel-query resolution of $P=\{p_1,\dots,p_N\}$7, upsampled to $P=\{p_1,\dots,p_N\}$8 for supervision (Shi et al., 2 Aug 2025).

Talk2Occ reuses 11,959 natural-language prompts from Talk2Car, each tied to one referent object (Shi et al., 2 Aug 2025). After filtering, 9,925 objects remain, split into 8,949 training samples and 976 validation samples (Shi et al., 2 Aug 2025). The dataset also distinguishes Unique scenes, in which the referred category appears once, from Multiple scenes, in which several instances of that category are present (Shi et al., 2 Aug 2025). Table 1 in the source paper reports, for example, 4,515 car instances with about 180 occupied voxels per object, 2,603 pedestrian instances with about 32 voxels, 981 trucks with about 446 voxels, and 344 buses with about 871 voxels (Shi et al., 2 Aug 2025).

This benchmark design places GroundingOcc in direct contrast with box-based visual grounding. The data construction also suggests that the task is particularly sensitive to long or irregular objects, partially occupied volumes, and same-class disambiguation in dense traffic scenes (Shi et al., 2 Aug 2025).

3. Model architecture and coarse-to-fine grounding

GroundingOcc is a single-stage, end-to-end model that fuses multi-view RGB, LiDAR, and language (Shi et al., 2 Aug 2025). Its architecture consists of a multimodal encoder, auxiliary 2D grounding and depth modules, an occupancy head for voxel-wise prediction, and a grounding head for 3D box refinement (Shi et al., 2 Aug 2025).

The visual backbone is ResNet101-DCN, producing multi-scale features $P=\{p_1,\dots,p_N\}$9 at $I=\{I_1,\dots,I_K\}$0 and $I=\{I_1,\dots,I_K\}$1 resolution (Shi et al., 2 Aug 2025). The text encoder is RoBERTa-base, yielding token embeddings

$I=\{I_1,\dots,I_K\}$2

used throughout the model (Shi et al., 2 Aug 2025). LiDAR is voxelized through parameterized voxelization and 3D sparse convolutions to produce voxel features $I=\{I_1,\dots,I_K\}$3, while camera voxel features $I=\{I_1,\dots,I_K\}$4 are obtained by a 3D voxel-query mechanism conditioned on text embeddings (Shi et al., 2 Aug 2025). These two streams are adaptively fused into fused voxel features $I=\{I_1,\dots,I_K\}$5 (Shi et al., 2 Aug 2025).

A Vision-Language PAN, inspired by YOLO-World, conditions image features on text by computing

$I=\{I_1,\dots,I_K\}$6

at each scale $I=\{I_1,\dots,I_K\}$7 (Shi et al., 2 Aug 2025). This produces language-aware feature pyramids that feed both the 2D grounding branch and the voxel-query pipeline (Shi et al., 2 Aug 2025).

The occupancy head upsamples fused coarse voxel features

$I=\{I_1,\dots,I_K\}$8

with 3D deconvolutions to high-resolution voxel features

$I=\{I_1,\dots,I_K\}$9

then predicts occupancy grounding logits $\mathcal{O}\in\{0,1,2\}^{H\times W\times Z}$0 and a binary occupancy mask $\mathcal{O}\in\{0,1,2\}^{H\times W\times Z}$1 (Shi et al., 2 Aug 2025). The grounding head separately performs BEV pooling over $\mathcal{O}\in\{0,1,2\}^{H\times W\times Z}$2, then uses a Deformable-DETR-like decoder to regress referred-object box parameters $\mathcal{O}\in\{0,1,2\}^{H\times W\times Z}$3, $\mathcal{O}\in\{0,1,2\}^{H\times W\times Z}$4, and yaw $\mathcal{O}\in\{0,1,2\}^{H\times W\times Z}$5 (Shi et al., 2 Aug 2025).

The model is explicitly described as coarse-to-fine in three senses: coarse voxel queries are refined into a dense occupancy grid; coarse 2D and depth cues inform fine 3D voxel prediction; and coarse 3D box localization can optionally refine fine voxel occupancy in the GroundingOcc-Refine variant (Shi et al., 2 Aug 2025).

4. Auxiliary modules, geometric supervision, and training objectives

The 2D grounding module is intended to provide an easier intermediate grounding signal (Shi et al., 2 Aug 2025). A transformer decoder over language-conditioned image features computes

$\mathcal{O}\in\{0,1,2\}^{H\times W\times Z}$6

and convolutional heads then predict box offsets $\mathcal{O}\in\{0,1,2\}^{H\times W\times Z}$7, 2D center $\mathcal{O}\in\{0,1,2\}^{H\times W\times Z}$8, and centerness $\mathcal{O}\in\{0,1,2\}^{H\times W\times Z}$9 (Shi et al., 2 Aug 2025). The resulting 2D spatial information is fused into 3D grounding queries through

$\mathcal{O} \in \{0,1,2\}^{H\times W\times Z},$0

where $\mathcal{O} \in \{0,1,2\}^{H\times W\times Z},$1 are grounding-query contents and $\mathcal{O} \in \{0,1,2\}^{H\times W\times Z},$2 encodes normalized 3D coordinates (Shi et al., 2 Aug 2025).

The depth-estimation module uses occupancy-derived ray-casting instead of sparse projected LiDAR for supervision (Shi et al., 2 Aug 2025). For each pixel,

$\mathcal{O} \in \{0,1,2\}^{H\times W\times Z},$3

defines the ray in camera coordinates, which is transformed to ego coordinates and voxelized. The depth target is the smallest sampled distance $\mathcal{O} \in \{0,1,2\}^{H\times W\times Z},$4 whose voxel is not free: $\mathcal{O} \in \{0,1,2\}^{H\times W\times Z},$5 This makes depth supervision geometrically consistent with the occupancy field itself (Shi et al., 2 Aug 2025).

The occupancy objective is

$\mathcal{O} \in \{0,1,2\}^{H\times W\times Z},$6

with $\mathcal{O} \in \{0,1,2\}^{H\times W\times Z},$7 (Shi et al., 2 Aug 2025). Here $\mathcal{O} \in \{0,1,2\}^{H\times W\times Z},$8 is voxel cross-entropy, $\mathcal{O} \in \{0,1,2\}^{H\times W\times Z},$9 is focal loss on $0$0, $0$1 is Lovasz-Softmax, and the final two terms are geometric and semantic scene-class affinity losses from MonoScene (Shi et al., 2 Aug 2025). The 2D grounding loss is

$0$2

with $0$3, and the 3D loss is

$0$4

with $0$5 (Shi et al., 2 Aug 2025). The overall objective is

$0$6

Training uses 4 NVIDIA RTX 4090 GPUs, batch size 1 per GPU, AdamW, initial learning rate $0$7, weight decay 0.01, cosine annealing, and data augmentation including image cropping, color distortion, and GridMask (Shi et al., 2 Aug 2025). The image resolution is $0$8 (Shi et al., 2 Aug 2025).

A plausible interpretation is that GroundingOcc treats 2D grounding and depth not as auxiliary outputs for their own sake, but as geometric regularizers for voxel grounding. That reading is consistent with the reported ablations (Shi et al., 2 Aug 2025).

5. Empirical performance and ablation findings

GroundingOcc is evaluated against random and two-stage baselines: GT-Rand, Box-Rand, L-baseline, C-baseline, M-baseline, GroundingOcc, and GroundingOcc-Refine (Shi et al., 2 Aug 2025). The overall results on Talk2Occ are as follows.

Method	[email protected]	[email protected]
GT-Rand	4.81	3.48
Box-Rand	5.22	5.02
L-baseline	11.06	1.43
C-baseline	15.98	1.74
M-baseline	21.10	2.46
GroundingOcc	27.35	7.47
GroundingOcc-Refine	32.68	9.01

GroundingOcc therefore improves over the multi-modal two-stage baseline by +6.25 at [email protected] and +5.01 at [email protected], while GroundingOcc-Refine adds a further +5.33 and +1.54 over the single-stage model (Shi et al., 2 Aug 2025). In the harder Multiple split, the gains remain pronounced: M-baseline reports 22.48 / 2.59, GroundingOcc 28.58 / 7.68, and GroundingOcc-Refine 34.01 / 9.49 (Shi et al., 2 Aug 2025). The source paper attributes this to improved language-based disambiguation among same-class objects, aided by depth and 2D grounding (Shi et al., 2 Aug 2025).

Module ablations isolate the contribution of successive components. Starting from M-baseline at 21.10 / 2.46, adding multi-frame fusion yields 21.90 / 3.07, adding the depth predictor yields 24.46 / 5.40, and adding the 2D grounding head yields 26.94 / 5.94 (Shi et al., 2 Aug 2025). Occupancy-loss ablations further show that semantic supervision alone gives 26.94 / 5.94, adding geometric supervision gives 29.30 / 8.07, adding $0$9 gives 31.67 / 8.68, and adding $1$0 reaches 32.68 / 9.01 (Shi et al., 2 Aug 2025).

These ablations indicate that geometric supervision is not merely auxiliary but materially changes voxel-grounding quality (Shi et al., 2 Aug 2025). The qualitative analysis in the source text also reports that the multi-modal baseline tends to produce “rough” occupancy and poor depth alignment, whereas GroundingOcc-Refine yields tighter referred-object occupancy, with remaining failure modes in ambiguous referring expressions and scenes containing multiple similar objects (Shi et al., 2 Aug 2025).

6. Relation to occupancy grounding, occlusion grounding, and physically grounded completion

GroundingOcc belongs to a broader line of work that replaces coarse localization with structured grounding. In indoor 3D occupancy prediction, Occupancy Grounding (OG) equips semantic occupancy with instance segmentation by predicting a 3D affinity field and associating 2D Grounded-SAM masks to 3D occupancy instances through pixel-to-voxel transformation and clustering (Dong et al., 2023). OG is trained with

$1$1

where the affinity field regresses voxel offsets to instance centers and supports voxel-wise grounding after DBSCAN clustering (Dong et al., 2023). Relative to OG, GroundingOcc moves from indoor single-image occupancy plus external 2D grounding to an end-to-end, multi-modal, language-conditioned driving setting with direct referred-object occupancy supervision (Shi et al., 2 Aug 2025, Dong et al., 2023).

A second adjacent direction appears in “Physically Grounded 3D Generative Reconstruction under Hand Occlusion using Proprioception and Multi-Contact Touch”, which frames occlusion grounding as metric-scale amodal object reconstruction under severe hand occlusion (Caddeo et al., 10 Apr 2026). There, the invisible geometry is constrained by a camera-aligned signed distance field, proprioceptive hand geometry, touch-derived contact voxels, and non-interpenetration objectives (Caddeo et al., 10 Apr 2026). This work does not define GroundingOcc as a benchmark name, but it uses the term in the sense of grounding occluded regions by embedding hand and object into a common metric grid and constraining completion with physical interaction (Caddeo et al., 10 Apr 2026). This suggests a broader research meaning of GroundingOcc: not only grounding language in occupancy, but grounding missing or ambiguous spatial structure through physically valid occupancy constraints.

A third relevant direction is AnchorDiff, a training-free concept-grounding method for MM-DiTs that decouples semantic anchoring from structural refinement through graph propagation on image self-attention (Zhang et al., 26 May 2026). Although it operates in 2D image-token space rather than voxel space, its core principle—using semantics only to choose an anchor and using structure to control propagation—addresses concept leakage in cluttered scenes (Zhang et al., 26 May 2026). A plausible implication is that similar anchor-and-propagate schemes could be adapted to occupancy grounding where several same-class objects coexist.

The autonomous-driving GroundingOcc paper also identifies clear limitations: dependence on nuScenes-derived data and sensor configuration, the resolution trade-off imposed by $1$2 voxels and a $1$3 grid, limited linguistic diversity inherited from Talk2Car, possible annotation noise from automatically derived occupancy labels, and errors in ambiguous scenes where language does not uniquely identify an object (Shi et al., 2 Aug 2025). The source text highlights future directions including more diverse benchmarks, open-vocabulary grounding, stronger language understanding, and continuous geometric representations such as NeRF or implicit fields (Shi et al., 2 Aug 2025).

Taken together, these works position GroundingOcc as a technically specific model and benchmark for language-guided 3D occupancy grounding in autonomous driving (Shi et al., 2 Aug 2025), and as a broader paradigm in which grounding is performed over structured spatial fields—occupancy grids, SDFs, or instance-aware voxel volumes—rather than over boxes alone (Dong et al., 2023, Caddeo et al., 10 Apr 2026).