SGCDet: Multi-View 3D Object Detection

• SGCDet is a multi-view indoor 3D object detection framework that adaptively integrates geometric and contextual features from posed RGB images using deformable, attention-driven feature lifting.
• It employs a coarse-to-fine sparse volume construction strategy that concentrates computations on high-probability occupied voxels, enhancing efficiency and detection performance.
• The framework achieves state-of-the-art results on ScanNet and ARKitScenes with full supervision from 3D bounding box annotations, eliminating the need for explicit ground-truth geometry.

SGCDet is a multi-view indoor 3D object detection framework designed to enhance volumetric feature aggregation by adaptively integrating geometric and contextual cues from multiple posed RGB images. Unlike prior approaches that rely on fixed voxel-image correspondences or require explicit ground-truth scene geometry, SGCDet introduces a geometry and context aware aggregation module that performs deformable, attention-driven feature lifting from images to voxels, and a sparse volume construction strategy that focuses computation on high-probability occupancies. The approach achieves state-of-the-art results on established datasets while maintaining computational efficiency, and is fully supervised by 3D bounding box annotations without reliance on ground-truth depth or Truncated Signed Distance Fields (TSDF) (Zhang et al., 24 Jul 2025).

1. Architectural Overview

SGCDet operates in three sequential modules: 2D feature extraction, adaptive 3D volume construction, and a detection head. For 2D feature extraction, a ResNet-50 backbone with Feature Pyramid Network (FPN) yields multi-scale feature maps from $N$ posed RGB images $\{I_n\}$. Adaptive 3D volume construction begins by estimating per-view depth distributions $D_n\in\mathbb{R}^{H\times W\times D}$ using DepthNet to encode geometric priors. The coarse-to-fine sparse volume construction process iteratively refines a voxel grid, starting from a low resolution $$V_0\in\mathbb{R}^{X/2^L\times Y/2^L\times Z/2^L\times C}$$, with each stage $l$ upsampling, predicting occupancy $O_l = \mathcal{F}(V_l^{init})$, and aggregating features for the k\%-highest-probability voxels $P_l$ using the geometry and context aware aggregation module before updating voxel features. The detection head is an anchor-free 3D head outputting per-voxel centerness, classification, and box regression signals.

2. Geometry and Context Aware Aggregation

The core operation in SGCDet is the mapping of rich multi-view 2D features into the 3D voxel space via an adaptive, attention-based lifting procedure. For a 3D voxel center $p=(x,y,z)^\top$, SGCDet computes its projection into each view $n$:

$p_n = (u_n, v_n, d_n)^\top = K_n E_n [p; 1]$

where $\{I_n\}$0/$\{I_n\}$1 are camera intrinsics/extrinsics. 2D features $\{I_n\}$2 are lifted to per-pixel-per-depth space as:

$\{I_n\}$3

Within each image, intra-view deformable aggregation replaces rigid sampling at $\{I_n\}$4 with a sum over $\{I_n\}$5 offset positions, using trilinear sampling $\{I_n\}$6:

$\{I_n\}$7

Offsets $\{I_n\}$8 and weights $\{I_n\}$9 are computed from $D_n\in\mathbb{R}^{H\times W\times D}$0 via a linear layer; $D_n\in\mathbb{R}^{H\times W\times D}$1 is a learned projection. Across $D_n\in\mathbb{R}^{H\times W\times D}$2 views, features are fused using multi-head attention with averaged query:

$D_n\in\mathbb{R}^{H\times W\times D}$3

$D_n\in\mathbb{R}^{H\times W\times D}$4

yielding the aggregated 3D voxel feature $D_n\in\mathbb{R}^{H\times W\times D}$5. This design enlarges the effective receptive field across the image set and accommodates occlusions and geometric uncertainty without requiring perfect depth estimates.

3. Sparse Volume Construction and Occupancy Prediction

SGCDet introduces a coarse-to-fine, sparse volumetric construction strategy to improve computational efficiency by targeting high-likelihood occupied regions. After each upsampling stage $D_n\in\mathbb{R}^{H\times W\times D}$6, a lightweight head $D_n\in\mathbb{R}^{H\times W\times D}$7 predicts occupancy probabilities $D_n\in\mathbb{R}^{H\times W\times D}$8 for all voxels:

$D_n\in\mathbb{R}^{H\times W\times D}$9

Occupancy pseudo-labels $$V_0\in\mathbb{R}^{X/2^L\times Y/2^L\times Z/2^L\times C}$$0 are generated from 3D bounding boxes, with $$V_0\in\mathbb{R}^{X/2^L\times Y/2^L\times Z/2^L\times C}$$1 if the center is inside any ground truth box and $$V_0\in\mathbb{R}^{X/2^L\times Y/2^L\times Z/2^L\times C}$$2 otherwise. The top $$V_0\in\mathbb{R}^{X/2^L\times Y/2^L\times Z/2^L\times C}$$3 entries in $$V_0\in\mathbb{R}^{X/2^L\times Y/2^L\times Z/2^L\times C}$$4 define $$V_0\in\mathbb{R}^{X/2^L\times Y/2^L\times Z/2^L\times C}$$5. Only $$V_0\in\mathbb{R}^{X/2^L\times Y/2^L\times Z/2^L\times C}$$6 undergoes the computationally intensive feature aggregation. This process yields resource savings while matching dense reconstructions in detection performance.

4. Training Paradigm

SGCDet is trained with supervision provided solely by 3D box annotations. Occupancy loss leverages pseudo-labels from 3D boxes rather than requiring ground-truth geometry such as TSDF or depth maps. The training objective comprises a detection loss and an occupancy loss:

$$V_0\in\mathbb{R}^{X/2^L\times Y/2^L\times Z/2^L\times C}$$7

Detection loss is split into three components: $$V_0\in\mathbb{R}^{X/2^L\times Y/2^L\times Z/2^L\times C}$$8 (binary cross-entropy for centerness), $$V_0\in\mathbb{R}^{X/2^L\times Y/2^L\times Z/2^L\times C}$$9 (IoU-based regression), and $l$0 (focal loss for classification). Occupancy loss is computed as:

$l$1

This design eliminates dependence on ground-truth geometric representations and allows supervision of the full pipeline based on box-level annotations.

5. Experimental Results and Ablations

SGCDet achieves state-of-the-art results on ScanNet, ScanNet200, and ARKitScenes. Key detection metrics (mean Average Precision, mAP) are summarized below. All results use voxelizations of $l$2 (ScanNet/ARKitScenes) or $l$3 (ScanNet200).

Dataset	Method	[email protected]	[email protected]
ScanNet	ImVoxelNet	46.7	23.4
	NeRF-Det	53.5	27.4
	MVSDet	56.2	31.3
	SGCDet	61.2	35.2
ARKitScenes	MVSDet	60.7	40.1
	SGCDet	62.3	44.7
	SGCDet-L	70.4	57.0

Significant findings from ablation studies include:

• Adding 3D deformable sampling improves ScanNet [email protected] by 3.5%, and multi-view attention provides a further +1.7%.
• Sparse volume reconstruction (2-stage, top-25% selection) matches dense performance while reducing training memory by 35% and increasing FPS by 40%.
• Removing occupancy loss impacts [email protected] by –6.7%.
• Excluding either mono-view or multi-view branches from DepthNet reduces mAP by 4–5 points.
• Using ground-truth depth provides an upper bound of (64.3, 42.3) on ScanNet.

6. Comparative Performance and Advantages

SGCDet provides both accuracy and efficiency advantages over previous voxel-based and multi-view detector baselines. Compared to MVSDet, SGCDet realizes gains of +5.0 [email protected] and +3.9 [email protected] on ScanNet, despite requiring no ground-truth geometry. Resource-wise, SGCDet reduces training memory by 42.9%, training time by 47.2%, inference memory by 50%, and increases inference speed by 40.8% (relative to MVSDet). Key contributors to these improvements are the geometry and context aware aggregation module—increasing receptive field and mitigating occlusion and geometric ambiguity—and the sparse construction focus on potentially occupied voxels. The design supports supervision with only 3D box labels, eliminating costly requirements for explicit scene geometry.

7. Implementation Details and Reproducibility

SGCDet operates with 40 input views during training and 100 during inference, using RGB images of $l$4 (features at $l$5). DepthNet infers depth in $l$6 bins over $l$7m, $l$8m], with $l$9 nearest-view cost volume support. Intra-view deformable aggregation uses $O_l = \mathcal{F}(V_l^{init})$0 points per view. Sparse volume construction is a two-stage process, selecting the top 25% of voxels at each stage. Model variants include SGCDet ($O_l = \mathcal{F}(V_l^{init})$1, $O_l = \mathcal{F}(V_l^{init})$2 volume, $O_l = \mathcal{F}(V_l^{init})$3m voxels) and SGCDet-L ($O_l = \mathcal{F}(V_l^{init})$4, $O_l = \mathcal{F}(V_l^{init})$5, $O_l = \mathcal{F}(V_l^{init})$6m voxels). Optimization is via AdamW (learning rate $O_l = \mathcal{F}(V_l^{init})$7, cosine decay); training proceeds for 12 epochs (ScanNet/ARKitScenes) or 30 epochs (ScanNet200) on NVIDIA A6000 GPUs. The complete source code, including training scripts, configuration files, and dataset preparation instructions, is available at https://github.com/RM-Zhang/SGCDet (Zhang et al., 24 Jul 2025).