Group3D: Multi-View Open-Vocabulary 3D Detection

• Group3D is a multi-view open-vocabulary 3D detection framework that integrates semantic constraints during instance construction for improved object detection.
• It leverages a training-free, end-to-end pipeline using off-the-shelf models such as GPT-5.1, SAM 3, and Depth Anything 3 to generate scene-adaptive vocabularies and reliable segmentation.
• Experimental results on ScanNet and ARKitScenes demonstrate that semantic gating effectively reduces over-merging errors and enhances detection accuracy in zero-shot settings.

Searching arXiv for the Group3D paper and closely related open-vocabulary 3D detection context. Group3D is a multi-view open-vocabulary 3D object detection framework that integrates semantic constraints directly into instance construction rather than assigning categories only after geometry-based merging. It operates on RGB observations of a static indoor scene, supports both pose-known and pose-free settings, and is presented as a self-contained, zero-shot pipeline in which all major components are frozen off-the-shelf models rather than newly trained modules. The central premise is that multi-view 3D instance formation should be constrained simultaneously by geometric consistency and by semantic compatibility derived from a multimodal LLM (MLLM), so that category variability across views can be absorbed without permitting semantically implausible fragment associations (Kim et al., 23 Mar 2026).

1. Problem setting and motivation

Open-vocabulary 3D object detection seeks to localize and recognize objects beyond a fixed training taxonomy. In multi-view RGB settings, recent approaches often decouple geometry-based instance construction from semantic labeling, first generating class-agnostic fragments and then assigning open-vocabulary categories post hoc. According to the Group3D formulation, this decoupling leaves instance construction governed primarily by geometric consistency, without semantic constraints during merging (Kim et al., 23 Mar 2026).

The paper identifies a specific failure mode in this design choice. When geometric evidence is view-dependent and incomplete, geometry-only merging can produce irreversible association errors, including over-merging of distinct objects and fragmentation of a single instance. The method is therefore motivated by the claim that semantic information should enter at merge time rather than only at the final labeling stage. In the Group3D design, semantic compatibility is encoded through MLLM-produced groups over a scene-adaptive vocabulary, and these groups constrain which fragments may be merged.

This suggests a shift in emphasis within open-vocabulary 3D detection: semantic uncertainty is not treated merely as a late-stage classification problem, but as a structural constraint on 3D instance assembly itself. A plausible implication is that the framework is particularly suited to settings in which cross-view geometric overlap is weak or reconstruction quality is imperfect, since these are precisely the regimes in which unconstrained geometric aggregation is most brittle.

2. End-to-end processing pipeline

Group3D takes as input $N$ RGB images of a static indoor scene, optionally with known camera poses $\{T_n\}$. The paper describes five major stages, followed by multi-view evidence accumulation for final label selection and box construction (Kim et al., 23 Mar 2026).

First, a scene vocabulary is constructed via an MLLM. For each view $I_n$, the MLLM, specified as GPT-5.1, is prompted to list up to $K=5$ dominant object categories. The returned terms are canonicalized by lower-casing and removing modifiers, and then unioned across views to form the Scene Vocabulary Memory $V$.

Second, Group3D performs category-aware 2D segmentation. For each $\ell_i \in V$ and each image $I_n$, SAM 3 is run to produce a binary mask $m_{n,i}(u,v)\in\{0,1\}$ and a score

$s_{n,i}=s^{pres}_n \cdot s^{query}_{n,i}.$

Third, the segmented regions are lifted into 3D fragments. Depth $D_n(u,v)$ and pose $\{T_n\}$0 are obtained either from ground-truth inputs or from reconstruction backbones, specifically Depth Anything 3 or VGGT in the pose-free setting. Each mask pixel is back-projected according to

$\{T_n\}$1

and the fragment is defined as

$\{T_n\}$2

The fragments memory is stored as

$\{T_n\}$3

Fourth, the scene vocabulary is partitioned into semantic compatibility groups using the same MLLM. Fifth, fragments are merged under a group-gated procedure: fragments are sorted by size, and a new fragment $\{T_n\}$4 is merged into an existing cluster $\{T_n\}$5 only if the fragment label and cluster label belong to the same semantic group and the fragment and cluster satisfy the geometric overlap criterion. Otherwise a new cluster is initialized.

After merging, each cluster $\{T_n\}$6 contains a point set $\{T_n\}$7 and a multiset of labels $\{T_n\}$8. Group3D then accumulates multi-view evidence for each candidate label $\{T_n\}$9 using

$I_n$0

followed by

$I_n$1

where $I_n$2 is a hyperparameter. The final category is selected as

$I_n$3

with score $I_n$4, and the 3D box is the axis-aligned bounds of $I_n$5 (Kim et al., 23 Mar 2026).

3. Scene-adaptive vocabulary and semantic grouping

A defining component of Group3D is its use of a scene-adaptive vocabulary rather than a fixed category list. For each image, the MLLM is prompted with the instruction: “You are identifying the dominant object categories present… Return a single comma-separated line containing at most five object categories.” The per-view category hypotheses are then canonicalized and deduplicated over the $I_n$6 views to produce

$I_n$7

The paper states that this scene-specific set focuses both the segmentation queries and the subsequent grouping (Kim et al., 23 Mar 2026).

The semantic grouping stage is designed to address taxonomy noise, exemplified by category pairs such as “sofa” and “couch.” Group3D does not perform k-means or graph clustering. Instead, it prompts the same MLLM with an instruction of the form: “Group categories that could plausibly refer to the same physical object… Constraints: no structural attachments, no part-whole…” The output is a partition $I_n$8 over the vocabulary.

The induced group assignment function is

$I_n$9

with singleton groups for terms that are never merged. The grouping is explicitly defined as a partition:

$K=5$0

This grouping mechanism is more permissive than strict same-category matching and more selective than unconstrained geometry-only association. In the paper’s terms, it encodes plausible cross-view category equivalence. A plausible implication is that the MLLM is being used not primarily as a detector, but as a scene-level semantic regularizer that supplies a compatibility structure over otherwise noisy viewwise category predictions.

4. Merge-time constraints and instance construction

The merge-time criterion in Group3D is the conjunction of geometric consistency and semantic compatibility. This is the core technical mechanism by which semantic information is injected into 3D instance construction (Kim et al., 23 Mar 2026).

For geometric consistency, point sets are voxelized with voxel size $K=5$1:

$K=5$2

The paper defines two overlap measures:

$K=5$3

and

$K=5$4

Boolean geometric consistency is then

$K=5$5

if

$K=5$6

and otherwise $K=5$7, with $K=5$8 and $K=5$9.

Semantic compatibility is binary:

$V$0

The combined gating function is

$V$1

or equivalently

$V$2

The operational effect is that fragments are associated only when they satisfy both semantic compatibility and geometric consistency. The paper attributes two complementary benefits to this design. First, it mitigates geometry-driven over-merging. Second, it absorbs multi-view category variability by allowing labels that are semantically compatible, rather than only identical, to support the same 3D instance. The qualitative examples reported for ScanNet20 and ScanNet200 state that Group3D recovers chairs, sofas, and desks even under partial views where geometric IoU is low, and that over-merging failures such as merging a chair and an adjacent table are largely eliminated (Kim et al., 23 Mar 2026).

5. Training-free design and implementation parameters

Group3D does not train any new network end-to-end with 3D supervision. The paper explicitly states that all components, including the MLLM, SAM 3, and the depth/pose reconstructor, are frozen off-the-shelf. Consequently, there are no detection, grouping, or gating losses to specify (Kim et al., 23 Mar 2026).

The implementation details listed in the paper are central to reproducibility. The input consists of $V$3 uniformly sampled frames, resized to $V$4 for reconstruction. The number of category hypotheses per view is fixed at $V$5. The MLLM is GPT-5.1 in the main configuration, with Qwen3-VL-8B used in ablation. SAM 3 is the segmentation backbone, with Grounded SAM 2 used in ablation. In pose-free reconstruction, Depth Anything 3 is used for ScanNet and VGGT for ARKitScenes. The voxel size and gating thresholds are $V$6, $V$7, and $V$8. The evidence weight function is

$V$9

with $\ell_i \in V$0 set to reward more than one view. All experiments are reported as running on NVIDIA A6000, and no fine-tuning is required.

Because the system is fully zero-shot and training-free, its performance depends on the interplay among prompt-derived vocabulary construction, category-aware mask generation, geometric lifting, and semantically gated aggregation. This suggests that Group3D is best understood as a compositional framework rather than a newly optimized detector head. A plausible implication is that its empirical behavior reflects the calibration and failure modes of its constituent frozen modules, especially the MLLM grouping stage and the quality of pose-free reconstruction.

6. Evaluation, ablations, and empirical profile

The evaluation protocol covers ScanNet V2 under three vocabularies—ScanNet20, ScanNet60, and ScanNet200—and ARKitScenes. Metrics are 3D mAP at IoU thresholds $\ell_i \in V$1 and $\ell_i \in V$2, denoted $\ell_i \in V$3 and $\ell_i \in V$4. The reported regimes include pose-known and pose-free configurations, as well as zero-shot evaluation without 3D training on target classes. Comparisons are made against point-cloud-based open-vocabulary detectors such as PointCLIPv2 and OV-Uni3DETR, and against multi-view pipelines including OpenM3D and Zoo3D (Kim et al., 23 Mar 2026).

The paper reports the following key quantitative results.

Setting	Baseline(s)	Group3D
ScanNet20, pose known, zero-shot	Zoo3D₀: 30.5/17.3; Zoo3D₁: 32.8/15.5	51.1/27.4
ScanNet60	Zoo3D₀: 22.0/10.4	29.1/13.9
ScanNet200, pose-free	Zoo3D₁: 16.5/6.3	17.9/8.7
ARKitScenes	Zoo3D₁: 16.1/3.5	20.5/5.9

These results are presented as evidence that Group3D achieves state-of-the-art performance in multi-view open-vocabulary 3D detection while showing strong generalization in zero-shot scenarios (Kim et al., 23 Mar 2026).

The ablation studies isolate the contribution of semantic gating. In the paper’s Tab. 5, removing semantic gating and merging all fragments by geometry gives $\ell_i \in V$5, strict same-category merging gives $\ell_i \in V$6, and full semantic compatibility grouping gives $\ell_i \in V$7, corresponding to a gain of $\ell_i \in V$8 over strict same-category merging. Sensitivity analyses report nearly identical performance for $\ell_i \in V$9, namely $I_n$0 versus $I_n$1. The MLLM ablation reports GPT-5.1 at $I_n$2 versus Qwen3-VL-8B at $I_n$3. Reconstruction ablation reports Depth Anything 3 at $I_n$4 versus VGGT at $I_n$5. Segmentation ablation reports SAM 3 at $I_n$6 versus Grounded SAM 2 at $I_n$7 (Kim et al., 23 Mar 2026).

The qualitative analysis further emphasizes the role of semantic gates in noisy or incomplete geometric regimes. In pose-free settings, where reconstruction noise is high, the paper states that semantic gates prevent spurious merges that would otherwise collapse distinct objects. Taken together, these findings support the narrower claim that the main gain comes from integrating language-driven semantic compatibility into the merge operation itself, rather than from introducing new learned 3D supervision or novel loss functions.