UniGround: Training-Free 3D Visual Grounding

• The paper introduces a training-free framework for 3D visual grounding that localizes objects via geometric-semantic scene parsing and multi-view 2D segmentation.
• UniGround decomposes the task into global candidate filtering and local precision grounding, using off-the-shelf 2D and vision-language models without 3D-specific training.
• The method achieves robust zero-shot performance on benchmarks by effectively generalizing across varied sensor inputs and complex spatial scenes.

Searching arXiv for the specified UniGround paper to ground the article in the latest source. {"query":"(Zhang et al., 9 Mar 2026) UniGround Universal 3D Visual Grounding via Training-Free Scene Parsing", "max_results": 5} arXiv search: UniGround 3D visual grounding paper and closely related entries. UniGround is a training-free framework for 3D visual grounding (3DVG), the task of localizing an object in a 3D scene from a natural-language description by predicting a 3D oriented bounding box. In the formulation used by the paper, the scene is represented as a colored point cloud together with first-person RGB-D frames with known camera poses, the input is a free-form natural-language instruction, and the output is a 3D bounding box with center, orientation, and dimensions. UniGround deliberately avoids any 3D task-specific learning: it builds a geometric-semantic scene representation from off-the-shelf 2D and vision-LLMs plus explicit 3D topology, and then performs reasoning with a Vision-LLM (VLM) over multi-scale prompts. The paper frames this design as a move from “open-vocabulary” 3DVG to “open-world” 3DVG, with the objective of grounding any object in any scene as long as the scene can be reconstructed (Zhang et al., 9 Mar 2026).

1. Problem formulation and conceptual framing

UniGround defines 3DVG as a zero-shot task with respect to 3D grounding datasets: no training or finetuning is done on ScanRefer, EmbodiedScan, or other 3D datasets for grounding. This zero-shot condition applies not only to language matching but also to 3D perception, because the system does not rely on a supervised 3D detector or 3D instance segmentation backbone tied to a fixed taxonomy (Zhang et al., 9 Mar 2026).

The motivation is a critique of prevailing 3DVG pipelines. Existing methods commonly follow either a detect-match strategy or an end-to-end strategy in which 3D perception is delegated to supervised 3D instance segmentation or detection models, often trained on closed-set 3D datasets with limited object classes and scene layouts. The paper identifies two consequences. First, generalization in geometry is limited: novel object categories, novel shapes, reconstruction noise, different sensor characteristics, and different layouts can cause fragmented, missing, or merged instances. Second, spatial reasoning is constrained: even when a VLM is used, it is typically restricted to the candidate set produced by a closed-set detector, and global context is often represented by only a few renders from arbitrary fixed viewpoints.

A central distinction in the paper is between open-vocabulary and open-world 3DVG. Open-vocabulary 3DVG can handle arbitrary text queries because it uses large VLMs, but it still depends on a closed-set supervised 3D perception module. Open-world 3DVG, by contrast, removes dataset-specific 3D segmentation and detection training, derives 3D instances from training-free 2D segmentation plus 3D topology and multi-view semantics, and imposes no architectural or training constraints tied to a specific 3D dataset or object taxonomy. A common misconception is that open-vocabulary capability on the language side is sufficient for open-world grounding; UniGround argues that geometry must also be open-world, otherwise the VLM remains bounded by the candidate set produced by inherited 3D perception.

2. System architecture and training-free design

UniGround decomposes grounding into two stages: Global Candidate Filtering and Local Precision Grounding. The first stage constructs 3D object candidates using a training-free 2D-to-3D instance segmentation pipeline plus multi-view semantic encoding, then ranks them against the query. The second stage applies multi-scale visual prompting and structured reasoning over the reduced candidate set to identify the target object precisely (Zhang et al., 9 Mar 2026).

The pipeline is training-free in 3D throughout. Stage One uses SAM, VCCS, region growing, and a Perception Encoder (PE), all pre-trained on 2D or vision-language tasks. Stage Two uses a large VLM, with GPT-5 as the main reasoning backbone in the primary experiments and GPT-4o, GLM-4.5, and Qwen-2.5 in ablations. No backpropagation or additional training on 3D datasets is performed.

This modularization separates geometric candidate discovery from high-level semantic and relational reasoning. Geometry is derived from explicit 3D topology and 2D segmentation consistency rather than a learned 3D detector. Semantics are supplied by PE embeddings and VLM reasoning rather than by class labels inherited from a supervised 3D model. This suggests a deliberate decoupling of 3D parsing and language reasoning: Stage One maximizes candidate recall under domain shift, while Stage Two expends expensive VLM reasoning only on a small candidate set.

3. Global Candidate Filtering

Stage One begins from a colored point cloud and partitions it into superpoints using VCCS plus region growing. These superpoints are intended to preserve local geometric continuity, approximate planarity and smoothness, and color or appearance consistency. They function as over-segmented geometric primitives that can subsequently be merged into object instances (Zhang et al., 9 Mar 2026).

The core instance-construction mechanism is training-free 2D-to-3D segmentation via multi-view SAM consistency and 3D topology. For each pair of adjacent superpoints, the method considers all camera views in which both are visible, applies SAM to each RGB frame, computes visibility statistics and SAM-derived semantic features, and aggregates these signals into a pairwise similarity. Progressive merging is then applied over five stages, with the similarity threshold linearly relaxed from $0.9$ to $0.5$. The result is a set of 3D instance candidates, each represented by a point set and an implicit bounding box.

After instance construction, UniGround performs multi-view semantic encoding. Each 3D candidate is projected back into visible RGB frames using camera poses and depth maps, and the projected pixels are used as prompts to refine object masks with SAM. For semantic robustness, UniGround uses multi-scale crops: an object region is cropped and progressively rescaled back to the original resolution so that both fine details and broader contextual cues are preserved. These crops are encoded by the Perception Encoder, and the per-view embeddings are averaged across multiple views to suppress view-specific noise and form a stable object embedding. The implementation samples 10 visible views per candidate.

Candidate filtering is then reduced to cosine similarity in the PE joint image-text space. For candidate $i$ and query $q$,

$$s_i = \frac{\langle \mathbf{e}_i^{\text{obj}}, \mathbf{e}_q^{\text{text}} \rangle} {\|\mathbf{e}_i^{\text{obj}}\|_2 \, \|\mathbf{e}_q^{\text{text}}\|_2}.$$

Candidates are ranked by $s_i$, and the top-$u$ are retained for Stage Two, with $u=5$ for ScanRefer and $u=10$ for EmbodiedScan. At this point, each retained candidate has a 3D instance or bbox and a multi-view semantic embedding.

4. Local Precision Grounding and structured reasoning

Stage Two refines grounding by combining global spatial prompts with local candidate evidence. The global prompt is built from orbit renderings of the point cloud. If $c_i \in \mathbb{R}^3$ is the center of candidate $0.5$0, the reference point for rendering is

$0.5$1

and orbit cameras are placed according to

$0.5$2

In practice, UniGround uses three global viewpoints. The rendered images include a global 3D coordinate frame and unique visual IDs for candidates so that the VLM can interpret expressions such as “to the left of” or “in front of” in relation to indexed objects (Zhang et al., 9 Mar 2026).

Local evidence is not derived from point-cloud crops. The paper explicitly avoids direct point-cloud rendering for local semantics because such crops can be sparse and aliased. Instead, for each candidate, UniGround selects three first-person RGB frames in which the object occupies a large proportion of pixels and the camera poses are maximally separated in 3D. Each selected frame is annotated with a 2D bounding box around the candidate region. This constitutes the candidate-specific visual evidence $0.5$3.

Reasoning is organized as a structured protocol. The VLM first performs semantic reasoning by naming each candidate from $0.5$4,

$0.5$5

and aligning the set of inferred names with the text query through an internal MatchTarget step. It then performs spatial reasoning by parsing the query into a relation set $0.5$6, using the global prompt $0.5$7, an axis-language mapping $0.5$8, and the candidate set $0.5$9 to infer

$i$0

where $i$1 is the predicted candidate index and $i$2 is a textual explanation. Finally, closed-loop correction re-prompts the VLM with both global and local cues if $i$3 is inconsistent with the semantic naming result or otherwise invalid, and a projection step ensures that the final answer lies in the valid candidate set.

The paper describes this inference pattern as “multi-scale visual prompting & structured reasoning.” Its role is to combine macro-level layout evidence from orbit renders with micro-level semantic evidence from first-person crops. The ablations indicate that neither level is sufficient in isolation: global renders alone are weak for object identification, while local evidence without explicit relational prompting is weaker on complex spatial descriptions.

5. Benchmarks, zero-shot performance, and robustness

UniGround is evaluated on ScanRefer, EmbodiedScan, and a real-world benchmark. ScanRefer contains 51,500 descriptions over 800 ScanNet scenes, with a validation split of 9,508 queries and standard metrics [email protected] and [email protected]. EmbodiedScan contains 5,000+ scans, 1M language prompts, and 160K 3D bboxes; the authors select approximately 6,000 tasks from 30 ARKitScenes and report [email protected]. The real-world benchmark contains four indoor environments—Office, Lounge, Corridor, and Conference—with 25 queries over 5 target objects per environment (Zhang et al., 9 Mar 2026).

On ScanRefer, UniGround achieves $i$4 [email protected] and $i$5 [email protected]. On EmbodiedScan, it achieves $i$6 [email protected]. The paper states that this establishes a new state-of-the-art among zero-shot methods on EmbodiedScan without any 3D supervision. Relative to the best prior open-world method reported in the table, LLM-G with GPT-4T, UniGround improves by $i$7 points at [email protected] and $i$8 points at [email protected] on ScanRefer. On EmbodiedScan, UniGround’s $i$9 [email protected] exceeds the $q$0 reported for the fully supervised Embodied Perceptron.

The ScanRefer comparison is more nuanced when open-vocabulary zero-shot methods are included. UniGround’s $q$1 is better at [email protected] than ZSVG3D ($q$2) and SeeGround ($q$3), and is roughly on par with VoG ($q$4) at [email protected]. It is lower than SeqVLM ($q$5) and SPAZER ($q$6) on ScanRefer. The paper attributes this difference to the fact that those methods assume 3D bounding boxes and class priors from supervised detectors, whereas UniGround uses no 3D priors at all. On EmbodiedScan, where cross-dataset generalization becomes dominant, SeeGround drops to $q$7 [email protected] and SeqVLM to $q$8, while UniGround reaches $q$9. The paper interprets this as evidence that training-free geometry generalizes more effectively than ScanNet-trained 3D detectors when transferred to ARKitScenes.

The real-world evaluation is intended to test robustness under uncontrolled reconstruction conditions and substantial domain shift. UniGround records $$s_i = \frac{\langle \mathbf{e}_i^{\text{obj}}, \mathbf{e}_q^{\text{text}} \rangle} {\|\mathbf{e}_i^{\text{obj}}\|_2 \, \|\mathbf{e}_q^{\text{text}}\|_2}.$$0 in Conference, $$s_i = \frac{\langle \mathbf{e}_i^{\text{obj}}, \mathbf{e}_q^{\text{text}} \rangle} {\|\mathbf{e}_i^{\text{obj}}\|_2 \, \|\mathbf{e}_q^{\text{text}}\|_2}.$$1 in Lounge, $$s_i = \frac{\langle \mathbf{e}_i^{\text{obj}}, \mathbf{e}_q^{\text{text}} \rangle} {\|\mathbf{e}_i^{\text{obj}}\|_2 \, \|\mathbf{e}_q^{\text{text}}\|_2}.$$2 in Office, and $$s_i = \frac{\langle \mathbf{e}_i^{\text{obj}}, \mathbf{e}_q^{\text{text}} \rangle} {\|\mathbf{e}_i^{\text{obj}}\|_2 \, \|\mathbf{e}_q^{\text{text}}\|_2}.$$3 in Corridor. The corresponding results for SeeGround are $$s_i = \frac{\langle \mathbf{e}_i^{\text{obj}}, \mathbf{e}_q^{\text{text}} \rangle} {\|\mathbf{e}_i^{\text{obj}}\|_2 \, \|\mathbf{e}_q^{\text{text}}\|_2}.$$4, $$s_i = \frac{\langle \mathbf{e}_i^{\text{obj}}, \mathbf{e}_q^{\text{text}} \rangle} {\|\mathbf{e}_i^{\text{obj}}\|_2 \, \|\mathbf{e}_q^{\text{text}}\|_2}.$$5, $$s_i = \frac{\langle \mathbf{e}_i^{\text{obj}}, \mathbf{e}_q^{\text{text}} \rangle} {\|\mathbf{e}_i^{\text{obj}}\|_2 \, \|\mathbf{e}_q^{\text{text}}\|_2}.$$6, and $$s_i = \frac{\langle \mathbf{e}_i^{\text{obj}}, \mathbf{e}_q^{\text{text}} \rangle} {\|\mathbf{e}_i^{\text{obj}}\|_2 \, \|\mathbf{e}_q^{\text{text}}\|_2}.$$7, and for SeqVLM $$s_i = \frac{\langle \mathbf{e}_i^{\text{obj}}, \mathbf{e}_q^{\text{text}} \rangle} {\|\mathbf{e}_i^{\text{obj}}\|_2 \, \|\mathbf{e}_q^{\text{text}}\|_2}.$$8, $$s_i = \frac{\langle \mathbf{e}_i^{\text{obj}}, \mathbf{e}_q^{\text{text}} \rangle} {\|\mathbf{e}_i^{\text{obj}}\|_2 \, \|\mathbf{e}_q^{\text{text}}\|_2}.$$9, $s_i$0, and $s_i$1. The paper states that, on average, UniGround is more than $s_i$2 better than these open-vocabulary baselines in success rate. The explanation given is that real-world reconstructions introduce noise, partial scans, and pose drift, which degrade proposal quality for supervised 3D detectors, whereas UniGround’s training-free superpoint plus multi-view SAM segmentation remains more stable and Stage Two can verify and correct predictions through multi-scale prompting.

6. Ablations, limitations, and significance

The ablations are performed on 1,059 randomly sampled ScanRefer tasks across 20 scenes. Varying the number of candidates passed from Stage One to Stage Two reveals a rapid-growth regime for $s_i$3: [email protected] rises from $s_i$4 to approximately $s_i$5 at $s_i$6. For $s_i$7, improvements are slower, and beyond $s_i$8 the curve saturates; from $s_i$9 to $u$0, the gain is about $u$1. The default $u$2 is therefore a trade-off between recall and VLM cost (Zhang et al., 9 Mar 2026).

Prompting ablations show that the combination of spatial prompts, semantic prompts, and visual chain-of-thought is the strongest configuration. With GPT-5, the full configuration reaches $u$3 [email protected] and $u$4 [email protected], compared with $u$5 for spatial plus semantic without visual CoT, $u$6 for semantic plus visual CoT without spatial prompting, and $u$7 for spatial plus visual CoT without semantic evidence. The largest drop occurs when semantic evidence is removed, indicating that global renders alone are insufficient for candidate identification. Different VLM backbones also change performance materially: GPT-4o gives $u$8, GLM-4.5 $u$9, Qwen-2.5 $u=5$0, and GPT-5 $u=5$1. The paper interprets this as evidence that 3DVG is reasoning-intensive.

The paper identifies four limitations. First, computational cost remains substantial because Stage One applies SAM over many views and Stage Two uses multi-image prompts to a large VLM; the system is not real-time. Second, the method depends on reconstruction quality: sparse or mis-registered RGB-D reconstructions degrade superpoint segmentation and candidate projection. Third, errors and biases in upstream foundation models propagate into grounding, since SAM, PE, and the VLM are all used as-is. Fourth, complex relational queries and long reasoning chains remain difficult, especially when descriptions involve many references or temporal content.

The proposed future directions are the development of 3D-aware foundation models, more explicit spatial reasoning modules or symbolic planners that collaborate with VLMs, efficiency improvements toward real-time deployment in robots and AR devices, and extensions to richer tasks such as multi-object grounding and sequential grounding in dynamic scenes. Within that framing, UniGround’s main significance is methodological rather than taxonomic: it replaces learned 3D instance perception with training-free scene parsing, and uses VLM-based structured reasoning only after a geometry-aware candidate reduction stage. This suggests that competitive 3D grounding can be obtained without dataset-specific 3D supervision, provided that candidate construction, multi-view semantics, and relational prompting are sufficiently well aligned.