AgentGrounder: Zero-Shot 3D Visual Grounding

• AgentGrounder is a zero-shot 3D visual grounding framework that uses colored point clouds and a two-stage pipeline with an offline Object Lookup Table and an online tool-driven agent.
• It decomposes natural language queries into candidate retrieval, geometric reasoning, and optional rendering to efficiently localize objects without task-specific 3D training.
• Empirical results show accuracy gains over baselines by reducing context overload and cascading errors, validating its practical impact in complex 3D scenes.

AgentGrounder is a zero-shot 3D visual grounding framework that localizes an object in a 3D scene from a natural-language description by operating directly on colored point clouds and by delegating query decomposition, candidate retrieval, geometric reasoning, and optional visual disambiguation to a tool-driven multimodal language-model agent. In its named formulation, the system builds an offline Object Lookup Table and then performs online grounding without task-specific 3D training, with the stated goals of reducing prompt overload from irrelevant objects and avoiding cascading errors from early anchor-target matching (Huynh et al., 25 May 2026). The label also appears more loosely in adjacent literature: one training-free visual grounding paper notes that “AgentGrounder” is effectively a naming variant for its method “GroundingAgent” (Luo et al., 24 Nov 2025).

1. Definition and nomenclature

In the specific sense established by the paper titled "AgentGrounder: Zero-Shot 3D Visual Pointcloud Grounding using Multimodal LLMs" (Huynh et al., 25 May 2026), AgentGrounder denotes a two-stage 3D visual grounding system. Its offline stage applies a 3D model to a colored point cloud to build an Object Lookup Table (OLT) containing instance IDs, semantic labels, centers, and axis-aligned 3D boxes. Its online stage uses a tool-driven agent to decompose a query, retrieve relevant candidates from the OLT, compute geometric scores, and render object-centric images only when the query requires additional visual evidence such as color, material, texture, or viewpoint-sensitive cues.

The surrounding literature uses the term more broadly. The visual grounding method "GroundingAgent" explicitly states that “AgentGrounder” appears to be a naming variant and retains its own title for consistency (Luo et al., 24 Nov 2025). Other papers use the term as a conceptual umbrella for agent-based grounding systems in 3D perception, geo-localization, GUI grounding, deterministic evaluation, and epistemically constrained coding. This suggests that the term now has both a narrow referent—a particular zero-shot 3DVG method—and a broader architectural connotation centered on explicit tool use, selective evidence acquisition, and grounded decision procedures.

2. Task setting and motivation

AgentGrounder addresses 3D Visual Grounding (3DVG): given a 3D scene $S$ and a natural-language description $Q$, the system must return the target object’s 3D bounding box. In the zero-shot setting, the grounding pipeline avoids task-specific 3D supervision and instead uses pretrained components and general-purpose multimodal LLMs (Huynh et al., 25 May 2026).

The method is motivated by two limitations of straightforward LVLM-based 3D grounding. First, rendering many scene views and prompting a model over all visible objects tends to overload the context window with irrelevant entities and increase latency. Second, pipelines that commit early to an anchor-target match can propagate errors into later rendering and final selection. The same paper also argues that standard 3D segmentation tools expose only closed-set semantic labels and basic geometry, so naively feeding their output wholesale to an LVLM forces the model to sift through a large object set (Huynh et al., 25 May 2026).

A nearby zero-shot line is represented by "VLM-Grounder," which performs 3D grounding from multi-view 2D RGB images, depth maps, and calibrated camera parameters, dynamically stitches image sequences, and lifts 2D masks to 3D through multi-view projection rather than operating on point clouds directly (Xu et al., 2024). The contrast is structural: VLM-Grounder reasons over image evidence first and reconstructs the 3D box afterward, whereas AgentGrounder keeps a point-cloud-derived object inventory explicit from the start.

3. Offline representation: the Object Lookup Table

The offline stage of AgentGrounder begins with 3D instance segmentation over a colored point cloud. In the reported implementation, Mask3D produces instance masks and semantic labels,

$$\mathrm{Seg}(S) \rightarrow \{(\mathrm{mask}_i,\mathrm{sem}_i)\}_{i=1}^N.$$

For each instance, the method computes an axis-aligned box from per-axis point extents. If $P_i=\{p_k\in\mathbb{R}^3\}$ denotes the points of instance $i$, then the box is determined by $$x_{\min}^i,x_{\max}^i,y_{\min}^i,y_{\max}^i,z_{\min}^i,z_{\max}^i$$, with center

$$c_i=\left[\frac{x_{\min}^i+x_{\max}^i}{2},\frac{y_{\min}^i+y_{\max}^i}{2},\frac{z_{\min}^i+z_{\max}^i}{2}\right]$$

and size

$$d_i=[x_{\max}^i-x_{\min}^i,\; y_{\max}^i-y_{\min}^i,\; z_{\max}^i-z_{\min}^i].$$

The resulting OLT is stored as

$O=\{(id_i,\ell_i,c_i,d_i)\}_{i=1}^N.$

No per-object visual attributes or feature embeddings are cached; when the query uses synonyms not present in $\ell_i$, label mismatch is handled online through semantic mapping by the LVLM (Huynh et al., 25 May 2026).

This representation is deliberately compact. The paper characterizes the OLT as a light, structured object metadata table with a small memory footprint and $Q$0 storage. The design significance is operational rather than merely representational: once the OLT exists, the agent can avoid prompting over the full scene and can perform deterministic spatial computations before any visual rendering is invoked.

4. Online agent loop and scoring logic

The online stage runs on Qwen3-VL-32B-Instruct, served locally through Ollama and orchestrated with LangChain. The agent first writes a plan and extracts candidate labels $Q$1 and spatial predicates $Q$2 from the query,

$Q$3

Selective retrieval then constructs a candidate set

$Q$4

where $Q$5 extends the extracted labels with synonym mappings proposed by the LVLM. If the query is “the red chair,” the initial candidates are all chairs; if a term such as “sofa” is absent from the scene labels, the agent may map it to a stored label such as “couch” before continuing (Huynh et al., 25 May 2026).

For each candidate, AgentGrounder computes a deterministic geometric score. With Euclidean distance

$Q$6

the paper defines hinge-style per-predicate scores on $Q$7 for relations such as left, right, in front, behind, on top of, near, and between. The aggregate geometric score is

$Q$8

optionally augmented by a normalized size term derived from box volume when the query specifies size. The top geometric candidate is

$Q$9

Rendering is triggered only if the query contains explicit visual attributes, viewpoint-sensitive phrasing, or small score margins among the top candidates. The agent then renders object-centric views for a small subset $$\mathrm{Seg}(S) \rightarrow \{(\mathrm{mask}_i,\mathrm{sem}_i)\}_{i=1}^N.$$0, prompts the LVLM to choose the best match, and converts the result into a visual score $$\mathrm{Seg}(S) \rightarrow \{(\mathrm{mask}_i,\mathrm{sem}_i)\}_{i=1}^N.$$1. Final selection is

$$\mathrm{Seg}(S) \rightarrow \{(\mathrm{mask}_i,\mathrm{sem}_i)\}_{i=1}^N.$$2

with heuristic default $$\mathrm{Seg}(S) \rightarrow \{(\mathrm{mask}_i,\mathrm{sem}_i)\}_{i=1}^N.$$3 when visual inspection is used (Huynh et al., 25 May 2026).

The output is a structured answer containing the selected instance ID, the corresponding 3D box from the OLT, and a concise rationale. The reported argument for this loop is that it reduces context-window pressure by sending only a few relevant objects and, when necessary, only a few rendered views to the multimodal model.

5. Evaluation and empirical findings

AgentGrounder is evaluated on ScanRefer and Nr3D in a zero-shot protocol. Offline Mask3D segmentation is run once per scene to build the OLT; the online agent then grounds each query using Qwen3-VL-32B without task-specific 3DVG training. Evaluation uses 3D IoU and accuracy at threshold $$\mathrm{Seg}(S) \rightarrow \{(\mathrm{mask}_i,\mathrm{sem}_i)\}_{i=1}^N.$$4,

$$\mathrm{Seg}(S) \rightarrow \{(\mathrm{mask}_i,\mathrm{sem}_i)\}_{i=1}^N.$$5

For axis-aligned boxes, the 3D IoU is computed from axis-wise overlap extents and box volumes (Huynh et al., 25 May 2026).

Benchmark / split	SeeGround	AgentGrounder
ScanRefer [email protected] overall	39.4	41.9
ScanRefer Unique	68.9	73.7
ScanRefer Multiple	30.0	31.7
Nr3D overall	46.1	52.4
Nr3D Hard	38.3	45.4
Nr3D Easy	54.5	59.6
Nr3D View-Independent	48.2	54.5
Nr3D View-Dependent	42.3	47.9

In the reported setup, these correspond to a $$\mathrm{Seg}(S) \rightarrow \{(\mathrm{mask}_i,\mathrm{sem}_i)\}_{i=1}^N.$$6 [email protected] gain on ScanRefer and a $$\mathrm{Seg}(S) \rightarrow \{(\mathrm{mask}_i,\mathrm{sem}_i)\}_{i=1}^N.$$7 overall gain on Nr3D over SeeGround, with additional improvements on Unique, Multiple, Hard, Easy, View-Independent, and View-Dependent splits (Huynh et al., 25 May 2026).

Ablation on a held-out Nr3D subset of $$\mathrm{Seg}(S) \rightarrow \{(\mathrm{mask}_i,\mathrm{sem}_i)\}_{i=1}^N.$$8 isolates the role of the agent’s tools. Retrieval-only yields overall $$\mathrm{Seg}(S) \rightarrow \{(\mathrm{mask}_i,\mathrm{sem}_i)\}_{i=1}^N.$$9 [email protected]; adding distance reasoning raises this to $P_i=\{p_k\in\mathbb{R}^3\}$0; adding planning reaches $P_i=\{p_k\in\mathbb{R}^3\}$1; adding rendering to retrieval plus distance gives $P_i=\{p_k\in\mathbb{R}^3\}$2 overall and increases View-Dependent accuracy from $P_i=\{p_k\in\mathbb{R}^3\}$3 to $P_i=\{p_k\in\mathbb{R}^3\}$4; the full toolset reaches $P_i=\{p_k\in\mathbb{R}^3\}$5 overall, with complementary gains from geometry, planning, and selective rendering (Huynh et al., 25 May 2026).

The implementation is reported on a workstation with two NVIDIA RTX 4090 GPUs. Qwen3-VL-32B is served locally via Ollama, orchestration uses LangChain, and rendering is carried out by a lightweight point-cloud renderer centered on candidate boxes. The paper identifies rendering on demand as the dominant additional cost, but argues that restricting rendering to top-$P_i=\{p_k\in\mathbb{R}^3\}$6 candidates keeps latency manageable.

6. Position within the agentic grounding landscape

Recent work places AgentGrounder within a larger family of systems that treat grounding as an agentic process rather than a single forward pass. This suggests a broader pattern: a controller decomposes the query, invokes specialized tools, and grounds its output through explicit evidence paths or constraints.

System	Domain	Core grounding mechanism
VLM-Grounder (Xu et al., 2024)	Zero-shot 3DVG	Dynamic multi-view stitching, feedback, 2D-to-3D lifting
GroundingAgent (Luo et al., 24 Nov 2025)	REC	OVD + MLLM descriptions + LLM selection
SpotAgent (Jia et al., 10 Feb 2026)	Geo-localization	ReAct with web search, geocoding, image zoom
GUI-C$P_i=\{p_k\in\mathbb{R}^3\}$7 (Li et al., 29 May 2026)	GUI grounding	Area-gated coarse-to-fine refinement with GRPO
Phi-Ground (Zhang et al., 31 Jul 2025)	Computer-use grounding	Planner-generated Long RE plus coordinate prediction
GroundEval (Flynt, 22 Jun 2026)	Agent evaluation	Deterministic trajectory and evidence-path scoring
GROUNDING.md blueprint (Palmblad et al., 23 Apr 2026)	Agentic coding	Hard Constraints and Convention Parameters

Within 3D grounding, VLM-Grounder and AgentGrounder represent two different zero-shot decompositions. VLM-Grounder relies on stitched 2D images, a grounding-and-feedback loop, SAM-based segmentation, PATS matching, and multi-view ensemble projection; it reports 51.6% [email protected] on ScanRefer and 48.0% accuracy on Nr3D without point clouds or object priors (Xu et al., 2024). AgentGrounder instead begins with a point-cloud OLT and uses rendering only when geometry is insufficient (Huynh et al., 25 May 2026).

In 2D referring expression comprehension, GroundingAgent provides a training-free pipeline that composes open-vocabulary detectors, Llama-3.2-11B-Vision region descriptions, and a reasoning LLM; it reports an average zero-shot accuracy of 65.1% across RefCOCO, RefCOCO+, and RefCOCOg, and notes that replacing noisy region captions with the original query lifts the selection stage to about 90.6% (Luo et al., 24 Nov 2025). In geo-localization, SpotAgent formalizes grounding as a POMDP and a ReAct loop over WebSearch, GeoCoding, and ImageTool, achieving state-of-the-art retrieval-free performance on Im2GPS3k and large gains on the text-centric SVT benchmark (Jia et al., 10 Feb 2026).

In GUI grounding, GUI-C$P_i=\{p_k\in\mathbb{R}^3\}$8 treats cropping as a finite-horizon agent decision process and combines difficulty-aware sample weighting with deterministic area-gated refinement, reaching 46.4% average accuracy on ScreenSpot-Pro for the 3B model (Li et al., 29 May 2026). Phi-Ground, by contrast, advocates a two-stage planner-to-grounder workflow in which a stronger MLLM generates a “Long RE” and a compact grounding model predicts coordinates, reporting 55.0 on ScreenSpot-Pro and 36.2 on UI-Vision in the agent setting for the 7B-16C-DPO variant (Zhang et al., 31 Jul 2025).

Two further extensions broaden the meaning of groundedness itself. GroundEval replaces LLM-as-judge with deterministic scoring of what an agent searched, fetched, cited, and was permitted to access, arguing that final-answer plausibility cannot verify valid evidence use (Flynt, 22 Jun 2026). The GROUNDING.md proposal moves the concept into agentic coding by defining field-scoped Hard Constraints and Convention Parameters that override project- and session-level instructions (Palmblad et al., 23 Apr 2026).

7. Limitations, evaluation challenges, and future directions

For the named 3DVG system, reported failure cases include ambiguous language, occlusions or sparse scans, highly similar objects such as many identical chairs or cabinets, and material or color confusion when point-cloud rendering provides weak lighting or shading cues. The method also depends on offline instance segmentation quality: poor masks or mislabels propagate into the OLT and affect all subsequent steps. The paper further assumes a stable scene coordinate frame with $P_i=\{p_k\in\mathbb{R}^3\}$9 up and relies on labels being approximately correct, with LVLM-based synonym mapping handling only part of the mismatch problem (Huynh et al., 25 May 2026).

The paper’s stated extension directions include open-vocabulary 3D detectors for improved labels and instance discovery, learning-to-rank models that combine geometry and visual signals more finely, richer spatial-relation models over OLT objects, improved agent planning policies for when and how much to render, and interactive human-in-the-loop clarification for genuinely ambiguous scenes (Huynh et al., 25 May 2026). These proposals remain within the same design philosophy: keep the object inventory explicit, make tool use selective, and invoke expensive visual reasoning only when deterministic geometry is insufficient.

A broader issue emerges when AgentGrounder is interpreted as an agentic grounding paradigm rather than a single method. Deterministic evaluation work argues that answer correctness should be judged against the evidence path that produced it, including what the agent was permitted to access and whether required searches were actually performed (Flynt, 22 Jun 2026). A plausible implication is that future AgentGrounder-style systems may be evaluated not only by endpoint localization accuracy, but also by the inspectability and validity of their retrieval, rendering, and reasoning trajectories.