Task-Adaptive 3D Grounding Advances Vision-Language Models

• Task-adaptive 3D grounding is a family of vision-language methods that dynamically align query structures with 3D scene outputs to handle diverse interactions.
• It adapts granularity, modality, and output structure via specialized decoders and routing to support phrase-level, dense, and affordance-based grounding.
• Empirical results show improved accuracies—up to 4.6% gains on benchmarks like ScanRefer, Nr3D, and Sr3D—demonstrating robust performance in complex scenes.

Task-adaptive 3D grounding is a family of 3D vision-language formulations in which the grounded target, the supervision signal, the reasoning pathway, or the output structure changes with the task specification rather than remaining fixed as a single “one sentence to one object” mapping. Across recent work, this adaptation appears in several forms: phrase-aware grounding that augments target prediction with phrase-object alignment maps, unified architectures that switch between grounding and captioning by changing masks and lightweight heads, paragraph-level dense grounding over multiple objects, interaction-conditioned affordance grounding from egocentric video, language-guided affordance heatmap prediction over object point clouds, hierarchical multi-view grounding in autonomous driving, and instruction-conditioned functionality segmentation and metric grounding in 3D scenes (Yuan et al., 2022, Chen et al., 2022, Huang et al., 2023, Liu et al., 2024, Zhu et al., 7 Apr 2025, Li et al., 28 Mar 2025, Lin et al., 24 Mar 2026, Harsh et al., 6 Jul 2026).

1. From fixed referent localization to adaptive grounding formulations

Early 3D visual grounding is summarized in one paper as a task where a 3D scene is encoded into object proposals $O$, a query is encoded into token features $E$, and the model predicts a grounding score vector $S \in \mathbb{R}^{M}$ over proposals. A central criticism is that this formulation supervises only the final target object even when the sentence names multiple objects and relations, which is particularly limiting in cluttered 3D scenes with long-range dependencies (Yuan et al., 2022). That critique motivates a broader shift in which the grounding problem itself is adapted to the structure of the query, the modality of the query, or the task output.

One line of work adapts granularity. “Toward Explainable and Fine-Grained 3D Grounding through Referring Textual Phrases” defines 3D Phrase Aware Grounding (3DPAG), extending the output from $(S)$ to $(S, C)$, where $C \in \mathbb{R}^{M \times (L+1)}$ is a phrase-object alignment map with an extra [NoObj] slot (Yuan et al., 2022). “Dense Object Grounding in 3D Scenes” instead adapts from single-sentence grounding to 3D Dense Object Grounding (3D DOG), where a paragraph $S=\{S^k\}_{k=1}^K$ yields a set of target boxes $B^k$, and the model assumes that dense objects described in the same paragraph are semantically related and spatially concentrated (Huang et al., 2023).

Another line adapts the conditioning signal itself. “Grounding 3D Scene Affordance From Egocentric Interactions” formulates grounding over samples $\{\mathcal{S},\mathcal{V},y\}$, where the output is a fine-grained 3D affordance mask $\hat{\mathcal{M}}_a \in \mathbb{R}^{N \times 1}$ and an affordance class $E$0; the same scene may yield different grounded regions depending on the demonstrated interaction (Liu et al., 2024). “Grounding 3D Object Affordance with Language Instructions, Visual Observations and Interactions” similarly asks for a point-wise affordance probability map $E$1 conditioned on language instructions, interaction images, and point clouds rather than on a referring sentence alone (Zhu et al., 7 Apr 2025). “Mono3DVG: 3D Visual Grounding in Monocular Images” adapts the problem to monocular RGB input and geometry-rich text, explicitly arguing that monocular 3D grounding should exploit both appearance and geometry information in language descriptions (Zhan et al., 2023).

A further extension adapts grounding to downstream conversational or embodied requirements. “Ground3D-LMM: Fine-Grained 3D Point Grounding and Spatial Reasoning with LMM” defines 3D Grounded Measurement, where the model must identify a point-level 3D mask $E$2 and answer with physical quantities in real-world units (Harsh et al., 6 Jul 2026). “UniFunc3D: Unified Active Spatial-Temporal Grounding for 3D Functionality Segmentation” casts functionality segmentation as instruction-conditioned grounding of affordance-bearing regions in a reconstructed scene, while “Direct Action-Head Injection of A Grounded 3D Point Unlocks Spatial and Task Generalization” uses a single grounded 3D target point as the active sub-goal signal for a robot policy (Lin et al., 24 Mar 2026, Tsai et al., 26 Jun 2026). This suggests that task-adaptive 3D grounding has become a general interface between language, geometry, and action rather than a narrowly defined retrieval problem.

2. Adaptive axes: granularity, modality, structure, and output space

The literature converges on several recurring adaptation axes. One paper states them explicitly for 3DPAG: granularity (phrase-level instead of sentence-only), supervision (phrase-object labels), reasoning (target and non-target object interactions), and training objective (phrase-object alignment plus phrase-specific pre-training) (Yuan et al., 2022). Closely related work on paragraph grounding adds multiple-target adaptation, local-global refinement, scene-region adaptation, and inter-object dependency adaptation through contextual query generation and proposal-guided decoding (Huang et al., 2023).

A second axis is task unification under shared multimodal representations. “UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Grounding” treats 3D dense captioning and 3D visual grounding as two sides of the same multimodal scene understanding problem. The same multimodal fusion transformer parameters are reused while the task changes through attention masks, visual cues, and lightweight heads: bidirectional grounding uses fused box tokens $E$3, whereas captioning uses autoregressive text tokens $E$4 (Chen et al., 2022). This suggests that part of task adaptation in 3D grounding is architectural, with the heavy representation learner shared and only small task interfaces changing.

A third axis is modality adaptation. Some methods remain language-conditioned but enrich the visual side with geometry-aware branches, as in Mono3DVG-TR with a depth predictor, dual text-guided adapter, and depth-text-visual stacking attention (Zhan et al., 2023). Others change the query modality itself: Ego-SAG uses egocentric video to derive an interaction intention embedding $E$5 that guides 3D scene decoding, while LMAffordance3D fuses interaction images, point clouds, and language instructions with a frozen llava-v1.6-vicuna-7b backbone (Liu et al., 2024, Zhu et al., 7 Apr 2025). “NuGrounding” shifts the setting to multi-view autonomous driving and conditions localization on hierarchical prompts built from category, appearance, movement, and ego-relative relationship attributes (Li et al., 28 Mar 2025).

A fourth axis is output structure. Some models output a single 3D box, some a set of boxes, some phrase-level alignment maps, some point-wise or superpoint masks, and some joint text-plus-region outputs. “Ground3D-LMM” introduces a segmentation-aware trigger <SEG> attached to grounded phrases, coupling autoregressive answers with point-level mask decoding (Harsh et al., 6 Jul 2026). “GR3D” places explicit 2D grounding, implicit 2D grounding, and monocular 3D grounding in one framework, allowing the model to emit HTML-style 2D boxes, insert region tokens during chain-of-thought, or predict 3D boxes in camera view from grounded regions (Cheng et al., 28 May 2026). “N3D-VLM” serializes native 3D grounding as bbox(id, class, u, v, z, s_x, s_y, s_z), using structured 3D box language as a bridge between localization and reasoning (Wang et al., 18 Dec 2025). A plausible implication is that output heterogeneity is now a defining property of task-adaptive grounding systems rather than an exception.

3. Architectural mechanisms for adaptation

Several papers implement task adaptation through specialized decoders or query-routing mechanisms. “PD-APE: A Parallel Decoding Framework with Adaptive Position Encoding for 3D Visual Grounding” decomposes grounding into a target-object branch and a surrounding branch. The main branch uses target tokens $E$6 and box-surface-relative position encoding, while the surrounding branch uses context tokens $E$7 and an additional text-guided bias: $E$8 with $E$9 (Hou et al., 2024). The reported interpretation is that object-centric and context-centric reasoning should not be forced into one diffuse attention map.

“AugRefer” addresses adaptation through both data and decoding. Its Language-Spatial Adaptive Decoder (LSAD) applies language cross-attention, pairwise spatial attention, and global spatial attention in sequence, with the best order reported as language $S \in \mathbb{R}^{M}$0 pairwise $S \in \mathbb{R}^{M}$1 global. Global spatial attention uses

$S \in \mathbb{R}^{M}$2

where $S \in \mathbb{R}^{M}$3 is derived from normalized scene coordinates, while pairwise spatial attention injects relative geometry between candidate objects (Wang et al., 16 Jan 2025). In this design, the decoder adapts object reasoning to both the expression and the scene layout.

Other systems adapt by routing or gating. “NuGrounding” introduces two decoupled task tokens, [DET] and [EMB], plus a learnable context query $S \in \mathbb{R}^{M}$4. The context query is updated autoregressively after the model emits [DET], then used to select top-$S \in \mathbb{R}^{M}$5 semantically relevant 3D object queries before fusion decoding (Li et al., 28 Mar 2025). “OmniEVA: Embodied Versatile Planner via Task-Adaptive 3D-Grounded and Embodiment-aware Reasoning” proposes a Task-Adaptive Gated Router (TAGR) that takes a sentence embedding $S \in \mathbb{R}^{M}$6 and a pooled scene descriptor $S \in \mathbb{R}^{M}$7, predicts logits $S \in \mathbb{R}^{M}$8, samples a binary gate $S \in \mathbb{R}^{M}$9 with Gumbel-Softmax, and forms

$(S)$0

thereby injecting 3D positional information only when task and scene conditions warrant it (Liu et al., 11 Sep 2025).

A different adaptation mechanism appears in reasoning-oriented models. “ReasonGrounder” uses an LVLM to infer a target object $(S)$1 and explanation $(S)$2 from an implicit query $(S)$3, selects a reference view by CLIP similarity, estimates a task-relevant physical scale $(S)$4, recomputes Gaussian instance features $(S)$5, clusters them with HDBSCAN, and selects the Gaussian group whose cluster center best matches the query-relevant feature (Liu et al., 30 Mar 2025). “GR3D” adapts during generation itself: when an entity mention appears, the model predicts a 2D box, encodes the corresponding region token, inserts it into the text stream, and then continues reasoning conditioned on that grounded visual evidence (Cheng et al., 28 May 2026). This suggests a broader split between externally routed adaptation and generation-time adaptive grounding.

4. Data resources and evaluation protocols

Task-adaptive 3D grounding has been driven by new datasets that expose missing supervision regimes. For phrase-aware grounding, one paper manually labeled about 227K phrase-level annotations from 88K sentences across Nr3D, Sr3D, and ScanRefer, enabling phrase-object supervision and phrase-specific pre-training (Yuan et al., 2022). For dense paragraph grounding, 3DOGSFormer constructs paragraph-level benchmarks from ScanRefer, Nr3D, and Sr3D by grouping 2 \le K \le 12 sentence-level targets around a focused object and arranging them into a paragraph (Huang et al., 2023).

Affordance grounding datasets change both modality and target. VSAD pairs egocentric videos from Ego4D and EPIC-Kitchens-100 with 3D indoor scenes from ScanNetV2 and Matterport3D, yielding 3,814 egocentric video clips, 2,086 3D indoor scenes, 7,690 interactive areas, 17 affordance categories, and 16 interaction targets (Liu et al., 2024). AGPIL contains 41,628 point clouds, 30,972 images, 30,972 language instructions, 23 object classes, and 17 affordance types, with full-view, partial-view, and rotation-view settings plus seen and unseen protocols (Zhu et al., 7 Apr 2025).

New application domains require new benchmark structures. NuGrounding is built from 34,149 key frames from 850 videos in NuScenes and generates 2.2M textual prompts, averaging 63.7 prompts per frame, using a Hierarchy of Grounding (HoG) over four attribute families with

$(S)$6

template combinations (Li et al., 28 Mar 2025). GroundedAR-Bench evaluates live AR grounding over 214 scenes, 1,736 annotated objects, and 1,120 language queries with tasks spanning identification, relation grounding, tool-based queries, and assistive queries (Guo et al., 29 Nov 2025). ReasoningGD contributes over 10K scenes and 2 million annotations for open-vocabulary 3D grounding and amodal perception, while Ground3D-LMM introduces a large-scale dataset with roughly 2.5M question-answer pairs spanning eight tasks on ScanNet and ScanNet++ (Liu et al., 30 Mar 2025, Harsh et al., 6 Jul 2026). These benchmark designs suggest that adaptation is increasingly evaluated as a property of the task distribution itself, not merely of model architecture.

5. Empirical patterns and representative results

Across formulations, adaptive grounding mechanisms usually help most when contextual reasoning, strict localization, or modality mismatch is severe. In 3DPAG, phrase-aware supervision improves conventional 3DVG performance, with the paper reporting overall accuracy gains of 3.9% on Nr3D, 3.5% on Sr3D, and 4.6% on ScanRefer over the previous state of the art (Yuan et al., 2022). In PD-APE, separating target-object and surrounding-layout decoding yields stronger gains at stricter IoU thresholds: on ScanRefer, Overall [email protected]: 56.84 and Overall [email protected]: 45.65, surpassing EDA by +2.25 and +3.19, while the Hard subset on Nr3D reaches 50.6 (Hou et al., 2024).

Multi-target and multi-task adaptation show similar patterns. 3DOGSFormer reaches 64.73 @0.25 and 49.29 @0.5 overall on ScanRefer dense grounding, improving over the strongest dense baseline by 6.84 and 5.11 points, and also reports 73.5 on Nr3D and 80.8 on Sr3D (Huang et al., 2023). UniT3D emphasizes qualitative gains through a shared multimodal transformer across captioning and grounding, while DA4LG reports 83.8% and 86.8% accuracy on SNARE single-view and multi-view settings through multi-task domain adaptation with a visual adapter (Chen et al., 2022, Sun et al., 2024).

Interaction-conditioned and affordance-centric settings show especially large gaps over semantics-only baselines. On VSAD, Ego-SAG reaches 10.652 mAP, 18.243 AP50, 24.036 AP25, 36.829 mRC, 52.616 RC50, and 66.523 RC25, outperforming the strongest external baseline OpenIns3D by +2.304 mAP, +8.243 AP50, and +13.474 AP25 (Liu et al., 2024). In AGPIL, the paper states that LMAffordance3D is effective “even in unseen experimental settings,” but the provided text does not enumerate benchmark scores (Zhu et al., 7 Apr 2025). In AR, Words into World reports 5.4 ± 2.1 cm mean 3D localization error, 88.7% Success@10 cm, and 0.79 relation edge F1, while the full agent reduces median end-to-end latency from 6.59 s to 4.74 s relative to the no-coordinator ablation (Guo et al., 29 Nov 2025).

Grounding-first spatial reasoning frameworks also show that explicit grounded structure improves downstream reasoning. N3D-VLM-7B reports 89.7 open-ended and 92.1 numerical accuracy on N3D-Bench, while its grounding on RefCOCO reaches IoU 0.59 and offset 0.06, compared with Qwen3-VL-8B at 0.37 and 0.16 (Wang et al., 18 Dec 2025). Ground3D-LMM evaluates grounded measurement with mask IoU and metric criteria jointly, using GM-$(S)$7 with mask IoU $(S)$8 and $(S)$9 (Harsh et al., 6 Jul 2026). GR3D obtains 25.40 $(S, C)$0 on Omni3D and shows that 2D→3D outperforms direct 3D prediction, for example improving SUN $(S, C)$1 from 20.27 to 29.87 (Cheng et al., 28 May 2026). These results suggest a consistent empirical rule: adaptive grounding is most valuable when the model must decide not only where the target is but what type of grounding intermediate is needed to answer the task.

6. Limitations, controversies, and emerging directions

Several recurring limitations are explicit across papers. Many methods still depend on external parsing, masks, or proposal quality. PD-APE inherits text splitting from an off-the-shelf semantic decoupling tool, and its adaptation remains architectural rather than supervisory (Hou et al., 2024). ReasonGrounder depends on SAM masks, CLIP features, a pretrained Gaussian scene, and LVLM target inference; errors in any of these components can propagate into 3D grouping (Liu et al., 30 Mar 2025). GR3D notes that failures are mostly “missing or ambiguous 2D grounding,” indicating that the 2D-to-3D decomposition remains vulnerable to upstream errors (Cheng et al., 28 May 2026).

A second limitation is that some adaptive settings are still narrow in scope. Ego-SAG grounds affordance regions from egocentric video, but the task is still specifically video-conditioned affordance grounding rather than arbitrary instruction-conditioned policy grounding (Liu et al., 2024). LMAffordance3D models interaction through RGB images and language rather than explicit temporal dynamics or contact states (Zhu et al., 7 Apr 2025). UniFunc3D is training-free and active over frame selection, but the paper highlights failure on extremely small functional parts occupying less than 0.1% of image area and on scenes with severe occlusion (Lin et al., 24 Mar 2026). Ground3D-LMM supports object and part masks plus metric outputs, but the dataset framing indicates shared backbone adaptation through instruction following rather than specialized task heads, which may constrain expert specialization (Harsh et al., 6 Jul 2026).

A third limitation is that “adaptation” itself is heterogeneous. In some papers it means task-conditioned outputs; in others, shared architectures with changed masks; in others, controller-based tool orchestration; in others, action-head injection of a grounded 3D point. OmniEVA’s TAGR is a binary sample-level gate over 3D positional fusion rather than token-level routing (Liu et al., 11 Sep 2025). Words into World uses a controller policy rather than learned routing, and its dynamic scene graph is not a full dense 3D reconstruction (Guo et al., 29 Nov 2025). “Direct Action-Head Injection of A Grounded 3D Point Unlocks Spatial and Task Generalization” assumes that an adequate 2D grounding signal is already available and focuses on how to represent and inject it: $(S, C)$2 (Tsai et al., 26 Jun 2026). This suggests that current task-adaptive 3D grounding research spans at least four partially distinct agendas: finer supervision, task-unified representation learning, embodied grounding interfaces, and open-world or reasoning-centric grounding.

Future directions are mostly stated in the papers themselves. Ego-SAG suggests dynamic adaptive query strategies and stronger geometric modeling (Liu et al., 2024). AGO identifies prompt richness and temporal sequence integration as open issues for open-world occupancy (Li et al., 14 Apr 2025). N3D-VLM implies further scaling of lifted 3D data and broader category coverage (Wang et al., 18 Dec 2025). A plausible implication is that the field is moving toward systems in which grounded 3D structure is not only an output but a reusable computational substrate for explanation, reasoning, and action.