ChangingGrounding: Dynamic 3D Visual Grounding

• ChangingGrounding is a benchmark for active, memory-driven 3D visual grounding that integrates past scene memories with dynamic exploration to localize objects.
• It formalizes the integration of cross-modal retrieval, spatial reasoning, and efficient exploration, using metrics like IoU and exploration cost to assess performance.
• The Mem-ChangingGrounder model leverages pre-trained encoders, heuristic navigation policies, and multi-view fusion to achieve robust, cost-efficient object localization.

ChangingGrounding is a task and benchmark in active, memory-driven 3D visual grounding under dynamically evolving 3D environments. It quantifies the ability of an agent to localize objects by natural-language description within a changing scene, using both previously acquired visual memories and targeted exploration in the current, possibly altered, spatial context. The framework formalizes and measures the integration of cross-modal retrieval, spatial reasoning, and efficient exploration, in contrast to classic 3D visual grounding approaches which assume a static, fully reconstructed scene. ChangingGrounding establishes a standard for evaluating the robustness, sample efficiency, and memory utilization of real-world 3D visual grounding agents (Hu et al., 16 Oct 2025).

1. Formal Definition and Problem Setup

ChangingGrounding involves the following inputs and outputs:

• Inputs:
  • A language description $D_c$ specifying the referent via natural-language spatial relations, e.g., “the chair farthest from the cabinet”.
  • A memory $M_p$ of the previous scene $S_p$, consisting of a set of RGB-D frames and their camera poses, i.e., $\{(I^p_i, D^p_i, p^p_i)\}_{i=1}^{N_p}$.
  • Access to an unexplored, possibly altered, current scene $S_c$, where the agent can move and acquire new RGB-D observations $\{(I^c_j, D^c_j, p^c_j)\}$.
• Output:
  • A 3D bounding box $B_{pred} \in \mathbb{R}^6$ (with $$(x_{min}, y_{min}, z_{min}, x_{max}, y_{max}, z_{max})$$) delineating the grounded object in $S_c$.

The scene change model assumes object instances $O=\{o_1,\ldots,o_K\}$ have known 3D poses in $M_p$0, and that in $M_p$1 some subset $M_p$2 have moved or rotated by more than thresholds $M_p$3 or $M_p$4, respectively. This formalism assesses grounding in genuinely dynamic scenes (Hu et al., 16 Oct 2025).

2. The ChangingGrounding Benchmark

The ChangingGrounding benchmark is constructed from 3RScan, providing 478 environments with 1,482 time-step scans, all aligned to a common world frame. Each environment features 209 richly annotated movable target categories (requiring at least four scenes and observed motion) and 24 anchor objects. Referential prompts are templated as $M_p$5 $M_p$6 $M_p$7, spanning five spatial relations: Horizontal Proximity, Vertical Proximity, Between, Support, and Between (excluding Allocentric relations in the current iteration).

The full dataset comprises 266,916 unique natural-language descriptions. Evaluation employs:

• Localization Accuracy: $M_p$8, defined as the fraction of samples where $M_p$9.
• Exploration Cost: $S_p$0 (reported in $S_p$1s), combining the count of RGB-D collection actions $S_p$2, and motion cost $S_p$3, with $S_p$4 the summed translation distances over time divided by velocity, and $S_p$5 the summed angular travel divided by angular velocity.

Classic 3D visual grounding datasets (e.g., ScanRefer, ReferIt3D) lack scene dynamics, while ChangingGrounding explicitly evaluates the agent’s capacity to utilize memory, infer objects’ relocation, and efficiently act under partial observability (Hu et al., 16 Oct 2025).

3. Mem-ChangingGrounder: Memory-Guided Active Grounding

Mem-ChangingGrounder (MCG), the reference method, integrates cross-modal retrieval, heuristic action policies, and multi-view 3D fusion without learning from task-specific data.

Key Algorithmic Stages:

• Query Encoding: Encodes $S_p$6 with a pre-trained CLIP text encoder $S_p$7; a classifier distinguishes “verifiable” (direct memory-based) from “unverifiable” (anchor-based) queries.
• Cross-Modal Memory Selection: Encodes prior images $S_p$8 via $S_p$9, computes $\{(I^p_i, D^p_i, p^p_i)\}_{i=1}^{N_p}$0 for retrieval, typically top-$\{(I^p_i, D^p_i, p^p_i)\}_{i=1}^{N_p}$1.
• Exploration Policies:
  • Omnidirectional Scene Scanner (OSS): For a given pose, samples 20 poses in a ring of $\{(I^p_i, D^p_i, p^p_i)\}_{i=1}^{N_p}$2 steps.
  • Spatial Relation Aware Scanner (SRAS): For parsed spatial relations, synthesizes view trajectories based on the anchor's last-known pose.
• Active Grounding Pipeline:
  • If the target is verifiable, MCG attempts direct memory-based retrieval, verifying in $\{(I^p_i, D^p_i, p^p_i)\}_{i=1}^{N_p}$3; on failure, it initiates explorative SRAS around the anchor.
  • For unverifiable queries, the anchor is first retrieved and confirmed in $\{(I^p_i, D^p_i, p^p_i)\}_{i=1}^{N_p}$4, then relation-specific exploration localizes the target.
  • On all failures, the fallback triggers open-vocabulary detection and scanning from the “clearest” prior memory.
• Multi-View Fusion:
  • Segments the target in each view (e.g., via SAM), projects 2D mask pixels to 3D using depth and intrinsics, collects point clouds $\{(I^p_i, D^p_i, p^p_i)\}_{i=1}^{N_p}$5 across views, retains those whose centroids are within 0.25 m of the target center, and finally fuses $\{(I^p_i, D^p_i, p^p_i)\}_{i=1}^{N_p}$6.
  • Bounding box $\{(I^p_i, D^p_i, p^p_i)\}_{i=1}^{N_p}$7 is computed as the axis-aligned box encapsulating $\{(I^p_i, D^p_i, p^p_i)\}_{i=1}^{N_p}$8 (Hu et al., 16 Oct 2025).

4. Experimental Results and Ablation Analysis

Table: Core performance metrics (high-res setting):

Method	[email protected] (%)	$\{(I^p_i, D^p_i, p^p_i)\}_{i=1}^{N_p}$9 ($S_c$0s)
Wandering Grounding	32.4	17.51
Central Rotation	17.2	1.70
Memory-Only	26.0	0.00
Mem-ChangingGrounder	36.8	9.76

Mem-ChangingGrounder achieves the highest observed grounding accuracy (36.8% at $S_c$1) while consuming less than 10k seconds of motion cost, demonstrating over 2x reduction in exploration cost compared to naïve wandering (Hu et al., 16 Oct 2025).

Ablation investigations reveal:

• Removing the memory module increases motion cost (18.60) and reduces accuracy to 35.2%.
• Single-view fusion (no multi-view) reduces [email protected] to 28.0%.
• Outlier filtering in fusion restores full accuracy.
• The fallback stage has small impact on average but is necessary for edge cases.

5. Insights, Limitations, and Future Directions

Principal limitations of Mem-ChangingGrounder include:

• High dependence on the integrity of visual-linguistic matching (CLIP/GPT-4), leading to sensitivity to semantic drift or perceptual artifacts.
• Rendered RGB-D (simulated data) introduces some noise; application to real sensor data is expected to amplify artifacts.
• Errors in 2D detection and segmentation propagate into the 3D fusion stage.

Potential extensions include:

• End-to-end fine-tuning of grounding heads on fused 3D clouds to leverage task-specific features.
• Training compact policy networks to decide between memory lookup and active exploration.
• Expanding to allocentric relations and richer scene dynamics (e.g., object-level insertions/deletions, lighting changes).
• Addressing sim-to-real transfer via hybrid real and rendered memory pools.
• Integrating high-throughput vision-LLMs (e.g., FastVLM) for low-latency operation.

The paradigm demonstrably enables robust, cost-efficient grounding in highly dynamic spatial contexts. The framework can be directly leveraged for service robots in logistics, home assistance, and warehouse operations, where minimizing active exploration and costly re-scans is critical in non-static environments.

6. Significance and Impact on 3D Visual Grounding Research

ChangingGrounding advances the field by imposing strict benchmarking on closed-loop performance in non-stationary 3D domains, which more accurately models real-world deployment scenarios than prior work in static 3D visual grounding. It forces a methodological shift toward memory-centric reasoning, cross-modal inference, and adaptive, sample-efficient exploration policies. By providing both an expansive, large-scale dataset and evaluation metrics that penalize excessive exploration, it catalyzes progress in scaling 3D vision and LLMs from toy scenes to open-world environments with realistic temporal-spatial evolution. This framework represents a foundation for future research into scalable, robust, and efficient 3D understanding agents (Hu et al., 16 Oct 2025).