Mem-ChangingGrounder: Memory-Driven 3D Grounding

• Mem-ChangingGrounder is a memory-centric, zero-shot agent for localizing objects in dynamic 3D scenes using a retrieval-driven framework.
• It integrates cross-modal vision-language reasoning with tailored exploration policies (OSS and SRAS) to reduce costly re-scans while enhancing accuracy.
• The framework sets a new baseline on the ChangingGrounding benchmark by achieving superior localization accuracy and significantly lowered motion and exploration costs.

Mem-ChangingGrounder (MCG) is a memory-centric, zero-shot agent for 3D visual grounding in dynamic, real-world environments, introduced within the "ChangingGrounding" benchmark. Addressing the challenge of localizing referred objects via natural-language queries in scenes that are likely to have experienced structural changes, MCG pioneers a retrieval-driven framework, using cross-modal vision-language reasoning and active, memory-aware exploration to minimize the need for costly re-scans while maximizing localization accuracy (Hu et al., 16 Oct 2025).

1. Problem Formulation and Task Inputs

The ChangingGrounding task frames 3D visual grounding (3DVG) as an active, memory-driven inference problem. The agent is provided with: (1) a natural-language query $D_n$ describing the target object (e.g., “the chair farthest from the table”), (2) a set of past RGB-D observations $M_p = \{(I_i^p, p_i^p)\}$ from a previous scene scan $S_p$, and (3) access to the current, unexplored scene $S_c$, which supports on-demand re-rendering. The objective is to return a 3D bounding box $B$ localizing the referred object in $S_c$, maximizing the grounding accuracy while minimizing both robot exploration (action) cost $C_a$ and motion cost $C_m$. The formal mapping is:

$$\langle S_p,\, S_c,\, M_p,\, D_n \rangle \rightarrow B$$

Metrics used include accuracy at intersection-over-union thresholds (Acc@$τ$ with $M_p = \{(I_i^p, p_i^p)\}$0), new observation count ($M_p = \{(I_i^p, p_i^p)\}$1), and total motion cost $M_p = \{(I_i^p, p_i^p)\}$2, the latter aggregating translation and rotation costs normalized by $M_p = \{(I_i^p, p_i^p)\}$3 m/s and $M_p = \{(I_i^p, p_i^p)\}$4 rad/s, respectively.

2. Mem-ChangingGrounder Architecture

MCG comprises four principal modules and two spatial exploration policies:

1. Query Classification: A vision-LLM (VLM; GPT-4.1) determines if the referring expression $M_p = \{(I_i^p, p_i^p)\}$5 is "verifiable" (spatial constraint likely unchanged) or "unverifiable," thereby selecting the operative retrieval policy.
2. Memory Retrieval & Grounding:
   • Representation: Memories $M_p = \{(I_i^p, p_i^p)\}$6 are stored as $M_p = \{(I_i^p, p_i^p)\}$7 pairs, with $M_p = \{(I_i^p, p_i^p)\}$8 containing RGB, depth, and camera intrinsics $M_p = \{(I_i^p, p_i^p)\}$9, and $S_p$0 encoding world-to-camera pose.
   • Detection Pre-filter: GroundingDINO performs open-vocabulary detection to retain frames containing anchor or target classes.
   • Cross-modal Scoring: Given an image $S_p$1 and tokenized query $S_p$2, embedding functions $S_p$3 and $S_p$4 provide features $S_p$5 and $S_p$6, with $S_p$7 ranking relevance.
   • Grounding Paths:
     • Verifiable: Selects $S_p$8 matching $S_p$9 via VLM, navigates to pose $S_c$0 in $S_c$1, verifies object persistence, and, if needed, adapts routing (switches to SRAS if the anchor is static, OSS otherwise).
     • Unverifiable: Locates anchor $S_c$2, visits $S_c$3, executes SRAS if anchor persists, or falls back to OSS from the scene center.
3. Exploration Policies:
   • Omnidirectional Scene Scanner (OSS): At the current pose $S_c$4, generates 20 yaw-rotated ($S_c$5 increments) -20° down-tilted camera views, stitches these images, and uses the VLM to select the optimal one.
   • Spatial Relation Aware Scanner (SRAS): For spatial queries, generates a localized ring of 20 camera poses around the anchor, biased by predicted relation direction (e.g., for “up,” sampled local $S_c$6 and $S_c$7 rotations), with the VLM selecting the best viewpoint.
4. Fallback Mechanism: If no explicit memory match is found, the agent selects the clearest memory frame containing the target class, navigates to its pose, and applies OSS as a universal search.

3. Multi-View Fusion and 3D Box Estimation

Once an optimal viewpoint is established via exploration, object localization proceeds as follows:

• Single-view Reference Cloud: GroundingDINO identifies the 2D bounding box in $S_c$8; SAM-Huge extracts the mask $S_c$9. Pixels $B$0 are projected into 3D using known depth $B$1 and camera intrinsics $B$2:

$B$3

forming a point set $B$4.

• Surround-view Scanning: The axis-aligned bounding box (AABB) of $B$5 provides a center $B$6 and diagonal $B$7. A ring of 16 poses (tilt $B$8, radius $B$9) is generated around $S_c$0; GroundingDINO finds candidate boxes in each, selecting those with centroids nearest to $S_c$1 to yield $S_c$2.
• Fusion & Outlier Removal: All sets $S_c$3 undergo outlier filtering by comparing cluster AABB volumes, discarding clusters with volume $S_c$4 the next smaller. Residual points $S_c$5 are fused, and the final 3D box $S_c$6 is fit by taking coordinate-wise min/max or by optimizing for minimal residual inclusion error.

4. Implementation and Pipeline Details

MCG operates in a zero-shot regime, requiring no training on the ChangingGrounding benchmark data; instead, it uses off-the-shelf VLMs (GPT-4.1, temp=0.1, top-p=0.3) for all reasoning. Detection is performed using GroundingDINO; segmentation uses SAM-Huge. Image stitching and ensemble predictions leverage up to $S_c$7 images and $S_c$8 ensembles. No cap is imposed on retrieval retries, as the fallback system addresses failure cases robustly. All rendering and navigation control occur within a simulation pipeline built atop the 3RScan mesh suite, employing ScanNet intrinsics ($S_c$9 resolution, $C_a$0, $C_a$1, $C_a$2).

5. Quantitative Performance and Empirical Analysis

Evaluation on ChangingGrounding demonstrates that MCG achieves the highest localization accuracy among tested methods while incurring substantially lower exploration and motion costs. Representative results are as follows:

	[email protected] (low-res)	[email protected] (low-res)	$C_a$3	$C_a$4	[email protected] (high-res)	[email protected] (high-res)	$C_a$5	$C_a$6
Wandering G.	24.8%	10.8%	44.2	17.51	32.4%	12.8%	44.2	17.51
Central Rot.	16.8%	6.0%	18.0	1.70	17.2%	6.8%	18.0	1.70
Memory-Only	20.8%	10.0%	0.0	0.0	26.0%	12.4%	0.0	0.0
MCG (ours)	29.2%	14.8%	8.5	9.7	36.8%	18.0%	8.5	9.8

Ablation studies (high-res) show that removing memory degrades accuracy by 1.6pp and raises action cost by a factor of 3.8; removing fallback has negligible effect, while introducing multi-scan alone improves [email protected] by 5.6pp but with higher $C_a$7. Upgrading to GPT-4.1 increases [email protected] by 5.2pp. No significant accuracy impact is observed when using rendered versus real memory images.

6. Context and Implications

By formulating 3D visual grounding as an active, memory-reliant process and demonstrating memory-guided zero-shot grounding in dynamic scenes, MCG signifies a marked shift from traditional point cloud-based approaches that require costly, frequent full scans. The modular MCG framework—composed of vision-language retrieval, targeted motion planning, and robust geometric fusion—serves as a strong reference baseline for subsequent research. A plausible implication is that future 3DVG architectures may increasingly incorporate memory-based scene representations and flexible retrieval, guided by advanced VLMs, to adapt efficiently to evolving environments (Hu et al., 16 Oct 2025).