Sequential Grounding in the Wild Benchmark

• The paper demonstrates that SG3D formalizes sequential grounding by decomposing daily activities into multiple atomic steps with explicit temporal dependencies.
• It integrates multi-modal 3D scene graphs from diverse real-world datasets and uses GPT-4 generated instructions to ensure coherent, realistic task flows.
• Evaluation metrics such as step and task accuracy highlight challenges in reference resolution, underscoring the need for advanced, context-aware reasoning models.

Sequential grounding in the wild refers to a class of benchmarks that evaluate an agent’s ability to localize or interact with objects or spatial regions over a sequence of temporally dependent decisions, typically following complex, multi-step instructions grounded in real-world environments. Contemporary research in this area, exemplified by the SG3D benchmark for 3D task-oriented sequential grounding, moves beyond static object-centric evaluation and targets robustness in models’ capacity for context-aware, multi-step reasoning critical for embodied AI, multimodal LLMs, and HCI systems (Zhang et al., 2024, Lin et al., 26 Jun 2025, Shakeel et al., 20 Mar 2026).

1. Benchmark Scope and Motivation

Sequential grounding tasks require agents to iteratively interpret stepwise natural language instructions and localize or act upon targets within evolving world representations. This problem emerges when object references become contextually intertwined (e.g., pronoun use, disambiguation) or when successful completion at each step is contingent on prior actions. Traditional visual grounding benchmarks focus on single-step, static-object localization and thus fail to characterize difficulties inherent in practical, task-driven and temporally structured interaction scenarios (Zhang et al., 2024).

SG3D is the first large-scale real-world benchmark that formalizes task-oriented sequential grounding in 3D, encapsulating 22,346 daily activity tasks decomposed into 112,236 atomic, ordered steps across 4,895 complex real-world scenes. Other domains, such as clinical GUIs (MedSPOT (Shakeel et al., 20 Mar 2026)) or dialogue act grounding (Rifts (Shaikh et al., 18 Mar 2025)), similarly construct sequential protocols to measure error propagation, context maintenance, and workflow reliability.

2. Dataset Construction and Task Design

Data Sources and Representations:

SG3D samples RGB-D scene reconstructions from five large-scale 3D datasets (ScanNet, ARKitScenes, HM3D, 3RScan, MultiScan), each represented as an object-centric 3D scene graph where nodes correspond to instances and edges encode spatial relations. Each scene contains an average of 12–41 object instances and is annotated with automatically generated object captions (Zhang et al., 2024).

Task Generation Pipeline:

Tasks are generated by prompting GPT-4 with scene graph features to elicit a diverse set of realistic daily activities, each decomposed into ≤10 atomic steps. Each instruction step unambiguously references a single object instance (e.g., “[cabinet-7]”), and resulting scripts are subject to human verification. Quality assurance enforces semantic relevance, sequential coherence, and sufficiency for action localization.

Source Dataset	# Scenes	Objects/scene (avg)
ScanNet	693	30.7
ARKitScenes	1,575	12.1
HM3D	2,038	31.0
3RScan	472	31.5
MultiScan	117	40.8

Example Task:

• “Make me a cup of coffee.”
  1. Go to the long desk against the wall. [cups-15]
  2. Pick up a white plastic cup on the desk. [cups-15]
  3. Fill it with coffee at the coffee maker. [coffee_maker-16]
  4. Walk to the side table beside the cabinet. [table-23]
  5. Place the cup on the table. [table-23]

The mean number of steps per task is ≈5.03, and task instruction+steps average 70.5 words, explicitly targeting multi-step reasoning.

3. Formal Problem Formulation

Given a 3D scene $S$ and an instruction sequence $I = \{i_1, \ldots, i_T\}$, the goal is to predict a grounding sequence $G = \{g_1, \ldots, g_T\}$, where each $g_t$ denotes the object location or ID corresponding to instruction $i_t$:

$f : (S, I) \to G$

A stepwise state-transition process is adopted:

$$g_t = f_\mathrm{step}(i_t, S, \{g_1, \ldots, g_{t-1}\})$$

or, introducing an internal state $h_{t-1}$:

$$(g_t, h_t) = F(i_t, S, h_{t-1}), \quad h_0 = \emptyset$$

This structure explicitly models temporal dependencies between steps, requiring agents to remember, resolve, or disambiguate references stemming from previous grounding outputs.

4. Evaluation Metrics

Sequential grounding benchmarks employ per-step and whole-task metrics:

• Step Accuracy (s-acc):

$$s\text{-}acc = \frac{1}{\sum_\text{tasks} T} \sum_\text{tasks} \sum_{t=1}^{T} 1[\hat g_t = g_t]$$

Quantifies average per-step detection/localization correctness.

• Task Accuracy (t-acc):

$I = \{i_1, \ldots, i_T\}$0

Only counts a task as correct if every step is grounded correctly.

• Navigation Metrics (optional):

For agent embodiment, path-based measures (traveled distance $I = \{i_1, \ldots, i_T\}$1 vs. shortest path $I = \{i_1, \ldots, i_T\}$2) can be incorporated.

This dual-metric regime enables precise diagnosis of error propagation: step-level failures typically accumulate, dramatically reducing the fraction of tasks solved end-to-end (Zhang et al., 2024).

5. Benchmarking Methodologies and Baselines

Adapted Baselines:

• 3D-VisTA (Dual-Stream): Processes object tokens with a 3D spatial transformer and task language with BERT, fusing modalities in a unified transformer.
• PQ3D (Query-Based): Employs CLIP for text, point-feature processing for 3D, and object queries.
• SG-LLM (LEO): Extends frozen LLMs with object “grounding tokens,” injecting PointNet++-derived object features via a grounding head (MLP scores per object). The [GRD] token guides prediction at each step.

Training Objectives:

• Grounding loss: Cross-entropy over $I = \{i_1, \ldots, i_T\}$3 object candidates.
• Action generation loss (SG-LLM only): Encourages correct intermediate textual actions.
• Total loss combines the above terms, weighted by a scalar $I = \{i_1, \ldots, i_T\}$4.

Results Table (averaged across scenes):

Method	Zero-Shot s-acc	Zero-Shot t-acc	Fine-Tune s-acc	Fine-Tune t-acc
3D-VisTA	26.9%	4.7%	60.9%	30.6%
PQ3D	29.7%	4.1%	57.3%	26.8%
SG-LLM (LEO)	61.2%	25.7%	62.8%	34.1%
GPT-4 w/ GT	73.4%	46.6%	—	—

Fine-tuning improves s-acc to ~60% but t-acc remains <35%, exposing persistent challenges in step coherence and reference resolution.

6. Error Analysis and Data Efficiency

Sequential context:

Removing previous-step history reduces t-acc by 3–5 points for LEO and 3D-VisTA, confirming models exploit temporal dependencies to some degree.

Data efficiency:

SG-LLM maintains >50% s-acc using only 25% of SG3D’s training data, whereas dual-stream and query-based methods degrade faster with data reduction (Zhang et al., 2024).

Qualitative error typologies:

• Misreferencing visually similar objects
• Pronoun or anaphora confusion (e.g., losing track of “it” across steps)
• Failing to leverage commonsense cues (e.g., object affordances not derived from language alone)

7. Impact, Limitations, and Future Directions

Impact:

SG3D bridges the methodological gap between static visual grounding datasets and embodied planning benchmarks by focusing exclusively on sequential object localization grounded in real-world 3D environments. By decoupling grounding from low-level motion control, SG3D facilitates fine-grained error attribution within the perception-language stack of embodied agents.

Limitations and Recommended Extensions:

• Incorporation of navigation waypoints, enabling measurement of execution efficiency and safety.
• Expansion to manipulation, linking grounded object IDs directly to platform-specific pick and place actions.
• Enrichment with complex instructions, including logical constructs (conditionals, loops, disjunctions).
• Evaluation in real/virtual robots to test sim-to-real grounding transfer.
• Application of advanced LLM prompting (chain-of-thought, reflection) to combat cumulative error propagation and reference drift.

SG3D is poised to catalyze progress at the intersection of 3D vision, sequential multimodal language understanding, and task-oriented embodied intelligence. Its strict metrics and large-scale real-world design address previously uncharacterized bottlenecks in multi-step grounding under rich, realistic conditions (Zhang et al., 2024).