Reference-Frame Task Grounding

• Reference-Frame Task Grounding is the process of mapping perceptions, language, and trajectories onto semantically appropriate coordinate systems to ensure task-relevant inference.
• Recent methods employ explicit coordinate transforms, inferred latent frames, and soft logical compositions to facilitate accurate grounding in both spatial and temporal domains.
• Empirical studies show that performance in robotics and dialogue degrades with frame-selection errors, highlighting the need for robust, dynamic frame-switching strategies.

Reference-frame task grounding is the problem of grounding perception, language, demonstrations, or trajectories in the coordinate system that is semantically appropriate for the task being solved. Across recent work, the relevant frame may be the 2D image plane, an intrinsic object-centric frame, a world or scene frame, a target-centric allocentric frame, a task-relevant $6\mathrm{DoF}$ local frame inferred from motion, or a deictic temporal frame anchored at “now” (Li et al., 2021, Ahrens et al., 2022, Zhang et al., 2024, Ding et al., 30 Aug 2025, Zhang et al., 19 Oct 2025, Wang et al., 5 Feb 2026). The topic has become central in embodied AI because performance degrades when models must resolve relations whose meanings depend on perspective, temporal anchoring, or task structure rather than on static appearance alone (Kamali et al., 21 Jun 2026).

1. Conceptual foundations

A reference frame specifies the coordinate system in which a relation is interpreted. In the cited literature, three spatial families recur. Egocentric or viewer-centric frames attach the coordinate system to the observer or camera; allocentric or world-centric frames use a fixed global scene axis; intrinsic or object-centric frames define relations relative to an object’s own orientation. More recent work adds target-centric allocentric frames, in which a local frame is instantiated around a query-relevant anchor, and task-relevant frames, in which the anchor is inferred from the geometry and dynamics of a demonstration rather than from a predefined object model (Ahrens et al., 2022, Zhang et al., 2024, Ding et al., 30 Aug 2025, Wang et al., 5 Feb 2026).

Task grounding differs from generic localization because the correct frame is often part of the inference problem. In GRiD-A-3D, relative-direction reasoning is explicitly defined in the intrinsic frame of the reference object, not in the camera frame (Ahrens et al., 2022). In SG3D, plans are generated from scene graphs and predominantly use allocentric or object-centric phrasing, while embodied execution may require maintaining the egocentric camera transform $T_{w \leftarrow c}$ as the agent moves (Zhang et al., 2024). In TReF-6, the relevant frame is neither a canonical object frame nor a tool frame; it is a latent “influence point” and its induced $SE(3)$ frame, inferred from a single trajectory and then semantically grounded in the scene (Ding et al., 30 Aug 2025).

The same principle extends beyond spatial reasoning. TUuD defines a temporal frame of reference in which relations are interpreted relative to a deictic origo, the current “now,” and evaluates whether LLM judgments change appropriately when that origo is moved along a timeline (Zhang et al., 19 Oct 2025). This suggests that reference-frame grounding is best understood as a general problem of viewpoint selection and viewpoint-consistent inference, rather than as a purely geometric subproblem.

2. Mathematical representations

Reference-frame task grounding is usually formalized through explicit coordinate transforms. In intrinsic spatial reasoning, GRiD-A-3D defines a reference object basis $(u_R, v_R, w_R)$ and transforms the target displacement $d = x_T - x_R$ into object-centric coordinates by

$d' = M_R^\top d, \qquad M_R = [u_R \; v_R \; w_R].$

Front/back, right/left, and above/below are then decided from the signs of the components of $d'$; in the dataset’s planar setting, horizontal direction labels are derived from the 2D projection of this transformed vector (Ahrens et al., 2022).

In manipulation, TReF-6 represents a Task-Relevant Frame as a full rigid transform

$$\mathbf{T} = \begin{bmatrix} \mathbf{R} & \mathbf{t}\ \mathbf{0}^\top & 1 \end{bmatrix},$$

with $\mathbf{R} \in SO(3)$ and $\mathbf{t} \in \mathbb{R}^3$. World points are mapped into the local frame by

$T_{w \leftarrow c}$0

and DMPs are fit and executed in these local coordinates rather than in the world frame (Ding et al., 30 Aug 2025). The key innovation is that the origin is set at an inferred influence point $T_{w \leftarrow c}$1 that maximizes directional agreement between the trajectory’s acceleration and directions pointing to $T_{w \leftarrow c}$2.

In explicit allocentric frame instantiation, Allocentric Perceiver reconstructs geometry in a world frame $T_{w \leftarrow c}$3, defines a query-conditioned target-centric frame $T_{w \leftarrow c}$4, and transforms points by

$T_{w \leftarrow c}$5

The rotation $T_{w \leftarrow c}$6 is built from a forward axis derived either intrinsically from the reference object’s facing direction or extrinsically from an auxiliary object, together with a default “down” axis and an orthogonal right axis (Wang et al., 5 Feb 2026).

SATURN generalizes this to anchor-conditioned predicate computation. For anchor $T_{w \leftarrow c}$7 and entities $T_{w \leftarrow c}$8, the relative displacement is

$T_{w \leftarrow c}$9

and soft predicate scores take the form

$SE(3)$0

Directional relations, directional combinations, and orientation relations are all expressed through this anchor-conditioned interface, which allows camera-centric, object-centric, and virtual-viewer frames to be composed in one program (Kamali et al., 21 Jun 2026).

Temporal grounding adopts an analogous formal structure. TUuD pairs a target event time $SE(3)$1 with a moving reference point $SE(3)$2 and asks models to score

$SE(3)$3

with aggregation by distance bins. The resulting profile is evaluated for a peak at $SE(3)$4 and decline into past and future as the deictic anchor shifts (Zhang et al., 19 Oct 2025).

3. Methodological families

One major family keeps the frame fixed and learns to ground directly in that representation. Referring Transformer operates entirely in the image’s 2D reference frame. It uses 2D sine/cosine positional encodings, contextualized lingual queries, and one-step decoding to regress normalized boxes $SE(3)$5 and segmentation masks without proposal generation or Hungarian matching (Li et al., 2021). GTI extends the same image-plane grounding problem to video by decomposing Tracking by Language into Grounding, Tracking, and Integration, then using learned $SE(3)$6-scores and $SE(3)$7-scores to decide whether to trust per-frame grounding or template-based tracking (Yang et al., 2019).

A second family treats the frame itself as an inferred latent variable. TReF-6 infers a task-relevant $SE(3)$8 frame from a single trajectory, semantically anchors it with a VLM and Grounded-SAM, and then uses that frame to parameterize DMP execution in novel scenes (Ding et al., 30 Aug 2025). Allocentric Perceiver similarly argues that perspective shift should be offloaded from implicit VLM reasoning to explicit computation: it reconstructs metric 3D state, instantiates a target-centric frame consistent with the query, and discards raw images at the final reasoning stage in favor of geometry-grounded text (Wang et al., 5 Feb 2026).

A third family combines explicit frame construction with symbolic or soft-logical reasoning. SATURN reconstructs an approximate 3D scene, derives soft perspective-aware predicates under chosen anchors, and composes them with a training-free Pythonic executor. The reasoning layer is not hard-thresholded; uncertainty is preserved through multi-hop inference by fuzzy operators such as $SE(3)$9 for conjunction and $(u_R, v_R, w_R)$0 for disjunction (Kamali et al., 21 Jun 2026). This design is distinct from purely neural VLM pipelines, which the paper argues degrade sharply as frame complexity and reasoning depth increase.

A fourth family emphasizes temporal or sequential context. SG3D frames grounding as stepwise selection of a target object sequence aligned to a multi-step plan, and SG-LLM introduces multiple special $(u_R, v_R, w_R)$1 tokens so that a 3D LLM can predict grounded objects across steps while maintaining discourse context (Zhang et al., 2024). TrackTeller addresses dynamic 3D driving scenes by grounding the current-frame referent from multi-frame LiDAR and image inputs, using UniScene fusion, language-aligned decoding, a memory bank, and FutureReg to merge current and propagated proposals (Yu et al., 25 Dec 2025). ToG-Bench makes the same shift in egocentric video, where one instruction may require explicit-implicit dual grounding and one-to-many grounding across time in the moving first-person image plane (Xu et al., 3 Dec 2025).

4. Embodied manipulation, navigation, and assistance

Reference-frame task grounding is especially consequential in robotics because execution must preserve task intent under geometric variation. TReF-6 is a one-shot imitation framework that learns a task-relevant $(u_R, v_R, w_R)$2 reference frame from a single human demonstration and uses it to parameterize a motion primitive that generalizes across object placements, orientations, and local geometry. In real robot experiments on a Kinova Gen3 with one demo per task, it reported overall success of $(u_R, v_R, w_R)$3 versus $(u_R, v_R, w_R)$4 for a baseline DMP in peg-in-hole dropping, $(u_R, v_R, w_R)$5 versus $(u_R, v_R, w_R)$6 in cabinet door opening, and $(u_R, v_R, w_R)$7 versus $(u_R, v_R, w_R)$8 in surface wiping (Ding et al., 30 Aug 2025).

In task-oriented 3D scene grounding, SG3D defines a mapping $(u_R, v_R, w_R)$9 from a reconstructed scene and multistep task plan to a sequence of grounded objects. The benchmark contains 22,346 tasks, 112,236 steps, and 4,895 real scenes. Fine-tuned models still show substantial difficulty with multi-step consistency: around $d = x_T - x_R$0 step and task accuracy for 3D-VisTA overall, around $d = x_T - x_R$1 for PQ3D, and around $d = x_T - x_R$2 for LEO/SG-LLM, with the last performing best overall (Zhang et al., 2024).

Human-assistance systems expose another variant of the same problem. Speech-to-Spatial grounds spoken instructions to an AR world frame $d = x_T - x_R$3 for relations such as left, right, above, below, in front of, and behind, while using camera and user frames for visibility and occlusion tests. In a study with $d = x_T - x_R$4, it reported locate-task median times of $d = x_T - x_R$5 s for Summary, $d = x_T - x_R$6 s for Full, and $d = x_T - x_R$7 s for Audio, with Summary significantly faster than both; in the move task it reported mean times of $d = x_T - x_R$8 s, $d = x_T - x_R$9 s, and $d' = M_R^\top d, \qquad M_R = [u_R \; v_R \; w_R].$0 s, respectively (Kim et al., 3 Feb 2026).

MICA grounds later assistance to a single egocentric demonstration by segmenting it into sub-tasks using gaze or speech, selecting keyframes, generating captions that encode intent and constraints, and retrieving the most relevant segments at question time. On 415 live questions from new users across 32 single-demonstration activities, frame-only retrieval was substantially weaker than multimodal grounding: Frames-as-Context with GPT-4o reached $d' = M_R^\top d, \qquad M_R = [u_R \; v_R \; w_R].$1, Eye Gaze alone $d' = M_R^\top d, \qquad M_R = [u_R \; v_R \; w_R].$2, Speech alone $d' = M_R^\top d, \qquad M_R = [u_R \; v_R \; w_R].$3, Eye Gaze + Speech $d' = M_R^\top d, \qquad M_R = [u_R \; v_R \; w_R].$4, and Eye Gaze + Speech + Summary $d' = M_R^\top d, \qquad M_R = [u_R \; v_R \; w_R].$5 (Sarch et al., 2 May 2025).

5. Dialogue, common ground, and temporal perspective

In dialogue, reference-frame grounding becomes a problem of establishing and reusing common ground under partial observability and perspective mismatch. The reference-centric model for OneCommon uses a structured reference resolver, a dot-indexed recurrent memory, and pragmatic generation that explicitly favors utterances resolvable by a listener. On static corpus evaluation it reported choice accuracy of $d' = M_R^\top d, \qquad M_R = [u_R \; v_R \; w_R].$6, compared with $d' = M_R^\top d, \qquad M_R = [u_R \; v_R \; w_R].$7 for the reimplemented prior baseline, and in human evaluation at overlap size 4 it achieved $d' = M_R^\top d, \qquad M_R = [u_R \; v_R \; w_R].$8 success versus $d' = M_R^\top d, \qquad M_R = [u_R \; v_R \; w_R].$9 for UA’20 (Fried et al., 2021).

The common-ground literature makes explicit that acknowledgment behavior is not sufficient evidence of grounding. IndiRef evaluates whether systems can later exploit shared relational knowledge in situated dialogues, including temporal, spatial, attributive, and inferred references, with perspective-sensitive deictics such as “my” and “your.” The evaluated full-dialog baselines showed that no model exceeded $d'$0 accuracy across all categories, and the paper argues for explicit representations such as typed knowledge graphs with speaker attribution and chronological event logs (Mohapatra et al., 14 Jan 2026).

Temporal frame grounding extends the same concerns from space to time. TUuD derives 19,448 time-event pairs and 6,100 event-event pairs from TimeQA-style timelines and asks models to rate the similarity between “now” and target events as the deictic anchor shifts. All four evaluated LLMs showed a clear peak near $d'$1 and a decline as $d'$2 increased; adaptation was consistent within approximately $d'$3 years but weakened beyond that range, with more variance in far-term judgments (Zhang et al., 19 Oct 2025). This suggests that temporal common ground, like spatial common ground, is highly sensitive to explicit frame specification and dynamic re-anchoring.

6. Benchmarks, metrics, and empirical patterns

The field now spans a broad range of diagnostic and application-driven benchmarks. GRiD-A-3D isolates intrinsic spatial grounding in synthetic VQA with 8,000 rendered images, 432,948 questions, six task types, and a controlled answer distribution. It showed that Existence and Orientation are learned earlier than relation-centric tasks, and that training can converge up to three times faster than on GRiD-3D while maintaining or improving performance on relation tasks (Ahrens et al., 2022).

Sequential and embodied benchmarks expose different failure modes. SG3D reports step accuracy and task accuracy for multistep object grounding in real 3D scenes (Zhang et al., 2024). ToG-Bench, built on egocentric ScanNet trajectories, contains 100 clips, 2,704 task-oriented instructions, and 4,194 object instances across 177 functional categories, and evaluates recognition, temporal grounding, spatial grounding, task aggregation, and strict all-or-nothing task metrics; it reports large explicit-versus-implicit and single-object-versus-multi-object gaps, with GPT-5 achieving T-Acc $d'$4 but T-m_vIoU only $d'$5 (Xu et al., 3 Dec 2025).

Dynamic-scene grounding has introduced tracking-style metrics into the frame-grounding literature. TrackTeller evaluates temporal multimodal 3D grounding on NuPrompt using AMOTA, AMOTP, Recall, TID, and FAF. The abstract reports a 70% relative improvement in Average Multi-Object Tracking Accuracy and a 3.15–3.4 times reduction in False Alarm Frequency over strong baselines; at $d'$6, the model achieves AMOTA $d'$7, Recall $d'$8, AMOTP $d'$9, TID $$\mathbf{T} = \begin{bmatrix} \mathbf{R} & \mathbf{t}\ \mathbf{0}^\top & 1 \end{bmatrix},$$0, and FAF $$\mathbf{T} = \begin{bmatrix} \mathbf{R} & \mathbf{t}\ \mathbf{0}^\top & 1 \end{bmatrix},$$1 (Yu et al., 25 Dec 2025).

Multi-perspective spatial reasoning benchmarks have become more explicit about frame composition. Allocentric Perceiver reports consistent and substantial gains of approximately 10% on allocentric tasks across multiple VLM backbones while maintaining strong egocentric performance (Wang et al., 5 Feb 2026). SATURN introduces 3D FORCE, which controls reasoning depth, view, and frame composition across spatial arrangement grounding and referring expression grounding, and reports $$\mathbf{T} = \begin{bmatrix} \mathbf{R} & \mathbf{t}\ \mathbf{0}^\top & 1 \end{bmatrix},$$2 overall on 3D FORCE and $$\mathbf{T} = \begin{bmatrix} \mathbf{R} & \mathbf{t}\ \mathbf{0}^\top & 1 \end{bmatrix},$$3 overall accuracy on MindCube, outperforming the strongest baseline on the latter by 14 percentage points (Kamali et al., 21 Jun 2026).

A common empirical pattern across these benchmarks is that performance drops when relations must be interpreted under nondefault frames, when frame choice changes within a query, or when early grounding errors propagate across time. This pattern is directly documented in SG3D, ToG-Bench, TrackTeller, Allocentric Perceiver, and SATURN (Zhang et al., 2024, Xu et al., 3 Dec 2025, Yu et al., 25 Dec 2025, Wang et al., 5 Feb 2026, Kamali et al., 21 Jun 2026).

7. Limitations and open problems

Current methods remain limited by perception quality, ambiguity, and frame-selection errors. TReF-6 failures correlate with segmentation errors and depth noise; its assumptions include rigid objects, quasi-static contact during the relevant phase, and RGB-D visibility for surface-normal estimation (Ding et al., 30 Aug 2025). Allocentric Perceiver depends on accurate 3D recovery and uses camera $$\mathbf{T} = \begin{bmatrix} \mathbf{R} & \mathbf{t}\ \mathbf{0}^\top & 1 \end{bmatrix},$$4 as the default down axis, which the paper notes may conflict with gravity-aligned interpretations when pitch or roll are large (Wang et al., 5 Feb 2026). SATURN still inherits errors from reconstruction, orientation estimation, and pose-constraint extraction, even though soft predicates are more robust than hard geometric thresholds (Kamali et al., 21 Jun 2026).

Sequential settings remain difficult because errors compound. SG3D identifies allocentric-versus-egocentric ambiguity, clutter, ambiguous prepositions, and early mis-groundings that lower task accuracy disproportionately relative to step accuracy (Zhang et al., 2024). ToG-Bench reports steady degradation of temporal and spatial localization with longer videos even when recognition remains relatively stable, and explicitly notes the absence of 3D camera or world coordinates in its current annotation format (Xu et al., 3 Dec 2025). TrackTeller mitigates occlusions and viewpoint changes with FutureReg and memory-based retrieval, but crowded scenes and vague temporal language remain challenging (Yu et al., 25 Dec 2025).

Human-centered systems face related but distinct limitations. Speech-to-Spatial currently grounds spatial relations only in an allocentric, axis-aligned world frame and does not yet support egocentric expressions such as “to your right” or complex chained relations such as “second to the right of” and “in between” (Kim et al., 3 Feb 2026). MICA depends on eye-tracking quality, speech quality, camera field of view, and prompt-time context length, and it has not fully tested generalization to entirely new environments (Sarch et al., 2 May 2025). In dialogue, the ontology-based common-ground approach in IndiRef can merge near-duplicate contexts, especially in highly similar multimodal settings such as Spot the Difference (Mohapatra et al., 14 Jan 2026).

The most consistent research direction is to make frame choice explicit rather than implicit. The surveyed work points to several compatible strategies: explicit coordinate transforms and frame instantiation, task-conditioned local frames, persistent common-ground memory with speaker attribution, uncertainty-preserving symbolic composition, and benchmark designs that separate egocentric, intrinsic, allocentric, and temporal deictic reasoning (Ahrens et al., 2022, Ding et al., 30 Aug 2025, Zhang et al., 19 Oct 2025, Mohapatra et al., 14 Jan 2026, Kamali et al., 21 Jun 2026). A plausible implication is that future systems will increasingly treat frame selection, frame maintenance, and frame switching as first-class inference problems rather than as fixed preprocessing conventions.