---
title: 'TraceSpatial: 3D Spatial Tracing Benchmark'
url: https://www.emergentmind.com/topics/tracespatial
type: topic
---

# TraceSpatial: 3D Spatial Tracing Benchmark

Searching arXiv for the specified papers and closely related usage of “TraceSpatial.”
TraceSpatial is a term used in recent research on spatial reasoning to denote, most explicitly, a large-scale dataset and benchmark suite for 3D spatial tracing in robotics introduced with RoboTracer, where the central problem is to convert a visual scene and a language instruction into an ordered sequence of 3D waypoints [2512.13660]. In adjacent work on multimodal large language models, the query is also associated with TRACE—“Textual Representation of Allocentric Context from Egocentric Video”—a prompting method that externalizes a structured allocentric text representation before answering spatial questions from egocentric video [2603.23404]. The two usages are related by a common emphasis on explicit intermediate spatial structure, but they target different tasks, data regimes, and evaluation protocols.

## 1. Scope and nomenclature

In the robotics literature, TraceSpatial denotes a dataset/benchmark suite for 3D spatial understanding, measuring, referring, and spatial tracing. In the egocentric-video reasoning literature, TRACE is a prompting method rather than a dataset, but it is presented in a query-centered discussion under the heading “TraceSpatial.” The distinction is operationally important because the former supervises waypoint prediction and robot-relevant spatial planning, whereas the latter constrains an MLLM’s intermediate reasoning trace into a textual allocentric world model [2512.13660; 2603.23404].

| Usage | Core artifact | Source |
|---|---|---|
| **TraceSpatial** | Large-scale dataset and benchmark suite for 3D spatial tracing in robotics | [2512.13660] |
| **TRACE** in query-centered discussion of “TraceSpatial” | Prompting method for structured allocentric spatial reasoning from egocentric video | [2603.23404] |

A recurring theme across both usages is that difficult spatial tasks are treated as failures of representation rather than failures of generic language reasoning. In the TraceSpatial dataset, this appears as explicit 3D traces, metric annotations, and process supervision. In TRACE, it appears as the construction of a textual allocentric context \(G=\langle M,T,E\rangle\), where \(M\) is meta-context, \(T\) is trajectory, and \(E\) is an entity registry.

## 2. TraceSpatial as a robotics dataset

TraceSpatial is introduced as a large-scale training corpus for 3D spatial understanding, measuring, referring, and spatial tracing. The paper describes it as containing **4.5M high-quality examples / samples** and **30M QA pairs**, covering **outdoor**, **indoor**, and **tabletop** scenes, supporting reasoning chains of **up to 9 steps**, including both **object-centric** and **end-effector-centric** traces, and spanning **3 single-/dual-arm robot configurations** [2512.13660]. The authors repeatedly emphasize that it is the **largest dataset for 3D spatial reasoning** in their setting.

The construction pipeline is explicitly multi-source and progressively increases geometric fidelity and embodied realism.

| Source family | Scene type | Purpose |
|---|---|---|
| **2D web images** (OpenImages) | Mostly outdoor + indoor natural images | Learn basic spatial concepts, broad-scale perception, metric-agnostic and pseudo-3D spatial QA |
| **3D scanning datasets** (CA-1M, ScanNet) | Indoor scenes | Learn accurate 3D spatial referring/measuring and object-centric tracing with metric geometry |
| **Manipulation videos** (DROID, AgiBot, RoboTwin simulation) | Tabletop / embodied scenes | Learn realistic end-effector-centric and object-centric spatial traces for robot manipulation |

For the web-image branch, the pipeline uses **SigLIP2** for coarse filtering and **Qwen2.5-VL-7B** for fine-grained filtering, reducing OpenImages from **1.7M** images to **934k**, then to **846k**, and finally to **466k** after later quality and variance constraints. Object semantics and localization are added with **RAM++**, **GroundingDINO**, and **SAM / SAM 2.1 variants**. Pseudo-3D reconstruction is obtained from **MoGe-2**, which provides **metric depth**, **camera intrinsics**, and **scaled point clouds**. The resulting pseudo-3D scene graphs support template QA, multiple-choice QA, fact QA, and reasoning QA, further diversified by **QwQ-32B**.

For the 3D scanning branch, **CA-1M** and **ScanNet** contribute **ground-truth depth**, **camera intrinsics/extrinsics**, and **oriented 3D bounding boxes**. For ScanNet, 2D boxes are derived by sampling **5,000 points** from each 3D bounding box surface, projecting them into the image, and retaining points satisfying a depth-consistency condition,
\[
D_{map} - 5\text{cm} < Z_{proj} < D_{map} + 5\text{cm},
\]
together with image-boundary validity. The enriched 3D scene graphs contain **28 spatial relation types**. The paper also standardizes human-like measurement annotation by defining **Length**, **Width**, and **Height** with semantic face-intersection conventions rather than raw dataset axes, and by using dynamic unit selection across meters, centimeters, feet, and inches.

For trace generation from static scans, the paper defines object roles—**Moving object** \(O_{src}\), **Reference object** \(O_{ref}\), **Via object** \(O_{via}\), and **Obstacle set** \(O_{obs}\)—and synthesizes traces in simulation with **RRT\*** planning in continuous 3D, structured endpoint sampling, broad-phase **AABB**, narrow-phase **OBB + SAT**, escape heuristics for initially colliding starts, **Catmull-Rom** smoothing, and **Ramer–Douglas–Peucker** compression to \(N \le 8\) keypoints. The primitive taxonomy comprises **Place Relative**, **Directional Move**, **Stacking**, **Active Bypass + Place**, and **Active Bypass + Stack**.

The embodied-data branch combines real-world sensing and simulation. From **DROID**, processed with the **ZED SDK**, the authors extract camera intrinsics, extrinsics, and depth, validating extrinsics with
\[
|D(\mathbf{u}) - z^{cam}_{eef}| < 5\text{ cm}.
\]
This yields **20.5k unique manipulation trajectories**, **46.8k end-effector-centric QA**, and **58.4k object-centric QA**. From **AgiBot**, after cleaning invalid extrinsics, poor segmentation, occluded samples, and temporally misaligned task labels, the final output is **977.5k samples**. From modified **RoboTwin 2.0**, the paper reports **443.1k unique spatial traces** and **1.329M QA pairs**, split into **914.1k** end-effector-centric and **415.1k** object-centric examples across **16 manipulation tasks**.

A notable quantitative property is that **48.2%** of TraceSpatial’s QA pairs are **metric-grounded**, which the paper highlights as much larger than prior datasets and describes as yielding a **14× higher proportion of absolute-scale data** than a cited prior benchmark.

## 3. Task formulation, representations, and supervision targets

TraceSpatial is explicitly framed around the claim that spatial tracing is not just “point prediction.” The task stack combines **3D spatial referring**, **3D spatial measuring**, and **multi-step reasoning**, with the target often being a collision-aware ordered 3D trace [2512.13660]. The dataset supports **spatial understanding**, **spatial measuring**, **spatial referring**, and **spatial tracing**, including **2D visual trace**, **3D spatial trace**, and **2D-to-3D lifting**.

At the sample level, the paper formalizes an instance with visual observation \(\mathcal{O}\), optional geometry \(\mathcal{G}\), text instruction or query \(\mathcal{T}\) or \(\mathcal{Q}\), and answer \(\mathcal{A}\). For trace prediction, the target is
\[
\tau = \{p_t\}_{t=1}^{T}, \qquad p_t=(u_t,v_t,d_t),
\]
where \((u_t,v_t)\) are image-plane coordinates and \(d_t\) is absolute depth. The paper explicitly uses this decoupled \((u,v,d)\) representation rather than direct \((x,y,z)\) because it **simplifies training**, **avoids forcing the VLM to internalize camera geometry**, **allows projection down to 2D**, and **supports reuse across referring/tracing tasks**.

The task space is broad. Spatial semantics include relative position, relative size, depth, distance, dimensions, object coordinates, and absolute metric estimates. Trace instruction types include statements such as “Place the {source object} to the {endpoint direction} of the {reference object},” “Move the {source object} {distance:.3f}m in the {endpoint direction} direction,” and multi-constraint instructions such as “Move the {source object} around the {via object} on its {via direction} side, then place it to the {endpoint direction} of the {reference object}.” End-effector-centric examples include “Pick the rightmost hamburger and place it on the keyboard in front of the laptop without collisions” and flower-watering tasks that require ordering targets and maintaining a **1–5 cm** hover offset.

Reasoning supervision is also explicit. The supplement notes structured intermediate steps of the form
\[
[\text{Perception Type}] [\text{Target Object}] : [\text{Value}],
\]
with examples such as `[Referring] [the second largest cup]: [(245, 147, 1.837)]`, `[Measuring] [the height of the second largest cup]: 20 centimeters`, and `[Scale] [Scene]: 2.342`. This gives the training set a process-annotation layer in addition to final answers.

The paper is not fully explicit about public data partitioning. It specifies the training subsets used at each stage—**4.5M** RGB+\(\mathcal{X}\) samples for metric alignment, then **9M** TraceSpatial samples for metric enhancement, a **14M** total SFT mixture once auxiliary datasets are included, and a **120k-sample** TraceSpatial subset with reasoning-process annotations for RFT—but it **does not provide a standard public train/validation/test partition for the full TraceSpatial corpus**, nor explicit scene-level or object-level holdout rules.

## 4. TraceSpatial-Bench

TraceSpatial-Bench is presented as the first benchmark specifically designed for **3D object-centric spatial trace prediction**. It contains **100 manually annotated scenes/images**, sourced from **CA-1M: 51** and **ScanNet: 49**, with **Pick & Place: 82** and **Push & Pull: 18** tasks, and focuses on cluttered indoor and tabletop scenes with precise geometry [2512.13660]. Each sample provides a source RGB image, absolute depth map, camera intrinsics/extrinsics, a target object start mask, a target destination 3D bounding box, a reference feasible 3D object-centric path, and sufficient geometry to reconstruct occupancy.

The benchmark is organized by reasoning-step complexity. The supplement reports prompts spanning **2 to 8 steps**, with counts and average word lengths: **2 steps: 7 prompts, 10.14 words**; **3 steps: 16, 14.81**; **4 steps: 16, 17.19**; **5 steps: 28, 22.29**; **6 steps: 21, 27.48**; **7 steps: 7, 30.86**; **8 steps: 5, 34.60**. The main paper describes samples as requiring **3–8 reasoning steps**; together, these statements indicate a medium- to high-compositional difficulty regime.

Evaluation uses start/end correctness and occupancy-aware rollout rather than trace IoU. In 2D, the predicted start point must lie inside the object’s ground-truth 2D mask, and at least one of the **final three predicted points** must lie inside the projected 2D destination box. In 3D, the predicted start point must be within **20 cm** of the target object’s point cloud, and at least one of the **final three predicted 3D points** must be within **20 cm** of the destination 3D box. Overall 3D success requires both start and end success and, when the object point cloud is moved along the predicted path, **no more than 20% of the object’s points intersect the environment occupancy map**. The paper also stresses that the reference path is **not the only valid answer**; multiple collision-free solutions may exist.

The main comparison shows a large gap between generic frontier VLMs and RoboTracer. **Gemini-2.5-Pro** with RGB input records **2D Start 31**, **2D End 33**, **3D Start 9**, **3D End 16**, and **Overall 3**. **Qwen3-VL-8B** records **60**, **21**, **47**, **22**, and **8**, respectively. **RoboTracer-2B-SFT** improves to **Overall 31** with RGB and **39** with **RGB+Intrinsics+Depth**. The final **RoboTracer-2B-RFT** model with process rewards reaches **Overall 39** with RGB, **40** with **RGB+Intrinsics**, and **45** with **RGB+Intrinsics+Depth**. The paper states that RoboTracer exceeds **Gemini-2.5-Pro by 36% accuracy** on TraceSpatial-Bench, referring to the jump from **3% overall** for Gemini to **39% overall** for RoboTracer-RFT with RGB, or **45%** with RGB, intrinsics, and depth.

A common misconception is that stronger general-purpose VLMs should already solve the benchmark if their 2D localization is adequate. The reported results argue otherwise: the paper attributes many baseline failures to poor metric depth understanding, causing traces that “float” or collide.

## 5. Training role in RoboTracer and control integration

TraceSpatial functions not only as a QA corpus but as a training substrate for metric-grounded 3D trace prediction. In RoboTracer, supervised fine-tuning includes a scale decoder with regression supervision,
\[
\mathcal{L} = \mathcal{L}_{ntp} + 0.1 \left\|\log(\hat{s})-\operatorname{stopgrad}(\log(s^*))\right\|_2^2,
\]
where \(\mathcal{L}_{ntp}\) is next-token prediction loss, \(\hat{s}\) is the predicted metric scale factor, and \(s^*\) is ground-truth scale [2512.13660]. The stated purpose is to improve absolute-scale awareness, including from RGB inputs.

Reinforcement fine-tuning uses **GRPO**. Given state \(s=(\mathcal{O},\mathcal{Q})\), the policy samples
\[
a_i \sim \pi_\theta(a \mid \mathcal{O}, \mathcal{Q}), \quad i=1,\dots,N.
\]
Outcome-based rewards include **Outcome Format Reward** \(R_{OF}\), **Point Reward** \(R_P\), and a trajectory-level **Trace Reward** \(R_T\). The point reward is defined as
\[
R_P = \frac{1}{2}\bigl[f(p_1,\hat{p}_1) + f(p_T,\hat{p}_T)\bigr], \qquad
f(p,p') = \max\bigl(0,\; 1-\|p-p'\|_2^2\bigr),
\]
with all \((u,v,d)\) normalized to \([0,1]\) and depth normalized by maximum scene depth. Process-based rewards include **Process Format Reward** \(R_{PF}\) and **Accuracy Reward** \(R_{Acc}\). For referring, 2D location error within **10% of image longer side** gives reward **0.5**, and depth within **\(\pm 30\%\)** of ground truth gives reward **0.5**. For measuring and scale, values within **\(\pm 30\%\)** receive reward **1**.

The supplement gives the final reward as
\[
r_i = R_{OF}(a_i) + R_P(a_i) + \alpha R_{PF}(a_i) + \alpha R_{Acc}(a_i),
\qquad \alpha = 0.25.
\]
The main text earlier mentions \(R_T\) as well, while the supplement’s final compact equation omits it; the paper presents the reward suite conceptually as consisting of format reward, point reward, trace reward, process format reward, and process accuracy reward. Group-relative normalization is
\[
A_i = \frac{r_i - \mathrm{mean}(\{r_j\})}{\mathrm{std}(\{r_j\})}.
\]
The ablation claim reported in the paper is that adding process rewards improves **overall TraceSpatial-Bench success by 4% absolute** and helps more on **3D/metric** aspects than outcome-only rewards.

The paper strongly emphasizes that TraceSpatial is intended as a bridge from language to control. In **19 RoboTwin hard tasks** in cluttered scenes, RoboTracer is reported at **75.4%** average success on **12 seen** tasks, **44.4%** on **7 unseen** tasks, and **64.0%** overall. Representative scores include **Place A2B Left: 84**, **Move Playingcard Away: 94**, **Click Alarmclock: 79**, **Place Burger Fries: 99**, **Unseen Place Empty Cup: 85**, and **Unseen Place Container Plate: 52**.

Real-world integration is demonstrated on **UR5** and **Unitree G1 humanoid**. For UR5 grasping, RoboTracer predicts an end-effector-centric spatial trace; the endpoint seeds **SAM2**; segmentation filters the object point cloud from an **Intel RealSense L515**; **AnyGrasp** predicts a grasp pose; and the pose is transformed into robot coordinates. Final placement converts \((u,v)\) plus observed depth and camera intrinsics into 3D, while intermediate motion follows the predicted 3D waypoints directly. The system replans at about **1.5 Hz**. A similar setup on G1 uses a head-mounted **Intel RealSense D435**. On the reported real-world tasks, RoboTracer achieves **60.00** success on “Pick the rightmost hamburger and place it on the keyboard in front of the laptop without collisions” and **30.00** on “Water flowers from right to left with watering can hovering 1–5 cm above each flower,” while **OpenVLA** and **RoboRefer** are both reported at **0.00** on both tasks.

## 6. Relation to TRACE and broader spatial reasoning

A distinct but conceptually related line of work addresses 3D spatial reasoning in MLLMs by changing the intermediate representation rather than training on metric trace targets. TRACE—**Textual Representation of Allocentric Context from Egocentric Video**—treats spatial question answering over egocentric video as a two-stage problem in which the model first generates a structured allocentric textual representation and then reasons over it [2603.23404]. The representation is
\[
G = \langle M, T, E \rangle,
\]
where \(M\) is **meta-context**, \(T\) is **trajectory**, and \(E\) is an **entity registry**. Inference is formalized as
\[
\hat{A} = \arg\max_{A,G} P(A \mid G, V, Q)\cdot P(G \mid V, Q),
\]
with input video \(V\), question \(Q\), and answer \(A\).

TRACE’s meta-context fixes a room-aligned coordinate frame through room topology, grid alignment, and initial camera heading. The trajectory is a sequence such as
\[
T=\{(t_k,p_k,\theta_k)\}_{k=0}^{K},
\]
or, in the text schema, steps with time, 2D floor-plane position, an eight-way discrete heading, and an action phrase. The entity registry lists objects individually with `id`, `category`, `first_seen_at`, `estimated_pos`, `approx_size`, `visual_signature`, and `spatial_relation`, with optional `state` and `orientation`. A core design rule is strict serialization: objects are represented as `chair_01`, `chair_02`, and so forth rather than grouped counts.

The prompting protocol has one-stage, two-stage, and text-only variants. The standard implementation is **one-stage inference**, where the model generates TRACE and the final answer jointly. On **VSI-Bench**, TRACE reaches **60.15** average on **Gemini 3. Pro**, compared with **52.61** for Direct, **53.65** for CoT, **58.88** for ToT, **59.52** for LtM, and **59.72** for Cognitive Map. On **Qwen2.5-VL-72B**, TRACE reaches **39.38** versus **36.28** for Direct. On **OST-Bench**, TRACE reaches **60.42** on Gemini 3. Pro versus **59.22** for Direct, and **65.04** on **MiMo-VL-7B** versus **62.65** for Direct.

The ablations show that the representation itself and the act of generating it both matter. On VSI-Bench with Gemini 3. Pro, **One-Stage TRACE** gives **60.15**, **Two-Stage TRACE** gives **58.52**, and **Text-Only TRACE** gives **52.27**, nearly matching **Video Direct** at **52.61**. Component removals show **TRACE w/o Trajectory: 29.19** and **TRACE w/o Entity Registry: 25.87** against **TRACE: 31.11** in the text-only comparison table, indicating that the entity registry is the most important single component. The paper also reports that generic reasoning prompts such as CoT, ToT, and LtM are not sufficient for 3D spatial tasks and may even hurt performance for some backbones.

This comparison clarifies the broader role of TraceSpatial as a research theme. In the robotics dataset, explicit structure appears as metric traces, depth, camera geometry, occupancy validation, and reasoning-process annotations. In TRACE, explicit structure appears as allocentric text that stabilizes coordinate systems, object identity, and camera motion. A plausible implication is that recent spatial-reasoning work is converging on the same methodological claim: persistent intermediate spatial state is more effective than unconstrained natural-language reasoning when the target problem requires coordinate consistency, object grounding, and multi-step composition.

Source: https://www.emergentmind.com/topics/tracespatial