TraceSpatial: 3D Spatial Tracing Benchmark

• TraceSpatial is a large-scale dataset and benchmark suite for 3D spatial tracing in robotics, offering 4.5M samples and 30M QA pairs to support multi-step spatial reasoning.
• It employs explicit 3D metric representations, pseudo-3D reconstruction, and multi-source filtering to improve spatial understanding across indoor, outdoor, and tabletop scenes.
• TraceSpatial bridges language and control by integrating supervised and reinforcement fine-tuning, significantly enhancing collision-aware spatial planning and waypoint prediction.

Searching arXiv for the specified papers and closely related usage of “TraceSpatial.” TraceSpatial is a term used in recent research on spatial reasoning to denote, most explicitly, a large-scale dataset and benchmark suite for 3D spatial tracing in robotics introduced with RoboTracer, where the central problem is to convert a visual scene and a language instruction into an ordered sequence of 3D waypoints (Zhou et al., 15 Dec 2025). In adjacent work on multimodal LLMs, the query is also associated with TRACE—“Textual Representation of Allocentric Context from Egocentric Video”—a prompting method that externalizes a structured allocentric text representation before answering spatial questions from egocentric video (Hua et al., 24 Mar 2026). The two usages are related by a common emphasis on explicit intermediate spatial structure, but they target different tasks, data regimes, and evaluation protocols.

1. Scope and nomenclature

In the robotics literature, TraceSpatial denotes a dataset/benchmark suite for 3D spatial understanding, measuring, referring, and spatial tracing. In the egocentric-video reasoning literature, TRACE is a prompting method rather than a dataset, but it is presented in a query-centered discussion under the heading “TraceSpatial.” The distinction is operationally important because the former supervises waypoint prediction and robot-relevant spatial planning, whereas the latter constrains an MLLM’s intermediate reasoning trace into a textual allocentric world model (Zhou et al., 15 Dec 2025, Hua et al., 24 Mar 2026).

Usage	Core artifact	Source
TraceSpatial	Large-scale dataset and benchmark suite for 3D spatial tracing in robotics	(Zhou et al., 15 Dec 2025)
TRACE in query-centered discussion of “TraceSpatial”	Prompting method for structured allocentric spatial reasoning from egocentric video	(Hua et al., 24 Mar 2026)

A recurring theme across both usages is that difficult spatial tasks are treated as failures of representation rather than failures of generic language reasoning. In the TraceSpatial dataset, this appears as explicit 3D traces, metric annotations, and process supervision. In TRACE, it appears as the construction of a textual allocentric context $G=\langle M,T,E\rangle$, where $M$ is meta-context, $T$ is trajectory, and $E$ is an entity registry.

2. TraceSpatial as a robotics dataset

TraceSpatial is introduced as a large-scale training corpus for 3D spatial understanding, measuring, referring, and spatial tracing. The paper describes it as containing 4.5M high-quality examples / samples and 30M QA pairs, covering outdoor, indoor, and tabletop scenes, supporting reasoning chains of up to 9 steps, including both object-centric and end-effector-centric traces, and spanning 3 single-/dual-arm robot configurations (Zhou et al., 15 Dec 2025). The authors repeatedly emphasize that it is the largest dataset for 3D spatial reasoning in their setting.

The construction pipeline is explicitly multi-source and progressively increases geometric fidelity and embodied realism.

Source family	Scene type	Purpose
2D web images (OpenImages)	Mostly outdoor + indoor natural images	Learn basic spatial concepts, broad-scale perception, metric-agnostic and pseudo-3D spatial QA
3D scanning datasets (CA-1M, ScanNet)	Indoor scenes	Learn accurate 3D spatial referring/measuring and object-centric tracing with metric geometry
Manipulation videos (DROID, AgiBot, RoboTwin simulation)	Tabletop / embodied scenes	Learn realistic end-effector-centric and object-centric spatial traces for robot manipulation

For the web-image branch, the pipeline uses SigLIP2 for coarse filtering and Qwen2.5-VL-7B for fine-grained filtering, reducing OpenImages from 1.7M images to 934k, then to 846k, and finally to 466k after later quality and variance constraints. Object semantics and localization are added with RAM++, GroundingDINO, and SAM / SAM 2.1 variants. Pseudo-3D reconstruction is obtained from MoGe-2, which provides metric depth, camera intrinsics, and scaled point clouds. The resulting pseudo-3D scene graphs support template QA, multiple-choice QA, fact QA, and reasoning QA, further diversified by QwQ-32B.

For the 3D scanning branch, CA-1M and ScanNet contribute ground-truth depth, camera intrinsics/extrinsics, and oriented 3D bounding boxes. For ScanNet, 2D boxes are derived by sampling 5,000 points from each 3D bounding box surface, projecting them into the image, and retaining points satisfying a depth-consistency condition,

$$D_{map} - 5\text{cm} < Z_{proj} < D_{map} + 5\text{cm},$$

together with image-boundary validity. The enriched 3D scene graphs contain 28 spatial relation types. The paper also standardizes human-like measurement annotation by defining Length, Width, and Height with semantic face-intersection conventions rather than raw dataset axes, and by using dynamic unit selection across meters, centimeters, feet, and inches.

For trace generation from static scans, the paper defines object roles—Moving object $O_{src}$, Reference object $O_{ref}$, Via object $O_{via}$, and Obstacle set $O_{obs}$—and synthesizes traces in simulation with RRT* planning in continuous 3D, structured endpoint sampling, broad-phase AABB, narrow-phase OBB + SAT, escape heuristics for initially colliding starts, Catmull-Rom smoothing, and Ramer–Douglas–Peucker compression to $N \le 8$ keypoints. The primitive taxonomy comprises Place Relative, Directional Move, Stacking, Active Bypass + Place, and Active Bypass + Stack.

The embodied-data branch combines real-world sensing and simulation. From DROID, processed with the ZED SDK, the authors extract camera intrinsics, extrinsics, and depth, validating extrinsics with

$M$0

This yields 20.5k unique manipulation trajectories, 46.8k end-effector-centric QA, and 58.4k object-centric QA. From AgiBot, after cleaning invalid extrinsics, poor segmentation, occluded samples, and temporally misaligned task labels, the final output is 977.5k samples. From modified RoboTwin 2.0, the paper reports 443.1k unique spatial traces and 1.329M QA pairs, split into 914.1k end-effector-centric and 415.1k object-centric examples across 16 manipulation tasks.

A notable quantitative property is that 48.2% of TraceSpatial’s QA pairs are metric-grounded, which the paper highlights as much larger than prior datasets and describes as yielding a 14× higher proportion of absolute-scale data than a cited prior benchmark.

3. Task formulation, representations, and supervision targets

TraceSpatial is explicitly framed around the claim that spatial tracing is not just “point prediction.” The task stack combines 3D spatial referring, 3D spatial measuring, and multi-step reasoning, with the target often being a collision-aware ordered 3D trace (Zhou et al., 15 Dec 2025). The dataset supports spatial understanding, spatial measuring, spatial referring, and spatial tracing, including 2D visual trace, 3D spatial trace, and 2D-to-3D lifting.

At the sample level, the paper formalizes an instance with visual observation $M$1, optional geometry $M$2, text instruction or query $M$3 or $M$4, and answer $M$5. For trace prediction, the target is

$M$6

where $M$7 are image-plane coordinates and $M$8 is absolute depth. The paper explicitly uses this decoupled $M$9 representation rather than direct $T$0 because it simplifies training, avoids forcing the VLM to internalize camera geometry, allows projection down to 2D, and supports reuse across referring/tracing tasks.

The task space is broad. Spatial semantics include relative position, relative size, depth, distance, dimensions, object coordinates, and absolute metric estimates. Trace instruction types include statements such as “Place the {source object} to the {endpoint direction} of the {reference object},” “Move the {source object} {distance:.3f}m in the {endpoint direction} direction,” and multi-constraint instructions such as “Move the {source object} around the {via object} on its {via direction} side, then place it to the {endpoint direction} of the {reference object}.” End-effector-centric examples include “Pick the rightmost hamburger and place it on the keyboard in front of the laptop without collisions” and flower-watering tasks that require ordering targets and maintaining a 1–5 cm hover offset.

Reasoning supervision is also explicit. The supplement notes structured intermediate steps of the form

$T$1

with examples such as [Referring] [the second largest cup]: [(245, 147, 1.837)], [Measuring] [the height of the second largest cup]: 20 centimeters, and [Scale] [Scene]: 2.342. This gives the training set a process-annotation layer in addition to final answers.

The paper is not fully explicit about public data partitioning. It specifies the training subsets used at each stage—4.5M RGB+$T$2 samples for metric alignment, then 9M TraceSpatial samples for metric enhancement, a 14M total SFT mixture once auxiliary datasets are included, and a 120k-sample TraceSpatial subset with reasoning-process annotations for RFT—but it does not provide a standard public train/validation/test partition for the full TraceSpatial corpus, nor explicit scene-level or object-level holdout rules.

4. TraceSpatial-Bench

TraceSpatial-Bench is presented as the first benchmark specifically designed for 3D object-centric spatial trace prediction. It contains 100 manually annotated scenes/images, sourced from CA-1M: 51 and ScanNet: 49, with Pick & Place: 82 and Push & Pull: 18 tasks, and focuses on cluttered indoor and tabletop scenes with precise geometry (Zhou et al., 15 Dec 2025). Each sample provides a source RGB image, absolute depth map, camera intrinsics/extrinsics, a target object start mask, a target destination 3D bounding box, a reference feasible 3D object-centric path, and sufficient geometry to reconstruct occupancy.

The benchmark is organized by reasoning-step complexity. The supplement reports prompts spanning 2 to 8 steps, with counts and average word lengths: 2 steps: 7 prompts, 10.14 words; 3 steps: 16, 14.81; 4 steps: 16, 17.19; 5 steps: 28, 22.29; 6 steps: 21, 27.48; 7 steps: 7, 30.86; 8 steps: 5, 34.60. The main paper describes samples as requiring 3–8 reasoning steps; together, these statements indicate a medium- to high-compositional difficulty regime.

Evaluation uses start/end correctness and occupancy-aware rollout rather than trace IoU. In 2D, the predicted start point must lie inside the object’s ground-truth 2D mask, and at least one of the final three predicted points must lie inside the projected 2D destination box. In 3D, the predicted start point must be within 20 cm of the target object’s point cloud, and at least one of the final three predicted 3D points must be within 20 cm of the destination 3D box. Overall 3D success requires both start and end success and, when the object point cloud is moved along the predicted path, no more than 20% of the object’s points intersect the environment occupancy map. The paper also stresses that the reference path is not the only valid answer; multiple collision-free solutions may exist.

The main comparison shows a large gap between generic frontier VLMs and RoboTracer. Gemini-2.5-Pro with RGB input records 2D Start 31, 2D End 33, 3D Start 9, 3D End 16, and Overall 3. Qwen3-VL-8B records 60, 21, 47, 22, and 8, respectively. RoboTracer-2B-SFT improves to Overall 31 with RGB and 39 with RGB+Intrinsics+Depth. The final RoboTracer-2B-RFT model with process rewards reaches Overall 39 with RGB, 40 with RGB+Intrinsics, and 45 with RGB+Intrinsics+Depth. The paper states that RoboTracer exceeds Gemini-2.5-Pro by 36% accuracy on TraceSpatial-Bench, referring to the jump from 3% overall for Gemini to 39% overall for RoboTracer-RFT with RGB, or 45% with RGB, intrinsics, and depth.

A common misconception is that stronger general-purpose VLMs should already solve the benchmark if their 2D localization is adequate. The reported results argue otherwise: the paper attributes many baseline failures to poor metric depth understanding, causing traces that “float” or collide.

5. Training role in RoboTracer and control integration

TraceSpatial functions not only as a QA corpus but as a training substrate for metric-grounded 3D trace prediction. In RoboTracer, supervised fine-tuning includes a scale decoder with regression supervision,

$T$3

where $T$4 is next-token prediction loss, $T$5 is the predicted metric scale factor, and $T$6 is ground-truth scale (Zhou et al., 15 Dec 2025). The stated purpose is to improve absolute-scale awareness, including from RGB inputs.

Reinforcement fine-tuning uses GRPO. Given state $T$7, the policy samples

$T$8

Outcome-based rewards include Outcome Format Reward $T$9, Point Reward $E$0, and a trajectory-level Trace Reward $E$1. The point reward is defined as

$E$2

with all $E$3 normalized to $E$4 and depth normalized by maximum scene depth. Process-based rewards include Process Format Reward $E$5 and Accuracy Reward $E$6. For referring, 2D location error within 10% of image longer side gives reward 0.5, and depth within $E$7 of ground truth gives reward 0.5. For measuring and scale, values within $E$8 receive reward 1.

The supplement gives the final reward as

$E$9

The main text earlier mentions $$D_{map} - 5\text{cm} < Z_{proj} < D_{map} + 5\text{cm},$$0 as well, while the supplement’s final compact equation omits it; the paper presents the reward suite conceptually as consisting of format reward, point reward, trace reward, process format reward, and process accuracy reward. Group-relative normalization is

$$D_{map} - 5\text{cm} < Z_{proj} < D_{map} + 5\text{cm},$$1

The ablation claim reported in the paper is that adding process rewards improves overall TraceSpatial-Bench success by 4% absolute and helps more on 3D/metric aspects than outcome-only rewards.

The paper strongly emphasizes that TraceSpatial is intended as a bridge from language to control. In 19 RoboTwin hard tasks in cluttered scenes, RoboTracer is reported at 75.4% average success on 12 seen tasks, 44.4% on 7 unseen tasks, and 64.0% overall. Representative scores include Place A2B Left: 84, Move Playingcard Away: 94, Click Alarmclock: 79, Place Burger Fries: 99, Unseen Place Empty Cup: 85, and Unseen Place Container Plate: 52.

Real-world integration is demonstrated on UR5 and Unitree G1 humanoid. For UR5 grasping, RoboTracer predicts an end-effector-centric spatial trace; the endpoint seeds SAM2; segmentation filters the object point cloud from an Intel RealSense L515; AnyGrasp predicts a grasp pose; and the pose is transformed into robot coordinates. Final placement converts $$D_{map} - 5\text{cm} < Z_{proj} < D_{map} + 5\text{cm},$$2 plus observed depth and camera intrinsics into 3D, while intermediate motion follows the predicted 3D waypoints directly. The system replans at about 1.5 Hz. A similar setup on G1 uses a head-mounted Intel RealSense D435. On the reported real-world tasks, RoboTracer achieves 60.00 success on “Pick the rightmost hamburger and place it on the keyboard in front of the laptop without collisions” and 30.00 on “Water flowers from right to left with watering can hovering 1–5 cm above each flower,” while OpenVLA and RoboRefer are both reported at 0.00 on both tasks.

6. Relation to TRACE and broader spatial reasoning

A distinct but conceptually related line of work addresses 3D spatial reasoning in MLLMs by changing the intermediate representation rather than training on metric trace targets. TRACE—Textual Representation of Allocentric Context from Egocentric Video—treats spatial question answering over egocentric video as a two-stage problem in which the model first generates a structured allocentric textual representation and then reasons over it (Hua et al., 24 Mar 2026). The representation is

$$D_{map} - 5\text{cm} < Z_{proj} < D_{map} + 5\text{cm},$$3

where $$D_{map} - 5\text{cm} < Z_{proj} < D_{map} + 5\text{cm},$$4 is meta-context, $$D_{map} - 5\text{cm} < Z_{proj} < D_{map} + 5\text{cm},$$5 is trajectory, and $$D_{map} - 5\text{cm} < Z_{proj} < D_{map} + 5\text{cm},$$6 is an entity registry. Inference is formalized as

$$D_{map} - 5\text{cm} < Z_{proj} < D_{map} + 5\text{cm},$$7

with input video $$D_{map} - 5\text{cm} < Z_{proj} < D_{map} + 5\text{cm},$$8, question $$D_{map} - 5\text{cm} < Z_{proj} < D_{map} + 5\text{cm},$$9, and answer $O_{src}$0.

TRACE’s meta-context fixes a room-aligned coordinate frame through room topology, grid alignment, and initial camera heading. The trajectory is a sequence such as

$O_{src}$1

or, in the text schema, steps with time, 2D floor-plane position, an eight-way discrete heading, and an action phrase. The entity registry lists objects individually with id, category, first_seen_at, estimated_pos, approx_size, visual_signature, and spatial_relation, with optional state and orientation. A core design rule is strict serialization: objects are represented as chair_01, chair_02, and so forth rather than grouped counts.

The prompting protocol has one-stage, two-stage, and text-only variants. The standard implementation is one-stage inference, where the model generates TRACE and the final answer jointly. On VSI-Bench, TRACE reaches 60.15 average on Gemini 3. Pro, compared with 52.61 for Direct, 53.65 for CoT, 58.88 for ToT, 59.52 for LtM, and 59.72 for Cognitive Map. On Qwen2.5-VL-72B, TRACE reaches 39.38 versus 36.28 for Direct. On OST-Bench, TRACE reaches 60.42 on Gemini 3. Pro versus 59.22 for Direct, and 65.04 on MiMo-VL-7B versus 62.65 for Direct.

The ablations show that the representation itself and the act of generating it both matter. On VSI-Bench with Gemini 3. Pro, One-Stage TRACE gives 60.15, Two-Stage TRACE gives 58.52, and Text-Only TRACE gives 52.27, nearly matching Video Direct at 52.61. Component removals show TRACE w/o Trajectory: 29.19 and TRACE w/o Entity Registry: 25.87 against TRACE: 31.11 in the text-only comparison table, indicating that the entity registry is the most important single component. The paper also reports that generic reasoning prompts such as CoT, ToT, and LtM are not sufficient for 3D spatial tasks and may even hurt performance for some backbones.

This comparison clarifies the broader role of TraceSpatial as a research theme. In the robotics dataset, explicit structure appears as metric traces, depth, camera geometry, occupancy validation, and reasoning-process annotations. In TRACE, explicit structure appears as allocentric text that stabilizes coordinate systems, object identity, and camera motion. A plausible implication is that recent spatial-reasoning work is converging on the same methodological claim: persistent intermediate spatial state is more effective than unconstrained natural-language reasoning when the target problem requires coordinate consistency, object grounding, and multi-step composition.