TraceSpatial: 3D Spatial Reasoning Dataset

• TraceSpatial is a large-scale, richly annotated dataset designed for evaluating fine-grained 3D spatial referring, measuring, and multi-step spatial tracing in vision-language models for robotics.
• It includes 4.5 million diverse scenes and 30 million QA pairs with compositional reasoning chains spanning 1 to 9 steps, covering outdoor imagery, indoor scans, and tabletop manipulation sequences.
• The dataset supports both supervised and reinforcement fine-tuning with rigorous quality assurance and closed-form evaluation metrics to benchmark spatial reasoning and robotic execution.

TraceSpatial is a large-scale, richly annotated dataset developed to advance fine-grained 3D spatial referring, measuring, and multi-step spatial tracing in vision-LLMs (VLMs) within robotics. Targeted at supporting supervised fine-tuning (SFT) and reinforcement fine-tuning (RFT), TraceSpatial contains 30 million question–answer (QA) pairs derived from 4.5 million visual scenes. The dataset spans diverse scene modalities—including outdoor imagery, indoor 3D scans, and tabletop manipulation sequences—and encodes multi-step, metric-grounded reasoning chains, facilitating both compositional visuospatial understanding and long-horizon robotic execution (Zhou et al., 15 Dec 2025).

1. Dataset Composition and Statistics

TraceSpatial comprises 4.5 million scenes—single instances of visual question answering (VQA) samples—stratified into three major categories:

Scene Type	Proportion (%)	Data Sources
Outdoor	35	Web images
Indoor	40	CA-1M, ScanNet 3D scans
Tabletop/Manip	25	DROID, AgiBot videos

The dataset contains approximately 6–7 QA pairs per sample, generating 30 million total QA instances. Questions range from 5 to 50 words (mean ≈ 21.7), with compositional reasoning chain lengths spanning 1 to 9 atomic “steps”—one perceptual inference per step (referring, measuring, or scale prediction). Distribution of reasoning chain length:

Reasoning Steps (K)	QA Proportion (%)
1–2	25
3–4	35
5–6	20
7–9	20

This stratification ensures coverage from simple perception to multi-stage spatial reasoning.

2. Data Structure and File Organization

On disk, TraceSpatial is partitioned into standard splits—train (3.6M samples), validation (450K), and test (450K)—with each split stored as a JSONL file:

data/ ├ train.jsonl ├ val.jsonl ├ test.jsonl └ metadata.json

The metadata file provides a global index of scene identifiers, camera intrinsics, and absolute-metric depth-map paths. Each JSONL sample adheres to the following schema:

{ "scene_id": string, "image_path": string, // RGB image "depth_path": string, // .png or .exr metric depth map "intrinsics": [fx, fy, cx, cy], "objects": [ // object annotations { "obj_id": string, "bbox2d": [x0, y0, x1, y1], "bbox3d": [x, y, z, w, l, h, ϕ], "mask_rle": string, "caption": string }, ... ], "qas": [ { "question": string, "answer": string, "reasoning_chain": [ "[Referring] ...", "[Measuring] ...", "[Scale] ...", ... ], "trace_3d": [ [u1,v1,d1], ..., [uT,vT,dT] ] }, ... ] }

Scene-level detail includes both 2D (bbox2d) and 3D (bbox3d) object localization, semantic captions, pixel-wise masks (RLE), and detailed camera calibration. QA entries may encode explicit reasoning chains and 3D spatial traces for direct supervision.

3. Annotation and Quality Assurance Protocols

QA pairs are generated via a hybrid pipeline:

• Template-based synthesis: Structured scene graphs (extracted from 2D/3D assets) serve as templates for question and answer instantiation.
• LLM-based paraphrasing: The QwQ-32B model is employed for diversification and linguistic paraphrase. No QA pairs are directly authored by humans; instead, human annotation is devoted to filtering and post-hoc cleaning.
• Metric accuracy: For 3D scans (CA-1M, ScanNet), ground-truth camera intrinsics, extrinsics, and depth maps are available. In manipulation videos (DROID, AgiBot), extrinsics undergo explicit depth-consistency validation. Synthetic tabletop samples (RoboTwin) guarantee perfect geometric annotations.
• Quality control: Images undergo multi-stage filtering (SigLIP2, Qwen-VL) to eliminate non-photorealistic content. Samples where keypoints are occluded by more than 30% are excluded. On a 5% held-out subset, inter-annotator agreement achieves Cohen’s κ ≈ 0.82.

4. Reasoning Complexity and Sample Types

Each reasoning chain is composed of sequential steps, each corresponding to an atomic perceptual task:

• Referring: Localize object/query in 2D or 3D space.
• Measuring: Determine metric attributes (e.g., object height).
• Scale Prediction: Reconcile inter-object/scenario scale.
• Tracing: Sequence of spatial points representing a trajectory or manipulation path.

Representative cases:

• Simple Example (1–2 steps):
  • Q: “How tall is the red cup?”
  • Chain: [Referring] → [Measuring]
• Complex Example (7–8 steps):
  • Q: “Water flowers from left to right with can hovering 1–5 cm above each.”
  • Chain: Iterated [Referring], [Measuring], [Scale], and [Tracing] over sequential targets

A “step” is defined as one atomic chain entry, $K$ as the total chain length.

5. Evaluation Protocols and Metrics

TraceSpatial’s evaluation suite consists of both accuracy-based and process-oriented metrics:

• Start-point accuracy:

$$\text{Acc}_{\text{start}} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}\{\|p^{(i)}_{\text{pred},1} - p^{(i)}_{\text{gt},1}\|_2 < \delta_s\}$$

• End-point accuracy:

$$\text{Acc}_{\text{end}} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}\{\|p^{(i)}_{\text{pred},T} - p^{(i)}_{\text{gt},T}\|_2 < \delta_e\}$$

• Overall trace success rate:

$$\text{SR} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}\{\text{Acc}_{\text{start}}=1,\, \text{Acc}_{\text{end}}=1,\, \text{trace}_i \text{ collision-free}\}$$

• Typical thresholds are $\delta_s,\,\delta_e \approx 0.1$ m.

These closed-form metrics allow fine-grained, reproducible benchmarking of spatial reasoning and execution performance. TraceSpatial-Bench is provided as a dedicated benchmark derived from the core dataset for standardized evaluation.

6. Practical Usage and Pipeline Integration

TraceSpatial is designed for immediate integration with standard deep learning and robotics frameworks. Key operational details include:

• Loading and preprocessing: PyTorch/TensorFlow pipelines can parse JSONL files. Example preprocessing routines and normalization are specified, including normalization of 3D trace points $(u, v, d)$ by image and depth map dimensions.
• Supervised fine-tuning (SFT): Chains with full “reasoning_chain” fields are suitable for direct supervision in VLMs.
• Reinforcement fine-tuning (RFT): Filtering for samples with at least 4 reasoning steps enables process-level reward supervision for complex spatial inference.
• Spatial representation interchange: Intrinsic matrices enable conversion between image- and world-coordinate representations.
• Robotics deployment: Example pseudocode demonstrates conversion of predicted 3D traces into world coordinates for robotic execution, such as UR5 pick-and-place pipelines.

7. Applications and Significance

TraceSpatial addresses key challenges in metric-grounded, multi-step spatial reasoning for embodied agents. It enables:

• Training 3D-aware VLMs to unify spatial referring, measurement, and tracing.
• Process-supervised learning via explicit stepwise “reasoning_chain” supervision.
• Benchmarking both perception and robotic execution, using standard closed-form metrics.
• Plug-and-play integration with reinforcement and supervised learning routines.

A plausible implication is that TraceSpatial may serve as a foundation for future research on compositional visuospatial reasoning and embodied AI in open-world and manipulation contexts.

For detailed methodology, dataset schema, and benchmarking protocols, see "RoboTracer: Mastering Spatial Trace with Reasoning in Vision-LLMs for Robotics" (Zhou et al., 15 Dec 2025).