---
title: TraceForge-123K Dataset
url: https://www.emergentmind.com/topics/traceforge-123k-dataset
type: topic
---

# TraceForge-123K Dataset

TraceForge-123K is a large-scale dataset comprising 123,000 video clips and 1.8 million synchronized multimodal triplets of observation, 3D trace, and natural-language instruction. Developed as part of TraceGen’s world modeling framework, TraceForge-123K provides the foundation for learning transferable 3D motion priors from cross-embodiment human and robot demonstrations. Its consistent, geometry-centric trace format and rich annotation schema facilitate research in world model pretraining, representation learning, and rapid task adaptation in robotics [2511.21690].

## 1. Construction Pipeline

TraceForge employs a four-stage pipeline to process raw video from heterogeneous sources—including in-lab robot, bimanual robot, egocentric human, and internet videos—into standardized demonstration episodes:

1. **Event Chunking & Instruction Generation:**  
   Each input video \(V_\mathrm{in}\), regardless of embodiment, is segmented into task-relevant clips \(\{V_i\}\). Chunking uses available start–end markers or discards intervals with negligible motion detected by point-tracking magnitude. For each segment, a vision-language model generates up to four instructions in JSON: a concise imperative, a multi-step directive, a human-like request, and, if available, a human-annotated caption.

2. **3D Point Tracking with Pose & Depth Estimation:**  
   Each clip’s reference frame is overlaid with a uniform \(20 \times 20\) grid of keypoints. The TAPIP3D+CoTracker3 tracker infers per-frame camera extrinsics \(\{T_{\mathrm{world}\rightarrow\mathrm{cam}_t}\}\) and depth maps \(D_t\), reconstructing each keypoint’s trajectory \(\mathbf{T}_{\mathrm{ref}}^{t:t+L} = [(x_i^t, y_i^t, z_i^t)]\). For approximately 20% of human-demo clips lacking reliable depth, only 2D traces are produced.

3. **World-to-Camera Alignment:**  
   World-frame points \(\mathbf{X}_{\mathrm{world}}\) are transformed to the reference camera frame:
   $$
   [X^c, Y^c, Z^c]^\top = T_{\mathrm{ref}}\,\mathbf{X}_{\mathrm{world}}
   $$
   providing screen-aligned traces compensating for camera motion.

4. **Speed Retargeting:**  
   Each raw trace is parameterized by cumulative arc length \(s \in [0,1]\) and resampled at \(L\) uniform values:
   $$
   s_i^t = \frac{\sum_{u=1}^{t} \|\mathbf{p}_i^u - \mathbf{p}_i^{u-1}\|}{\sum_{u=1}^{L} \|\mathbf{p}_i^u - \mathbf{p}_i^{u-1}\|}
   $$
   This ensures fixed-length, embodiment-agnostic traces across samples while preserving velocity profiles.

## 2. 3D Trace and Multimodal Triplet Representation

Each datum in TraceForge-123K is a (visual observation, 3D trace, language instruction) triplet:

- **Trace:**  
  $\mathbf{T}_{\mathrm{ref}}^{1:L} = \{\mathbf{p}_i^t = (x_i^t, y_i^t, z_i^t)\}_{i=1..K,\,t=1..L}$  
  where \(K = 400\) (for a \(20\times20\) keypoint grid) and \(L\) is typically 32 uniformly sampled timesteps, representing 1–2 seconds of future motion. The \((x, y)\) coordinates are pixel-aligned in the reference view; \(z\) is depth in camera units.

- **Observation and Language:**  
  Each trace is synchronized with corresponding visual frames (raw RGB), and up to four textual instructions. Language annotations include concise commands, multi-step directives, and natural human requests.

- **Modeling Protocol:**  
  TraceGen models the temporal differences:
  $$
  \Delta\mathbf{T}_{\mathrm{ref}}^t = \mathbf{T}_{\mathrm{ref}}^{t+1} - \mathbf{T}_{\mathrm{ref}}^t
  $$
  and employs a flow-based decoder to integrate these via an ODE. Pointwise mean error after decoding averages approximately 2 cm.

## 3. Annotation Schema and Data Format

Each clip is represented as a single JSON object with fields:

| Field              | Description                                                                                  | Example/Format                               |
|--------------------|---------------------------------------------------------------------------------------------|----------------------------------------------|
| video_id, chunk_id | Unique identifiers for source video and segment                                             | "XYZ123", 5                                  |
| frames             | List of frame indices included in the chunk                                                 | [10, 11, ..., 42]                            |
| cam_intrinsics     | Camera calibration for reference view                                                       | {...}                                       |
| cam_extrinsics     | Per-frame world-to-camera transforms                                                        | [...]                                        |
| depth_maps         | Per-frame depth arrays; missing if 2D-only                                                  | [...] or null                                |
| trace              | [K, L, 3] array: 400 keypoints, 32 timesteps, (x, y, z) per point                          | [[[x₁¹, y₁¹, z₁¹], ..., [x₁ᴸ, y₁ᴸ, z₁ᴸ]], …] |
| instructions       | Object with up to four textual instructions (including human-authored if present)            | See below                                    |

Minimal annotation example:
```json
{
  "video_id": "XYZ123",
  "chunk_id": 5,
  "frames": [10, 11, ..., 42],
  "cam_intrinsics": {...},
  "cam_extrinsics": [...],
  "trace": [[[x₁¹, y₁¹, z₁¹], …, [x₁ᴸ, y₁ᴸ, z₁ᴸ]], ...],
  "instructions": {
    "instruction_1": "Pick up the block.",
    "instruction_2": "Move gripper above block, lower and grasp.",
    "instruction_3": "Could you pick up that block for me?"
  }
}
```
This ensures standardized, machine-readable inputs for downstream model training or evaluation.

## 4. Dataset Statistics

TraceForge-123K’s key attributes are summarized below:

| Statistic                    | Value / Mode |
|------------------------------|--------------|
| Total clips                  | 123,000      |
| Total triplets               | 1,800,000    |
| Human-demo clips             | ~60,000 (20% 2D-only)  |
| Robot-demo clips             | ~63,000      |
| Distinct manipulation tasks  | ~50          |
| Avg. clip length             | 15–20 s (~450–600 frames) |
| Trace horizon \(L\)          | 32 timesteps (1–2 s) |
| Avg. spatial displacement    | ~0.7 m       |
| Language: unique instruction tokens | ~5,000        |
| Avg. instruction length      | 8–12 words   |

*This suggests extensive coverage of manipulation actions and broad linguistic diversity, enhancing generalization for cross-task and cross-domain learning.*

## 5. Data Splits, Benchmarks, and Model Performance

The dataset is partitioned as follows:

- **Training:** 100,000 clips (1.5 million triplets)
- **Validation:** 11,500 clips
- **Test:** 11,500 clips

TraceForge-123K supports benchmarking on four real-robot manipulation tasks (Clothes, Ball, Brush, Block) using the Franka R3 arm. Benchmark results for the TraceGen world model are:

- **Robot→Robot (“warmup,” 5 demonstrations):** 80% success rate (improving to 82.5% with 15 demos; 0% in zero-shot).  
- **Human→Robot (5 uncalibrated phone videos):** 67.5% success on a real robot.
- **Baselines:** NovaFlow and AVDC world models achieve 0–30% success and are >50× slower; prior 3D flow methods (3DFlowAction) require masks and give 0% zero-shot success.
- **Inference Speed:** TraceGen produces 32-step traces in ~1 second (i.e., 50–600× faster than video-based methods).

*This suggests that TraceForge-123K enables high data efficiency and rapid adaptation, including in challenging cross-embodiment scenarios.*

## 6. Usage, Access, and Licensing

**Recommended uses** include:

- Pretraining 3D motion priors for rapid adaptation between embodiments (e.g., human to robot).
- World-model finetuning in trace space for new robots, tools, or environments.
- Self-supervised representation learning combining geometric, visual, and linguistic cues.

**Access:**  
Download is available at [https://tracegen.github.io](https://tracegen.github.io) under a MIT-style research license. Users are required to cite “S. Lee et al., TraceGen: World Modeling in 3D Trace-Space… CVPR (2025)” and include dataset version and DOI in publications.

*This ensures reproducibility and encourages consistent benchmarking.*

## 7. Significance and Research Context

TraceForge-123K establishes a large-scale, multimodal corpus for representation learning, cross-embodiment skill transfer, and data-efficient policy adaptation in robotic manipulation. Its centimeter-accurate, speed-retargeted 3D traces and comprehensive language instructions abstract from domain- and embodiment-specific cues, providing a unifying geometric substrate for model-driven prediction, control, and imitation learning. Its adoption in TraceGen demonstrates substantial improvements in generalization and inference speed over prior video-based and 3D flow models, without requiring object segmentation or dense pixel-space generation [2511.21690].

Source: https://www.emergentmind.com/topics/traceforge-123k-dataset