Cross-Embodiment 3D Trace Dataset

• The dataset is a comprehensive collection of 3D movement traces aggregating human and robot demonstrations across varied scenes and tasks.
• It employs advanced data processing pipelines—including keypoint selection, 3D reconstruction, and temporal normalization—to ensure cross-embodiment consistency.
• Structured with geometric, semantic, and task-specific annotations, it underpins scalable policy learning and robust evaluation in diverse physical domains.

The Cross-Embodiment 3D Trace Dataset refers to a class of large-scale, heterogeneous corpora of 3D movement or interaction trajectory data curated to facilitate the development of world models and policy learners capable of generalizing across varied embodiments, scenes, and task distributions. Such datasets aggregate human and robot demonstrations, encode physical interactions in temporally-aligned geometric sequences, and, when available, include multi-modal annotations such as language instructions or semantic event labels. They underpin recent advances in scalable, embodiment-agnostic robot learning, multi-embodiment reinforcement learning, and foundation models for physical reasoning across domains.

1. Dataset Scope and Representative Corpora

Multiple projects have established canonical Cross-Embodiment 3D Trace Datasets, each tuned to distinct application domains and research paradigms. Major examples include:

• TraceForge-123K ("TraceGen" (Lee et al., 26 Nov 2025)): 123,000 episodes, 1.8 million observation–trace–language triplets compiled from eight open datasets, encompassing human and robot manipulation across tabletop, kitchen, and office scenes.
• ManiFlow-110k ("3DFlowAction" (Zhi et al., 6 Jun 2025)): 110,000 manipulation clips representing 3D optical object flows synthesized from both human and robot videos using an automated, object-centric pipeline.
• µ₀ TraceExtract Suite ("μ₀" (Lee et al., 11 Jun 2026)): ≳1,000 hours of simulation, robot, and human egocentric sequences, automatically segmented, re-parameterized, and captioned, yielding ≳2 million keypoint-trace segments for general manipulation.
• Open-H-Embodiment (Consortium et al., 22 Apr 2026): 124,019 clinical and simulated medical robot trajectories (124k episodes; 770 hours) across 20 robotic embodiments with full 3D kinematic streams.
• Cross-Embodiment Locomotion Dataset ("Cross-Embodiment Offline RL" (Abe et al., 20 Feb 2026)): 96 million state–action–reward transitions from 16 legged robot morphologies covering bipeds, quadrupeds, and hexapods for pre-training RL policy learners.

These datasets provide a statistically and morphologically broad substrate to train and evaluate cross-embodiment models, spanning platform-specific to fully actor-agnostic representations.

2. Data Acquisition and Processing Pipelines

Central to cross-embodiment trace datasets is automated or semi-automated extraction of 3D interaction trajectories from raw sensory streams (RGB and/or depth video, robot proprioception, and simulation logs) irrespective of embodiment idiosyncrasies. Key pipeline stages include:

• Keypoint Selection and Entity Clustering: Semantic entity segmentation (e.g., hands, objects, tools, contact regions) via deep visual descriptors (e.g., DINOv2), followed by spatially diverse keypoint sampling within each mask (Lee et al., 11 Jun 2026).
• Global–Local 3D Reconstruction: Camera intrinsics and extrinsics estimated using sparse/dense visual geometry toolkits (VGGT), with reference frame alignment across video chunks (Lee et al., 11 Jun 2026, Lee et al., 26 Nov 2025).
• 3D Flow Lifting and Tracking: Dense or sparse correspondences are tracked temporally, with 2D flows lifted to 3D using depth or stereo and camera calibration. Object-centric flows are synthesized using tracking (CoTracker3), object localization (bounding boxes), and depth prediction (DepthAnythingV2) (Zhi et al., 6 Jun 2025).
• Cross-Chunk Temporal Consistency: Keypoint tracks are initialized and stitched across video segments using cross-chunk propagation (Lee et al., 11 Jun 2026).
• Arc-Length Reparameterization and Speed Retargeting: Trajectories are resampled via arc-length normalization to ensure temporal consistency despite differing robot/human speeds or timing artifacts (Lee et al., 26 Nov 2025, Lee et al., 11 Jun 2026).
• Annotation and Event Segmentation: Temporal segmentation via motion cues (e.g., acceleration filtering), with hierarchical captioning via VLM and LLM modules for natural language event labels (Lee et al., 11 Jun 2026, Lee et al., 26 Nov 2025).

The resulting representation abstracts from low-level appearance and embodiment-specific actuation, focusing instead on physically meaningful movement descriptions and their temporal logic.

3. Data Structures and Annotation Schemes

Data within cross-embodiment 3D trace datasets is structured to maximize machine readability and compositional access. The following formats are representative:

Dataset/Project	Trajectory Format	Annotation Types
TraceForge-123K	screen-aligned 3D traces (K×L×3); per-episode files	3× instruction/caption per chunk
ManiFlow-110k	3D optical flow tensors (T×H×W×4 per clip)	Prompt string, bounding box
µ₀ TraceExtract	Keypoint B-splines (D=10), world-referenced chunks	Event + task captions
Open-H-Embodiment	Cartesian EEF pose, joint kinematics (.parquet)	Task family, environment meta
Offline RL Locomotion	(sₜ,aₜ,sₜ₊₁,rₜ,dₜ) at 20 Hz (.npz), metadata.json	Morphology graph, reward weights

• 3D Keypoints and Traces: Sequences of spatial coordinates represented as $\mathbb{R}^3$ points, parameterized by time or B-spline control points.
• Optical Flow: Dense 3D displacement fields (e.g., $$\mathrm{F}_i \in \mathbb{R}^{T \times H \times W \times 4}$$, where each pixel encodes spatial and depth increment plus visibility).
• Kinematic Chains: Full joint-space or end-effector pose streams, including orientations in SO(3) or 6D (Consortium et al., 22 Apr 2026).
• Annotations: Event/natural-language alignments at clip, segment, or step level; object/tool/interaction-type tags; and, where applicable, episode/task/scene taxonomy.

All timing information is normalized (e.g., fixed frame rates, chunk/clip lengths), and multi-view or multi-modal data is time-aligned.

4. Embodiment Diversity and Consistency Mechanisms

A defining property of cross-embodiment trace datasets is their explicit treatment of inter- and intra-embodiment heterogeneity:

• Source Diversity: Coverage includes humanoid and manipulation robots, human hands, tool-use, and, in specialized domains, medical robots, endoscopes, and flexible instruments (Consortium et al., 22 Apr 2026, Lee et al., 11 Jun 2026, Lee et al., 26 Nov 2025).
• Normalization Protocols: Coordination spaces are standardized—e.g., all traces expressed in reference camera frames, all actions in world-relative space, or z-normalized per-robot control vectors (Lee et al., 26 Nov 2025, Consortium et al., 22 Apr 2026).
• Morphology-Aware Pooling: Clustering by morphology graph similarity using metrics like Fused Gromov-Wasserstein, enforcing within-group consistency in RL policy updates and reducing inter-robot gradient conflict (Abe et al., 20 Feb 2026).
• Object-Centricity: Motion is tracked and annotated primarily for entities relevant to manipulation or task fulfillment, not the actuator alone (Zhi et al., 6 Jun 2025, Lee et al., 11 Jun 2026).
• Speed Retargeting: Time-warping/arc-length normalization and fixed-length trajectories facilitate learning from varied-speed actors (Lee et al., 26 Nov 2025, Lee et al., 11 Jun 2026).

These mechanisms support adaptation and joint learning across robots, humans, and varied sensor and kinematic regimes.

5. Evaluation Metrics and Downstream Applications

Evaluation of models trained on trace datasets leverages geometric, semantic, and task-specific criteria:

• Geometry: End-Point Error (EPE) in meters, Angular Error (AE) in degrees, trace-flow field smoothness (Laplacian regularization) (Zhi et al., 6 Jun 2025).
• Discrete Success: Fraction of manipulation tasks completed successfully; for policy transfer, task-specific success rates across embodiments (e.g., 80% for robot, 67.5% for uncalibrated human-to-robot) (Lee et al., 26 Nov 2025, Zhi et al., 6 Jun 2025).
• Semantic Alignment: Captioning accuracy, event-detection precision, and temporal alignment between traces and language (Lee et al., 11 Jun 2026).
• Conflict Mitigation: Gradient overlap/negative transfer metrics during RL pretraining, especially as the diversity and suboptimality of included demonstrations increase (Abe et al., 20 Feb 2026).

Applications include cross-platform policy pre-training, multi-embodiment world modeling, imitation and offline reinforcement learning, robotics foundation model construction, simulation-to-real transfer, and evaluation of instrumentation for surgical and industrial robots.

6. Dataset Accessibility and Licensing

Representative datasets are broadly available for academic research under open licenses:

• TraceForge-123K: Project page with downloads and documentation (https://tracegen.github.io) (Lee et al., 26 Nov 2025).
• ManiFlow-110k: Available in compressed NPZ and JSON format, split into canonical train/val/test partitions (Zhi et al., 6 Jun 2025).
• Open-H-Embodiment: Hosted at Hugging Face Datasets, CC-BY-4.0 license, full protocol and codebase on GitHub (https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-Open-H-Embodiment) (Consortium et al., 22 Apr 2026).
• Offline RL & μ₀ Locomotion: Data organization and code release instructions detailed in their respective appendices and repositories (Abe et al., 20 Feb 2026, Lee et al., 11 Jun 2026).

Each dataset specifies data modality, episode/task segmentation, granularity, and recommended splits for pretraining, policy learning, and model evaluation, enabling reproducibility and cross-benchmarking.

7. Cross-Embodiment 3D Traces in Contemporary Research

Trace-based representations constitute the backbone of recent advances in action-free world modeling, hierarchical event segmentation, and scalable cross-embodiment policy learning. They enable abstraction from hardware, sensor, and environmental details, supporting rapid adaptation (few-shot transfer), robustness across manipulation styles (e.g., human hand vs. robot gripper), and compositional integration with language supervision. Probabilistic flow and B-spline-based models, as exemplified in μ₀ and TraceGen, have demonstrated competitive or superior performance to action-labeled VLA models, validating the sufficiency of geometric trace supervision (Lee et al., 11 Jun 2026, Lee et al., 26 Nov 2025).

A plausible implication is that future datasets may further unify perception and action interfaces through standardized, annotation-rich 3D trace spaces, bridging physical intelligence across domains with minimal tuning or supervision.