Digital Twin Tracking Dataset (DTTD)

• Digital Twin Tracking Datasets are rigorously annotated multimodal datasets that combine real sensor recordings with high-fidelity digital twin reconstructions to facilitate cross-domain perception and tracking research.
• They feature extensive sensor suites with precise calibration and synchronization, enabling sub-millimeter pose validation and robust sensor fusion for accurate object tracking.
• DTTDs empower domain adaptation by bridging real and simulated environments with detailed annotations, driving advancements in autonomous driving, mobile robotics, and augmented reality.

Digital Twin Tracking Datasets (DTTDs) are rigorously annotated multimodal corpora integrating real sensor recordings and high-fidelity digital twin synopses of scenes, objects, and agent behaviors. They are engineered to enable the analysis, benchmarking, and development of perception, tracking, and representation algorithms in both physical and virtualized spaces. DTTDs are designed to facilitate direct sim-to-real or reality-to-simulation domain transfer, enabling quantifiable, reproducible study of sensor fusion, object localization, explicit scenario generation, and domain-adaptive algorithmic robustification across a variety of research areas, including autonomous driving, mobile robotics, augmented reality, and intelligent sensing (Neto et al., 2023, &&&1&&&, Huang et al., 2023, Feng et al., 2023, Chen et al., 14 Nov 2025, Rößle et al., 21 Jan 2026).

1. Scope, Architectures, and Foundational Principles

DTTDs systematically couple physical sensor deployments with virtualized scene reconstructions or synthetic emulations, ensuring that recorded phenomena are available for evaluation and manipulation in both domains. Typical architectures include:

• Physical sensor platforms (vehicles, handhelds) equipped with RGB cameras, depth/LiDAR, radar, IMUs, and GNSS/RTK units.
• Simulation backends and hardware-in-the-loop (HIL) environments where digital twins are instantiated with real or simulated sensor models, environmental variability (e.g., adverse weather, lighting), and agent dynamics preserved or composited from reference datasets.
• Synchronized acquisition and alignment mechanisms (e.g., PTP, hardware triggers, or software timebases) guarantee frame-accurate correspondence for cross-domain comparison.

The design of DTTDs is driven by the demand for millimeter- to centimeter-level pose fidelity, fine-grained temporal indexation, and consistent calibration regimes across real and synthetic domains (Neto et al., 2023, Feng et al., 2023, Rößle et al., 21 Jan 2026).

2. Sensors, Calibration, and Synchronization Protocols

DTTDs employ extensive sensor suites and meticulous calibration:

• Typical Modalities: High-resolution RGB cameras (1920×1080 or greater, global shutter), LiDARs (e.g., Velodyne HDL-64E, Ouster OS1-128, RoboSense Ruby Plus, HDL-64E), automotive/IFM radars, time-of-flight depth units (Apple LiDAR or Kinect Azure), GNSS/IMU modules (ADMA-G Pro, SP80, OptiTrack).
• Calibration Procedures:
  • Intrinsic: Determined per session/device (e.g., NVIDIA DriveWorks, ARKit, Azure SDK).
  • Extrinsic: Rigid-body transforms via checkerboard, target-based approaches, or marker-based registration (Aruco/OptiTrack). Extrinsics stored as 4×4 homogeneous matrices mapping world to sensor coordinates:

  $$T_{\text{world}\rightarrow \text{sensor}} = \begin{bmatrix} R & t \ 0 & 1 \end{bmatrix}$$

  with $$p_{\text{sensor}} = T_{\text{world}\rightarrow \text{sensor}}\, p_{\text{world}}$$.

• Synchronization: All major sensors operate at precisely specified rates (10–100 Hz); time offsets corrected by global index matching or Kalman filtering. Ground-truth sources (GNSS, OptiTrack) are linear-interpolated to match acquisition streams (Neto et al., 2023, Huang et al., 2023, Feng et al., 2023, Rößle et al., 21 Jan 2026).

3. Dataset Composition and Annotation Schemes

DTTDs emphasize both scale and annotation rigor:

Dataset (Year)	Modality	Frames	Annotation Types	Classes
TWICE/DTTD [2023]	Camera, Radar, LiDAR, IMU, GNSS	221,495	3D boxes, IDs, GNSS/IMU tracks	Car, Cyclist, Pedestrian, Truck
UrbanTwin [2025]	LiDAR (synthetic/replica)	10,000*3	3D boxes, segmentation, tracks	Car, Van, Bus, Truck, Bicycle, Moto
DrivIng [2026]	6x Camera, LiDAR, GNSS/IMU	63,043	3D boxes, IDs, classes per 10 Hz	12 classes (car, van, bus…)
DTTD v1 [2023]	RGB-D (Azure Kinect)	55,691	Per-pixel mask, 6DoF pose	10 objects, multi-light
DTTD-Mobile [2023]	RGB + Depth (iPhone LiDAR)	47,668	Per-frame 6DoF label, mask	18 objects, high pose accuracy
SynthSoM-Twin [2025]	RGB, Depth, LiDAR, mmWave radar	66,868	2D/3D bounding boxes, trajectories	Vehicle, Pedestrian

*All datasets include scenario, domain, or weather breakdowns, with official protocol splits for training, validation, and held-out testing (Neto et al., 2023, Shahbaz et al., 8 Sep 2025, Feng et al., 2023, Huang et al., 2023, Chen et al., 14 Nov 2025, Rößle et al., 21 Jan 2026).

Annotations include:

• 3D cuboid parameters in world and sensor frames
• Instance masks and segmentation (semantic and instance)
• Consistent persistent tracking IDs via annotation pipeline or direct actor mapping
• Per-frame pose matrices, camera intrinsics, distortion coefficients

Ground-truth pose is typically sub-millimeter accurate (OptiTrack, ADMA) or as per GNSS error estimates. In the presence of adverse weather or sensor dropout, interpolation events are logged.

4. Scenarios, Domains, and Environmental Variation

DTTDs encompass a wide range of domains and scene types to stress-test perception models:

• Controlled Test Tracks: E.g., CARISSMA proving ground with artificial rain, snow, and night setups (Neto et al., 2023).
• Urban/Suburban/Highway: Geo-registered public routes, real traffic, multi-day sequences under distinct lighting: day, dusk, night (Rößle et al., 21 Jan 2026).
• Domain Bridging: Each scenario instantiated both physically (real-world) and digitally (via HIL, CARLA/Unreal replay, or procedural generation in AirSim/Sionna RT).
• Object and Agent Types: Full vehicular taxonomies, vulnerable road users, and common artifacts.

Weather, lighting, and contextual variability are deliberately matched and compared across real/synthetic pairs to facilitate domain adaptation and sim-to-real robustness studies (Neto et al., 2023, Shahbaz et al., 8 Sep 2025, Chen et al., 14 Nov 2025, Rößle et al., 21 Jan 2026).

5. Evaluation Protocols and Metrication

Standardized quantitative evaluation is central to DTTD usage. Recommended metrics include:

• Detection & Instance Segmentation: Intersection-over-Union (IoU), mAP@IoU=0.5, class-wise AP.
• Multi-Object Tracking: MOTA, MOTP, CLEAR metrics, HOTA.

$$\mathrm{MOTA} = 1 - \frac{\sum_t (\mathrm{FN}_t + \mathrm{FP}_t + \mathrm{IDSW}_t)}{\sum_t \mathrm{GT}_t}$$

$$\mathrm{MOTP} = \frac{\sum_{i,t} d_{i,t}}{\sum_t c_t}$$

• 6DoF Pose Estimation: ADD, ADD-S, AUC of correct pose fraction over thresholds. For noisy depth: depth-ADD, quantifying per-pixel disparity between measured and reference depth.
• Sim-to-Real Gap Quantification: Statistical alignment (KL divergence of distributional metrics), structural similarity overlays, empirical generalization benchmarks (train synthetic, test real) (Shahbaz et al., 8 Sep 2025).
• Communication/Channel Regression (where relevant): Path-loss RMSE/MAE, Top-k beam forming accuracy.

Ground-truth and benchmark code is usually provided within the dataset codebase (e.g., via MMDetection3D, or custom scripts for PyTorch DataLoader integration) (Feng et al., 2023, Shahbaz et al., 8 Sep 2025, Rößle et al., 21 Jan 2026).

6. Utility Patterns, Domain Adaptation, and Best Practices

DTTDs are employed for:

• Training and evaluating perception models under adverse conditions using both real and perfectly aligned synthetic data.
• Cross-domain validation and domain adaptation, e.g., fine-tuning on synthetic, testing on real; empirical closure of the sim-to-real gap via explicit protocol.
• Scenario customization, augmentation, and edge-case synthesis: supporting fault injection, parameterized weather, traffic scaling, and agent manipulations in digital twin domains (Neto et al., 2023, Shahbaz et al., 8 Sep 2025, Chen et al., 14 Nov 2025).
• Multi-modal fusion and modality-agnostic transfer: compositing radar/radio, vision, and depth for end-to-end robust tracking and detection pipelines.

Best practices include:

• Rigid calibration and thorough logging of all transformation, time offset, and interpolation steps.
• Use of standardized interfaces (OSI, OpenLABEL, compatible train/val/test split protocols).
• Supervisory annotation refinement via deep learning detectors (e.g., YOLOv4) to align projected and observed bounding boxes.
• Validation of simulated sensors and environmental models against measured distributions (e.g., rainfall DSD in proving ground) (Neto et al., 2023, Shahbaz et al., 8 Sep 2025, Chen et al., 14 Nov 2025).

7. Impact, Limitations, and Ongoing Research Directions

DTTDs have quantitatively demonstrated that training on high-fidelity digital twin data can match or surpass conventional real-world–only models for core perception and tracking tasks (e.g., mAP improvements of up to +1.4 percentage points, MOTA >90% before domain adaptation in UrbanTwin (Shahbaz et al., 8 Sep 2025); equivalent performance at <15% real–data injection in SynthSoM-Twin (Chen et al., 14 Nov 2025)). The persistent challenge remains in closing residual simulation-to-reality gaps linked to unmodeled sensor imperfections, environmental subtleties, or annotation ambiguities.

The field is transitioning toward:

• Expanding coverage (e.g., DrivIng’s full 18 km urban-suburban-highway trajectory (Rößle et al., 21 Jan 2026)).
• Increasing multi-modal scope (joint radio, radar, LiDAR, vision in SynthSoM-Twin (Chen et al., 14 Nov 2025)).
• Enabling programmatic expansion and customization (CARLA/Unreal-based APIs for synthetic generation (Shahbaz et al., 8 Sep 2025)), supporting broader sim-to-real research beyond core object tracking.

A plausible implication is that continued refinement of digital twin generation, synchronization, and multi-modal integration will progressively erode the barriers between synthetic and real-world training, transforming open research in computer vision, robotics, and intelligent transportation systems.