Sensor2Sensor: Cross-Embodiment Sensor Conversion for Autonomous Driving

Abstract: Robust training and validation of Autonomous Driving Systems (ADS) require massive, diverse datasets. Proprietary data collected by Autonomous Vehicle (AV) fleets, while high-fidelity, are limited in scale, diversity of sensor configurations, as well as geographic and long-tail-behavioral coverage. In contrast, in-the-wild data from sources like dashcams offers immense scale and diversity, capturing critical long-tail scenarios and novel environments. However, this unstructured, in-the-wild video data is incompatible with ADS expecting structured, multi-modal sensor inputs for validation and training. To bridge this data gap, we propose Sensor2Sensor, a novel generative modeling paradigm that translates in-the-wild monocular dashcam videos into a high-fidelity, multi-modal sensor suite (AV logs) comprising multi-view camera images and LiDAR point clouds. A core challenge is the lack of paired training data. We address this by converting real AV logs into dashcam-style videos via 4D Gaussian Splatting (4DGS) reconstruction and novel-view rendering. Sensor2Sensor then utilizes a diffusion architecture to perform the generative conversion. We perform comprehensive quantitative evaluations on the fidelity and realism of the generated sensor data. We demonstrate Sensor2Sensor's practical utility by converting challenging in-the-wild internet and dashcam footage into realistic, multi-modal data formats, further unlocking vast external data sources for AV development.

• The paper introduces a novel generative pipeline that converts monocular dashcam inputs into synchronized multi-modal AV sensor logs, bridging the embodiment gap.
• It leverages 4D Gaussian Splatting to synthetically generate paired data from real-world logs and diverse dashcam views, enabling effective supervised training.
• Empirical evaluations demonstrate superior image and LiDAR generation quality with improved temporal stability and cross-modal alignment, validating its practical impact.

Cross-Embodiment Sensor Generation for Autonomous Driving: An Analysis of Sensor2Sensor

Motivation and Problem Statement

High-fidelity validation and training of autonomous driving stacks critically depend on rare, long-tail scenarios, which are underrepresented in proprietary AV fleet datasets due to cost and operational constraints. Conversely, consumer dashcam and third-party driving videos offer large-scale, diverse coverage of such edge cases but are generally limited to monocular video from unknown sensors and perspectives. This presents a severe embodiment gap: the structurally and geometrically mismatched sensor formats preclude direct utility for AV system development.

Sensor2Sensor addresses this domain adaptation bottleneck by proposing a generative paradigm that translates in-the-wild, monocular driving footage into fully-synchronized, multi-modal AV sensor logs that are specific to a target vehicle sensor suite. The approach represents a significant leap in leveraging heterogeneous, unpaired data for scalable, safety-critical AV system validation and simulation.

Figure 1: Sensor2Sensor synthesizes full AV sensor logs from arbitrary monocular video sources, enabling broadening of the training and validation corpus via cross-embodiment sensor conversion.

Methodology

Paired Data Synthesis via 4D Gaussian Splatting

A central innovation is the synthetic generation of large-scale paired (input, target) data, circumventing the absence of real-world synchronized dashcam/AV logs. Source AV logs with 360-degree camera and LiDAR coverage are reconstructed as dynamic 4D Gaussian Splatting (4DGS) scenes, supporting high-fidelity novel-view synthesis. Diverse virtual dashcam perspectives and optics—parameterized by sampled intrinsic and extrinsic distributions reflecting real-world dashcam heterogeneity—are rendered from these 4DGS reconstructions. This pairing aligns novel monocular views with their ground-truth AV sensor suite targets, yielding a supervised training corpus.

Figure 2: Synthetic paired-data curation pipeline: reconstructing dynamic 4DGS scenes from fleet data and rendering diverse monocular dashcam-style inputs.

Multi-Modal, Multi-Sensor Conditional Diffusion Model

Sensor2Sensor implements a generative latent diffusion architecture tailored for synchronized multi-view camera and LiDAR generation, conditioned on a single input monocular sequence. The architecture consists of modality-specific VAEs for both image and LiDAR “spin image” representations, and parallel UNet towers interconnected via cross-sensor attention. Multi-view spatial and temporal consistency is enforced through 3D attention modules operating over the latent space. The model accepts a “conditional” input view, explicitly partitioned as the unnoised reference (dashcam/monocular) and outputs eight AV-aligned camera perspectives plus LiDAR, discarding the conditioning input from the output loss.

Figure 3: Multi-modal, multi-view latent diffusion architecture with modality-specific VAEs and cross-sensor attention fusion.

Temporal Video Generation and DAgger Rollouts

For video, the model is extended with an auto-regressive structure. Each timestep’s multi-sensor output is conditioned not only on the current input frame but also on previous self-generated outputs. To mitigate distributional shift and error propagation (“drifting”) in long unrolled sequences, a DAgger-style dataset aggregation loop is employed: the network is periodically retrained on its own rollouts, blending ground-truth and synthetic contexts to enhance temporal robustness.

Figure 4: Video rollout comparison: DAgger training improves temporal consistency versus standard auto-regressive inference.

Results and Empirical Evaluation

The model is evaluated on a curated, synchronized fixed-camera-to-AV sensor suite dataset and an expansive selection of in-the-wild internet and dashcam videos. Baselines include state-of-the-art 3D reconstruction, multi-view generative models, and cross-modal diffusion variants.

Multi-view Image Generation: Sensor2Sensor achieves FID 6.47 and LPIPS 0.316—significantly outperforming both feedforward reconstructive models and adapted generative baselines. The generated views preserve geometric consistency, accurately maintain scene structure, and exhibit coherent inter-view appearance.

Figure 5: Sensor2Sensor reconstructions are geometrically faithful with sharp object and scene delineation versus baselines.

Video Generation: Achieving FVD 278.12 and lowest per-frame error metrics, Sensor2Sensor demonstrates substantial improvements in temporal stability without sacrificing frame-wise fidelity, outperforming prior approaches and ablations in both short and extended rollouts.

LiDAR Co-Generation: On LiDAR, Sensor2Sensor attains a Chamfer distance of 8.68, a 13.4% improvement over the closest baseline (X-Drive). The model is highly effective in generating noise-free, spatially accurate point clouds and shows robust cross-modal consistency, as visualized by direct alignment between synthesized LiDAR and corresponding image outputs.

Figure 6: Joint image and LiDAR generation exhibits strong cross-modal correspondence and accurate geometry, critical for AV perception.

Generalization to In-the-Wild Data: When applied to raw dashcam and internet videos with previously unseen optics, mounting, lighting, and semantics, Sensor2Sensor generates plausible, coherent sensor logs even in highly challenging conditions (e.g., night scenes, active accidents, long-tail incidents). In human evaluation, image and LiDAR generations are preferred in 83.5–84.6% and 58.5–68% of cases over leading baselines.

Figure 7: Generalization: Sensor2Sensor synthesizes full sensor suites from real-world dashcam and internet inputs, including rare and adverse scenarios.

Downstream Perception Robustness: Standard vehicle detection and semantic segmentation models, when applied to Sensor2Sensor-generated data, perform comparably to their application on real-world sensor logs, confirming the utility of the generated data for AV system validation and development.

Architectural and Training Effect Analysis

Ablation studies reveal that the model’s performance depends critically on:

• View Concatenation vs. Channel Concatenation: Conditioning via view concatenation in the latent space yields superior generative quality versus direct channel concatenation.
• Cross-Sensor Attention: Jointly attending over camera and LiDAR features within each UNet block is essential for cross-modal alignment.
• DAgger Training: Auto-regressive training with dataset aggregation leads to more stable, temporally coherent rollouts, as evidenced by FVD and FID improvements.

Implications and Future Directions

Sensor2Sensor enables the large-scale utilization of heterogeneous, uncalibrated video data sources for simulation, validation, and potential training of AV stacks, dramatically widening the space of real-world edge-case data available to developers. By successfully translating from single-view, monocular input to a synchronized, multi-view, multi-modal sensor suite, the approach opens avenues in:

• Systematic harvesting of third-party and consumer video for safety-critical validation.
• Crowdsourced or partner-assisted post-hoc scenario synthesis for rare event cataloging.
• Development of generalist, embodiment-agnostic autonomous driving models via data normalization.
• Potential adaptation to additional sensor modalities (e.g., radar, event cameras) or new AV configurations via retraining or architecture extension.

Persisting challenges include long-horizon temporal drift, which can be mitigated by expanding context modeling beyond previous-frame conditioning, more powerful video backbones, or explicit spatial-temporal consistency regularizers. Scaling to more diverse real-world sensor suites and formal systematic evaluation of sensor fidelity for simulation-to-real transfer are open research foci.

Conclusion

Sensor2Sensor operationalizes a generative sensor translation framework that bridges the embodiment gap between unstructured monocular video and structured AV sensor suites. By leveraging high-fidelity synthetic pairing and cross-modal diffusion, the system attains state-of-the-art performance on multi-view, multi-modal generation from unconstrained inputs. Sensor2Sensor’s demonstrated generalization to novel, long-tail, and real-world scenarios establishes a practical foundation for scalable, robust simulation, validation, and data augmentation in AV development pipelines (2605.22809).