SymDrive: 3D Driving Simulation

• SymDrive is a unified diffusion-based simulation framework that enables high-fidelity, controllable 3D driving scenes by combining 3D Gaussian Splatting with symmetric auto-regressive restoration.
• It leverages dual symmetric views and an auto-regressive restoration chain to recover detailed textures and maintain consistency across large lateral viewpoint shifts.
• The framework supports context-aware vehicle insertion through latent inpainting, enabling seamless asset harmonization and scene-consistent editing without additional retraining.

SymDrive is a unified diffusion-based simulation framework for high-fidelity, controllable 3D driving scenes, addressing the persistent challenges in photorealistic novel-view synthesis and interactive traffic editing such as vehicle insertion. The approach combines a Symmetric Auto-regressive Online Restoration paradigm with context-aware inpainting, achieving joint state-of-the-art rendering and seamless 3D asset harmonization. SymDrive builds on a 3D Gaussian Splatting (3DGS) backbone, leveraging dual-view symmetry and auto-regressive restoration to recover fine-grained details and maintain consistency across large lateral viewpoint shifts and manipulated scenes (Liu et al., 25 Dec 2025).

1. Challenges in Photorealistic Driving Simulation

Realistic 3D simulation for autonomous driving (AD) requires both photorealistic scene generation and interactive, artifact-free editing. Prior single-view renderers (e.g., NeRF, 3DGS) encounter two major obstacles:

• Large-angle novel-view synthesis: Significant lateral or angular viewpoint changes expose geometric incompleteness and texture degradation, producing blurred lane markings or distorted vehicle geometry where the input coverage is limited or occluded.
• Interactive traffic editing: Manipulating scene objects, such as inserting novel vehicles, often creates visible artifacts—holes, ghosting, or lighting mismatches—because separately handled foregrounds lack explicit context integration. Existing methods relying on synthetic perturbations or costly asset labeling further limit scalability.

The root causes are incomplete multi-view constraints and insufficient cross-view consistency: without additional, context-aware priors, conventional methods cannot reliably infer or restore fine details outside observed regions, nor harmonize new objects to background scene statistics (Liu et al., 25 Dec 2025).

2. SymDrive Unified Framework

SymDrive augments a 3DGS model with a diffusion-based “restorer” module operating on dual symmetric views and enables both high-quality view synthesis and training-free context-aware vehicle insertion within a cohesive pipeline.

1. 3DGS Backbone: Trained on ground-truth trajectories, reconstructs background and foreground vehicles as separate entities.
2. Diffusion Data Generation: For each GT image $I_0$ at pose $C_0$, symmetric lateral offsets $\pm d$ generate $I_{+d}$ and $I_{-d}$ for dual-view restoration training.
3. Restoration Diffusion Model: The model $v_\theta$, conditioned on diagnostic image pairs, learns to recover the central GT image, exploiting geometric correspondence and symmetry.
4. Auto-regressive Restoration Chain: At inference, a lateral “rollout” is constructed by sequentially restoring chained viewpoints using previously restored neighbors and additional raw renderings, thereby propagating ground-truth details outward.
5. 3DGS Refinement: The backbone is further fine-tuned using synthesized novel views as additional supervision, jointly optimizing RGB, SSIM, and depth losses.
6. Vehicle Insertion as Latent Inpainting: A masked RePaint-style inpainting loop harmonizes new 3DRealCar models, maintaining consistent lighting, shadows, and color statistics by iteratively resetting the unedited latent context (Liu et al., 25 Dec 2025).

3. Symmetric Auto-regressive Online Restoration Paradigm

The core technical innovation is the use of paired, ground-truth-guided, symmetric views for both restoration and enhancement, enabling robust cross-view feature fusion and accurate occlusion reasoning.

• Paired View Construction: For renderer $G$ and pose $C_0$, generate triplet $(I_{-d}, I_0, I_{+d})$, where $d$ is a fixed lateral shift.
• Dual-view Restoration Objective: Train $R_\theta$ to minimize the symmetric $L_1$ objective

$$L_\text{dual} = \mathbb{E}_{I_{-d}, I_d, I_0}\left[\|R_\theta(I_{-d}, I_d) - I_0\|_1 + \|R_\theta(I_d, I_{-d}) - I_0\|_1\right]$$

In practice, training occurs in VAE latent space using a flow-matching diffusion objective

$$L = \mathbb{E}_{z_0, \epsilon, t} \| v_\theta([z_{-d}; z_{0,t}; z_d], t) - (\epsilon - z_0) \|_2^2$$

• Auto-regressive Synthesis: Start from $I_0$, recursively restore $I_{kd}$ using the restored previous step and raw rendering at $(k+1)d$, initializing denoising from a partially noised latent to enforce structure alignment (Eq. 5).

This paradigm enables propagation of high-fidelity details from the center to novel rolls, maintaining geometric and appearance consistency even at large viewpoint displacements (Liu et al., 25 Dec 2025).

4. Context-aware Vehicle Insertion and Harmonization

SymDrive’s context-aware harmonization treats vehicle insertion as latent-space inpainting without requiring explicit retraining for each scenario.

• Insertion Pipeline:

1. Render scene with the new vehicle, obtaining $I_\text{insert}$ and encode to $z_\text{insert}$.
2. Construct binary mask $M$ isolating vehicle pixels.
3. Apply RePaint-style denoising, where at each step $t$ the background (outside $M$) is reset and only the vehicle region latent is updated.
4. Produce harmonized composite $\tilde{I}_\text{insert}$ with visually matched lighting and color.
5. Final refinement: optimize vehicle’s color and opacity in $G_v$ by matching $\tilde{I}_\text{insert}$ in pixel and SSIM losses.

This strategy achieves scene-consistent insertion of multiple vehicles, with shadows, highlights, and global statistics blending naturally with the existing scene, without supervision specific to new insertions (Liu et al., 25 Dec 2025).

5. Model Architecture and Training Regimen

SymDrive implements the following architecture:

• Diffusion Model: Backbone uses Flux.1-dev, a flow-matching diffusion network.
• Encoder/Decoder: Standard VAE for latent mapping.
• Conditioning: Input to $v_\theta$ is [$z_{-d}; z_{0,t}; z_d$], capturing symmetric context.
• Optimization: LoRA (rank 128) for efficient diffusion model fine-tuning; lateral shift $d=0.5$ m, 50 denoising steps (noise initiation at $N_\text{start} = 10$), 20k LoRA steps with 4×A100 GPUs.
• 3DGS Tuning: Refinement loss

$$L_\text{total} = L_\text{rgb} + \lambda_1 L_\text{ssim} + \lambda_2 L_\text{depth}$$

applied to both GT and restored views for 50k training steps.

This configuration supports both high-throughput training and deployment of the full restoration/insertion pipeline (Liu et al., 25 Dec 2025).

6. Empirical Evaluation

SymDrive’s efficacy is empirically validated on 8 Waymo scenes (~3 m lateral shift), using foreground overlap (NTA-IoU), lane marking overlap (NTL-IoU), and Fréchet Inception Distance (FID):

Method	NTA-IoU ↑	NTL-IoU ↑	FID ↓
Street Gaussians	0.498	53.19	130.75
FreeVS	0.505	53.26	104.23
DriveDreamer4D	0.457	53.30	113.45
ReconDreamer	0.539	54.58	93.56
ReconDreamer++*	0.566	56.89	75.22
Difix3D+	0.578	56.94	84.12
ReconDreamer++†	0.572	57.06	72.02
Ours (SymDrive)	0.582	57.91	74.82

For vehicle insertion harmonization:

Method	Capability	FID ↓
3DRealCar Insert	–	41.27
Difix3D+	novel-view restoration	53.64
CosXL-Edit	pixel-space editing	46.54
Ours	united insertion+rest	32.60

Qualitative results (see Figures 4–9 of (Liu et al., 25 Dec 2025)) demonstrate superior recovery of near-field details, robust lane marking synthesis, and plausible, scene-consistent vehicle insertions. Integrations with SUMO traffic inputs and Vision–Language reasoning (ReCogDrive) further validate the utility in simulated closed-loop settings.

7. Limitations and Prospective Directions

SymDrive is subject to several limitations:

• Far-range objects exhibit sparse sampling and temporal jitter, due to limited visual evidence in GT.
• Temporal consistency is not explicitly modeled in long rollouts; rare flicker can occur.
• The absence of a rigid-body physics engine precludes accurate modeling of collisions and physical contact dynamics.

Suggested directions for improvement include integration of video-diffusion priors (for temporal coherence and speed), embedding a full physics simulator, and extension to 360° panoramic and multi-modal (LiDAR+RGB) inputs (Liu et al., 25 Dec 2025).

SymDrive’s principal advance is its symmetric, dual-view conditioning and auto-regressive restoration chain, enabling joint high-detail recovery and context-aware editing in large-scale, controllable 3D driving simulations.