SimScale: Autonomous Driving Simulation

• SimScale is a scalable simulation platform that generates photorealistic, reactive scenarios for safety-critical and out-of-distribution autonomous driving simulations.
• It integrates neural rendering with a controlled perturbation pipeline to synthesize realistic sensor data and improve decision-making policy robustness.
• The system leverages pseudo-expert trajectories and detailed scene reconstruction via 3D Gaussian Splatting to create diverse training datasets.

SimScale is a scalable simulation system designed to advance autonomous driving by generating photorealistic, reactive simulations of safety-critical and out-of-distribution (OOD) scenarios, which are typically underrepresented in real-world driving logs. The framework leverages neural rendering techniques and pseudo-expert data generation to synthesize challenging, unseen states directly from real driving trajectories, enabling co-training strategies that measurably improve robustness and generalization for decision-making policies (Tian et al., 28 Nov 2025).

1. Motivation and Conceptual Foundation

Autonomous driving demands policies calibrated for rare, safety-critical, and OOD events, such as near-collisions and off-road deviations. Nominal real-world data, mainly collected during routine driving, lacks sufficient representation of these challenging cases. Direct expansion of such corpora is cost-inefficient and does not proportionally increase coverage of rare scenarios. SimScale addresses this problem by integrating a scalable perturbation pipeline that warps expert trajectories to explore new OOD states, neural rendering to create multi-view observations, and pseudo-expert trajectory generation to provide action labels for synthetic samples. As a result, autonomous agents can be trained on a richer and more diverse dataset while maintaining high fidelity to the underlying sensorimotor inputs encountered in the real world.

2. Simulation Pipeline and System Architecture

2.1 Scene Reconstruction and Neural Rendering

SimScale uses block-wise 3D Gaussian Splatting (3DGS) for scene reconstruction, modeling both static backgrounds and dynamic assets (vehicles). The simulation at each timestep $t$ utilizes camera intrinsics ($K_t$), extrinsics ($E_t$), and the $6$-DoF poses $\{(x_{i,t},y_{i,t},\theta_{i,t})\}_{i=0}^N$ for all entities. Blocks with novel-view PSNR below $27$ dB are excluded to ensure rendering quality. Exposure alignment and semantic grouping, often leveraging LiDAR data, standardize inputs across multiple views. The renderer $\Phi$ transforms the scene state to RGB images $I_t$ with a rendering loss function:

$$\mathcal{L}_{render} = \sum_{p \in \text{pixels}} \| I^{gt}(p) - I^{pred}(p) \|_1$$

2.2 Reactive Simulation and Data Extraction

Simulation proceeds in two main phases within each clip: the perturbation phase applies controlled trajectory adjustments to the ego vehicle, while agents follow the Intelligent Driver Model (IDM). This generates novel OOD ego states. The expert phase uses a pseudo-expert policy $\pi_{exp}$ to provide action labels and state rollouts, producing highly interactive and realistic sensor data through $\Phi$.

2.3 Trajectory Perturbation

SimScale constructs a vocabulary $\mathcal{V}_c$ of $16\,384$ human-derived trajectories. Perturbations shift trajectory endpoints with bounded longitudinal ($r_{lon} \leq 20\,\mathrm{m}$), lateral ($r_{lat}\leq 2\,\mathrm{m}$), and heading ($\Delta \text{heading} \leq 20^\circ$) changes, filtered for collision-free and feasible transitions.

3. Pseudo-Expert Trajectory Generation

Training supervision for perturbed states relies on two pseudo-expert strategies:

• Recovery-based Expert: Selects the closest human maneuver from vocabulary $\mathcal{V}_h$ by minimizing the $L_1$ distance in pose space

$$\tilde{a}^* = \arg\min_{a \in \mathcal{V}_h} \| \mathbf{m}(a) - \mathbf{m}_r \|_1$$

providing human-like corrective actions but with limited diversity.

• Planner-based Expert: Employs privileged rule-based planners (e.g., PDM-Closed), executing optimal, exploratory rollouts

$\tilde{a}^* = \mathbf{P}(\tilde{s}_{t:t+H})$

which may deviate stylistically from human data but enhance coverage.

All expert strategies filter rollouts for traffic rules, kinematic limits, and minimum sub-metrics except for lenient ego-progress criteria.

4. Co-Training Regimen and Learning Objectives

SimScale implements a joint co-training framework with a fixed real dataset ($\mathcal{D}$) and expanding sets of simulated data ($\mathcal{D}_{sim}$), forming a hybrid training distribution. Minibatches sample from $\mathcal{D} \cup \mathcal{D}_{sim}$, supporting multiple planner architectures:

• Imitation Loss (Regression/Diffusion):

$$\min_\theta \mathbb{E}_{(o,a) \sim \mathcal{D} \cup \mathcal{D}_{sim}} \left[ \mathcal{L}_{im}(a, \hat{a}) \right]$$

• Reward Distillation (Scoring-based):

$$\min_\theta \mathbb{E}_{(o,a,r)} [ \lambda \mathcal{L}_{im} + \mathcal{L}_r(r, \hat{r}) ]$$

A combined weighting scheme $$\mathcal{L} = \lambda_{real} \mathcal{L}(\mathcal{D}) + \lambda_{sim} \mathcal{L}(\mathcal{D}_{sim})$$, with typical $\lambda_{real} = \lambda_{sim} = 1$, balances both sources.

5. Datasets, Benchmarks, and Quantitative Results

SimScale utilizes the NAVSIM-v2 navtrain corpus for real-world driving scenarios ($100\,\mathrm{K}$ clips), supplemented with $140\,\mathrm{K}$ recovery-based and $185\,\mathrm{K}$ planner-based synthetic scenes. Evaluation spans "navhard" (challenging and synthetic OOD cases, two stages) and "navtest" (diverse real scenarios). The primary metric, EPDMS, aggregates sub-metrics for critical driving criteria:

$$\mathrm{EPDMS} = \left( \prod_{m \in \mathcal{M}_{\mathrm{pen}}} S_m \right) \cdot \left( \sum w_m S_m \right) / \sum w_m$$

Planner Model	Params (M)	navhard Stage 1+2	navhard ΔEPDMS	navtest ΔEPDMS	Best Mode
LTF (Regression)	56	24.4 → 30.2	+24%	—	Planner-based simulation
DiffusionDrive	61	27.5 → 32.8	+20%	—	Planner-based simulation
GTRS-Dense (V2-99)	83	41.9 → 47.2	+13%	+2.9	Reward-only scoring

Reward-only scoring for GTRS-Dense yields superior EPDMS, indicating reward supervision can suffice when aligned with task objectives.

6. Simulation Scaling and Ablation Analysis

Simulation data scaling exhibits distinct characteristics for different expert strategies and architectures. Planner-based expert scaling curves maintain linearity in $\log N$ (the total sample count), supporting continuous improvement. Recovery-based experts saturate quickly, reflecting limited OOD reach. DiffusionDrive (multi-modal) scales linearly, handling data diversity efficiently. LTF (uni-modal) degrades when simulation data exceeds real data due to demonstration confusion. Reactive (IDM-controlled) environments produce more realistic agent interactions and confer +1.5 EPDMS versus non-reactive setups, despite fewer samples. Ensemble methods averaging scores yield an additional +4–5 EPDMS.

7. Critical Insights, Limitations, and Prospects

SimScale reveals that high-fidelity, reactive simulation of OOD events, combined with feasible pseudo-expert policies, unlocks latent value in human driving logs. Sim-real co-training strategies robustly increase both planning robustness (navhard) and generalization (navtest), and these gains scale smoothly with simulated data volumes alone. Noteworthy findings include the significance of exploratory pseudo-experts, agent interaction modeling, and the advantage of multi-modal policy architectures. Reward-only supervision is viable for scoring planners. Limitations include the need for more diverse traffic generators (diffusion-based), self-evolving perturbations, richer sensor modalities (e.g., LiDAR), and integration with online RL and self-play paradigms. SimScale releases open-source 3DGS simulation datasets and training code to facilitate scalable simulation research in end-to-end autonomous driving (Tian et al., 28 Nov 2025).