SimGen: Simulator-conditioned Driving Scene Generation (2406.09386v3)

Abstract: Controllable synthetic data generation can substantially lower the annotation cost of training data. Prior works use diffusion models to generate driving images conditioned on the 3D object layout. However, those models are trained on small-scale datasets like nuScenes, which lack appearance and layout diversity. Moreover, overfitting often happens, where the trained models can only generate images based on the layout data from the validation set of the same dataset. In this work, we introduce a simulator-conditioned scene generation framework called SimGen that can learn to generate diverse driving scenes by mixing data from the simulator and the real world. It uses a novel cascade diffusion pipeline to address challenging sim-to-real gaps and multi-condition conflicts. A driving video dataset DIVA is collected to enhance the generative diversity of SimGen, which contains over 147.5 hours of real-world driving videos from 73 locations worldwide and simulated driving data from the MetaDrive simulator. SimGen achieves superior generation quality and diversity while preserving controllability based on the text prompt and the layout pulled from a simulator. We further demonstrate the improvements brought by SimGen for synthetic data augmentation on the BEV detection and segmentation task and showcase its capability in safety-critical data generation.

• The paper introduces SimGen, a novel simulator-conditioned framework that fuses real-world and simulated data to generate diverse driving scenes.
• It employs a cascade diffusion strategy to bridge the simulation-to-real gap, achieving realistic scene generation through progressive denoising and multimodal integration.
• Experimental results with the extensive DIVA dataset show improved FID scores and enhanced scene diversity, advancing autonomous system training.

An Analysis of "SimGen: Simulator-conditioned Driving Scene Generation"

The paper "SimGen: Simulator-conditioned Driving Scene Generation" introduces a robust framework, SimGen, aimed at enhancing the quality and diversity of driving scene generation for autonomous vehicle training. By leveraging both real-world data and simulator inputs, the authors seek to address the prevalent limitations in training data compilation for autonomous driving systems, which primarily hinge on high annotation costs and data diversity issues.

The SimGen framework is structured to blend data from both real-world scenarios and synthetic simulators, greatly expanding the diversity of generated scenes. This is facilitated through a novel cascade diffusion model designed to overcome the Sim2Real (Simulation to Reality) gap, inherently present in simulation-generated data. The authors further extend these capabilities by integrating textual prompts into the generation pipeline, enhancing the flexibility and control over the generated data scenarios.

Key Technical Contributions

1. Simulator-Conditioned Scene Generation Framework: The proposal of the SimGen framework introduces a significant shift from traditional data generation models. Unlike previous works which predominantly relied on small-scale, narrowly varied datasets (e.g., nuScenes), SimGen capitalizes on a sophisticated fusion of real and simulated dataset inputs. This fusion fosters the generation of a broader spectrum of driving scenes, addressing both appearance and layout diversity challenges.
2. Cascade Diffusion Strategy: The introduction of a cascade diffusion model is pivotal in bridging the Sim2Real gap. This model rigorously translates simulated conditions into realistic conditions, subsequently aiding the accurate generation of driving scenes. By introducing noise into the simulation conditions and refining these through a progressive denoising network, the generated scenes align more closely with real-world conditions.
3. The DIVA Dataset: A significant contribution of the paper is the curation of the DIVA dataset, comprising 147.5 hours of driving video, amalgamating real-world driving data sourced from YouTube and synthetic contributions from the MetaDrive simulator. This dataset is notable for its breadth, encompassing diverse geographical locations, weather conditions, and traffic scenarios, crucial for training generalized autonomous systems.
4. Multimodal Condition Integration: The framework employs a unified adapter to reconcile multimodal input conditions, including depth, semantic segmentation, and textual prompts. This adapter mitigates potential conflicts among input modalities, ensuring coherent scene generation.

Empirical Results and Impact

The empirical evaluations of SimGen demonstrate substantial improvements in both the quality and diversity of generated scenes compared to existing methodologies. The framework surpasses contemporaneous approaches in generating realistic and diverse driving scenarios, as evidenced by superior performance on frame-wise Fréchet Inception Distance (FID) and diversity metrics. Additionally, the authors highlight the utility of SimGen in augmenting real datasets for synthetic data generation, enriching perception model training.

Future Research and Implications

SimGen's contribution lies not only in enhancing data diversity but also in providing a framework applicable to myriad scenarios beyond what current datasets offer. The scalable and flexible nature of SimGen could significantly impact the development of autonomous vehicle systems by facilitating more comprehensive and realistic training environments, thus potentially improving system robustness and safety.

The paper opens avenues for future research, particularly in multi-view generation and real-time applications, which would further propel the capabilities of autonomous systems in real-world deployments. Moreover, extending the SimGen framework to encompass dynamic and interactive scenarios could revolutionize closed-loop evaluation processes, thereby providing a more holistic approach to autonomous vehicle testing.

In summary, the paper provides valuable insights and methodologies that enhance the simulation fidelity and diversity of driving scenes, directly contributing to the foundational resources required for advanced autonomous system development.