SLEDGE: Synthesizing Driving Environments with Generative Models and Rule-Based Traffic (2403.17933v2)

Abstract: SLEDGE is the first generative simulator for vehicle motion planning trained on real-world driving logs. Its core component is a learned model that is able to generate agent bounding boxes and lane graphs. The model's outputs serve as an initial state for rule-based traffic simulation. The unique properties of the entities to be generated for SLEDGE, such as their connectivity and variable count per scene, render the naive application of most modern generative models to this task non-trivial. Therefore, together with a systematic study of existing lane graph representations, we introduce a novel raster-to-vector autoencoder. It encodes agents and the lane graph into distinct channels in a rasterized latent map. This facilitates both lane-conditioned agent generation and combined generation of lanes and agents with a Diffusion Transformer. Using generated entities in SLEDGE enables greater control over the simulation, e.g. upsampling turns or increasing traffic density. Further, SLEDGE can support 500m long routes, a capability not found in existing data-driven simulators like nuPlan. It presents new challenges for planning algorithms, evidenced by failure rates of over 40% for PDM, the winner of the 2023 nuPlan challenge, when tested on hard routes and dense traffic generated by our model. Compared to nuPlan, SLEDGE requires 500$\times$ less storage to set up (<4 GB), making it a more accessible option and helping with democratizing future research in this field.

• The paper presents a novel framework using an RVAE and Diffusion Transformer to synthesize static lane graphs and dynamic agent movements.
• The approach achieves high fidelity and scalability by generating diverse driving scenes, including long routes up to 500 meters.
• The method offers enhanced simulation control, reducing storage needs while uncovering over 40% failure rates in state-of-the-art motion planning algorithms.

SLEDGE: Pioneering Generative Simulation for Enhanced Vehicle Motion Planning

Introduction

Current advancements in autonomous driving heavily rely on rigorous testing through simulation. However, generating simulation environments that accurately represent real-world driving conditions remains a significant challenge. Traditional methods either fall short in realism or demand extensive computational resources. SLEDGE introduces a novel approach by employing generative models to synthesize simulation environments for driving agents, particularly focusing on vehicle motion planning. This approach not only achieves high fidelity in replicating real-world scenarios but also offers remarkable efficiency and flexibility in generating diverse driving conditions.

Efficient and Scalable Generative Modeling

At the core of SLEDGE lies a novel generative model that skillfully synthesizes both the static elements of driving scenes, such as lane graphs, and dynamic elements like vehicle and pedestrian movements. The underlying architecture of this model, termed the raster-to-vector autoencoder (RVAE), coupled with a Diffusion Transformer (DiT), effectively encodes various entities into a rasterized latent map (RLM). This dual structure not only facilitates lane-conditioned agent generation but also supports the concurrent generation of lanes and agents. The approach remarkably outperforms existing baselines in generating realistic, simulation-ready driving scenes, as evidenced by comprehensive benchmarking metrics.

Enhanced Control and Versatility in Simulation

SLEDGE introduces unparalleled control over the simulation environment, allowing researchers to fine-tune conditions such as traffic density, route complexity, and scenario duration. This facilitates the generation of 'hard' routes with increased traffic density, presenting new challenges for motion planning algorithms. Notably, SLEDGE's capability to generate long routes, extending up to 500 meters,—which is not feasible with current data-driven simulators like nuPlan,—uncovers new failure modes in state-of-the-art planning algorithms. Such extensive testing scenarios were instrumental in revealing the limitations of the PDM-Closed planner, which showcased failure rates exceeding 40\% under complex conditions generated by SLEDGE.

Implications and Applications in Autonomous Driving

The generative nature of SLEDGE not only offers an efficient alternative to traditional, log-replay simulators by dramatically reducing storage requirements but also democratizes access to high-quality simulation environments. By enabling rigorous testing over a wide range of driving conditions, SLEDGE is set to significantly impact the development and validation of future autonomous driving systems. It presents an exciting avenue for exploring the intersection between generative AI and autonomous driving, pushing the boundaries of what is possible in vehicle motion planning research.

Future Directions

While SLEDGE marks a significant leap forward, it also opens up new challenges and opportunities for further research. Expanding the field of view, incorporating more complex lane representations, and enhancing the model's efficiency are crucial areas for future development. The integration of techniques for accelerating diffusion models holds the promise of making SLEDGE even more accessible and impactful. As the autonomous driving landscape continues to evolve, the continued refinement and application of generative simulators like SLEDGE will be pivotal in realizing safe, efficient, and intelligent autonomous vehicles.

In conclusion, SLEDGE represents a cutting-edge approach in leveraging generative models for synthesizing driving simulations. Its ability to generate highly realistic and diverse driving scenarios offers profound implications for the future of autonomous vehicle development and testing. As this technology continues to evolve, it stands to revolutionize the field of vehicle motion planning through enhanced realism, efficiency, and control in simulation environments.