GPUDrive: Data-driven, multi-agent driving simulation at 1 million FPS (2408.01584v3)

Abstract: Multi-agent learning algorithms have been successful at generating superhuman planning in various games but have had limited impact on the design of deployed multi-agent planners. A key bottleneck in applying these techniques to multi-agent planning is that they require billions of steps of experience. To enable the study of multi-agent planning at scale, we present GPUDrive. GPUDrive is a GPU-accelerated, multi-agent simulator built on top of the Madrona Game Engine capable of generating over a million simulation steps per second. Observation, reward, and dynamics functions are written directly in C++, allowing users to define complex, heterogeneous agent behaviors that are lowered to high-performance CUDA. Despite these low-level optimizations, GPUDrive is fully accessible through Python, offering a seamless and efficient workflow for multi-agent, closed-loop simulation. Using GPUDrive, we train reinforcement learning agents on the Waymo Open Motion Dataset, achieving efficient goal-reaching in minutes and scaling to thousands of scenarios in hours. We open-source the code and pre-trained agents at https://github.com/Emerge-Lab/gpudrive.

• The paper introduces GPUDrive, a simulation engine achieving over 1 million agent steps per second for multi-agent reinforcement learning.
• It employs a GPU-accelerated framework with C++ and CUDA, integrating real-world driving data to enhance simulation realism.
• Experiments demonstrate reduced training time for autonomous agents, enabling 95% goal-reaching rates in diverse driving scenarios.

GPUDrive: Data-driven, multi-agent driving simulation at 1 million FPS

The paper "GPUDrive: Data-driven, multi-agent driving simulation at 1 million FPS" by Kazemkhani et al. introduces GPUDrive, a high-performance simulation environment designed to train and evaluate multi-agent reinforcement learning (RL) algorithms for applications in autonomous driving. Leveraging the Madrona Game Engine, GPUDrive achieves an unprecedented simulation speed of over one million frames per second (FPS), facilitating the generation of vast quantities of experience necessary for effective multi-agent learning.

Problem Context and Motivation

Multi-agent learning has demonstrated exceptional performance in various game-theoretic and highly controlled environments. However, its application in real-world multi-agent planning, particularly in scenarios involving human-robot interaction, remains limited. This is primarily due to two significant challenges: 1) the necessity to develop human-compatible strategies without extensive data, and 2) the need for simulators capable of generating the billions of experience samples required by data-intensive RL algorithms.

GPUDrive addresses these challenges by providing a GPU-accelerated simulation framework capable of integrating real-world driving data with fast, high-volume simulational throughput. This design allows for the paper of autonomous planning and human behavior modeling within a robust and scalable simulation environment.

Technical Contributions

The primary contributions of GPUDrive are multifaceted:

1. High-Performance Simulation: GPUDrive can simulate over one million steps per second using consumer-grade and datacenter-class GPUs. This high throughput is facilitated by an efficient implementation in C++ and CUDA, leveraging the Madrona Game Engine's extensible Entity-Component-State (ECS) framework.
2. Complex Observation Spaces: The simulator integrates various sensor modalities, including LIDAR and human-like view cones, offering rich and versatile observation spaces suited for different types of agents.
3. Robust Agent Dynamics: GPUDrive uses both a standard Ackermann bicycle model and a simplified, invertible vehicle model. This flexibility supports detailed simulation of agent dynamics for diverse applications.
4. Integration with Real-World Data: By utilizing datasets such as the Waymo Open Motion Dataset, GPUDrive can mix logged human driving data with synthetic simulations, enhancing the realism and applicability of trained agents.

Numerical Results and Implications

The simulation engine's performance is quantitatively substantiated in the paper through various benchmarks. GPUDrive achieves a peak throughput of over one million agent steps per second (ASPS) on consumer-grade GPUs, significantly surpassing the fastest previous benchmarks, including CPU-based and other GPU-accelerated simulators. This performance accelerates reinforcement learning workflows, reducing the training time for RL agents to solve specific driving scenarios from hours to mere minutes.

Experiments conducted using Independent PPO (IPPO) demonstrate that GPUDrive can train a heterogeneous set of driving agents capable of achieving goal-reaching rates of 95\% across multiple scenarios within two hours. This rapid performance is transformative for research, allowing the exploration and validation of multi-agent learning algorithms at a scale and speed previously unattainable.

Theoretical and Practical Implications

GPUDrive has substantial implications for the development of autonomous driving systems and multi-agent RL:

• Theoretical Advancements: The simulator's capabilities enable researchers to investigate new RL methods under highly realistic and diverse conditions. This setup supports the paper of complex interactions between autonomous systems and human agents, leading to more effective and robust decision-making algorithms.
• Practical Applications: The rapid training capabilities facilitate the development and deployment of high-performing autonomous driving systems. GPUDrive's scalable architecture ensures it can be used in both academic and industrial research settings to expedite autonomous vehicle development and testing.

Future Directions

Future work may focus on optimizing the reinforcement learning framework to better leverage GPUDrive's high throughput capabilities. Investigating strategies to minimize reset-induced performance bottlenecks and further enhancing the accuracy and generalizability of trained agents are crucial next steps. Additionally, fully integrating detailed maps and enhancing collision detection for non-convex objects could further improve the simulator's realism and applicability to real-world scenarios.

Conclusion

GPUDrive represents a significant advancement in simulation technology for multi-agent reinforcement learning. Its high performance, versatility, and integration with real-world driving data provide a powerful tool for developing and testing autonomous driving systems. GPUDrive sets a new standard for simulation environments, enabling rapid experimentation and potentially accelerating the pace of advancements in autonomous vehicle technology and multi-agent planning research.