DriveTester: A Unified Platform for Simulation-Based Autonomous Driving Testing (2412.12656v1)

Abstract: Simulation-based testing plays a critical role in evaluating the safety and reliability of autonomous driving systems (ADSs). However, one of the key challenges in ADS testing is the complexity of preparing and configuring simulation environments, particularly in terms of compatibility and stability between the simulator and the ADS. This complexity often results in researchers dedicating significant effort to customize their own environments, leading to disparities in development platforms and underlying systems. Consequently, reproducing and comparing these methodologies on a unified ADS testing platform becomes difficult. To address these challenges, we introduce DriveTester, a unified simulation-based testing platform built on Apollo, one of the most widely used open-source, industrial-level ADS platforms. DriveTester provides a consistent and reliable environment, integrates a lightweight traffic simulator, and incorporates various state-of-the-art ADS testing techniques. This enables researchers to efficiently develop, test, and compare their methods within a standardized platform, fostering reproducibility and comparison across different ADS testing approaches. The code is available: https://github.com/MingfeiCheng/DriveTester.

• The paper presents DriveTester, a unified platform that integrates multiple simulation engines to standardize ADS testing and enhance reproducibility in research.
• It employs a modular testing engine with algorithms like AVFuzzer and a scenario runner that facilitates real-time simulation of dynamic autonomous driving scenarios.
• Experimental results show low latency and effective violation detection, paving the way for scalable and replicable evaluations in autonomous driving research.

Overview of "DriveTester: A Unified Platform for Simulation-Based Autonomous Driving Testing"

The paper introduces "DriveTester," a platform designed to streamline the simulation-based testing of Autonomous Driving Systems (ADS). This system specifically targets the challenges researchers face when dealing with varying simulation environments and platforms, with a primary interface rooted in Apollo, an industry-standard autonomous driving platform. DriveTester aims to mitigate compatibility issues and enhance reproducibility in ADS testing scenarios.

Simulation-based testing holds immense importance in evaluating ADS for safety and reliability before their adoption in real-world environments. However, due to the difficulty in aligning different simulators and ADS software, efforts to standardize testing protocols often face significant obstacles. Various simulation platforms, including those like LGSVL and CARLA, suffer from compatibility issues, which complicate the reproducibility of results and cross-platform testing of different ADS methodologies.

DriveTester endeavors to alleviate these issues by providing a unified testing environment that is feature-rich yet user-friendly for researchers. Built on Apollo, DriveTester incorporates a lightweight traffic simulator, a set of testing algorithms, and comprehensive configurability through the use of simple YAML configurations. This allows for more efficient method development, testing, and fair comparisons across different ADS testing approaches.

Key Components

1. Testing Engine:
   • This component allows DriveTester to incorporate multiple testing algorithms such as AVFuzzer, BehAVExplor, SAMOTA, and DriveFuzz. By offering adaptors like the Scenario Adaptor and Feedback Adaptor, the platform facilitates the seamless migration and execution of these algorithms. This modular approach paves the way for the integration of future testing methods without substantial overhauls.
2. Scenario Runner:
   • DriveTester utilizes a Scenario Runner to manage and execute the testing processes effectively. It includes a traffic simulator for dynamic scenario generation and an Apollo-Traffic Bridge to maintain real-time communication between the simulator and the Apollo ADS. This ensures that testing covers the full decision-making pipeline, from navigation to collision handling.
3. Configuration Options:
   • Flexibility is enhanced by the platform's use of YAML configuration files. These files help in easily defining system settings, scenario details, and testing configurations. This permits a streamlined workflow for setting up experiments, adjusting parameters, and executing tests, which are crucial for assessing algorithm performance and violation detection.

Experimental Results

The experimental evaluation of DriveTester underscores its practical utility. It demonstrates maintaining low latency while effectively identifying critical violations like collisions. This facilitates large-scale, parallel simulations that maximize resource utilization and expedite the testing process. The capability to handle search-based approaches with minimal configurations highlights the platform's efficiency.

Related Work and Implications

The paper situates DriveTester within the broader landscape of ADS research, acknowledging existing efforts like guided fuzzing and metamorphic testing while emphasizing the lack of standardization across platforms. By addressing compatibility challenges, DriveTester enables researchers from various domains to contribute more effectively to ADS testing without needing deep familiarity with platform-specific configurations.

Future Directions

The creation of DriveTester marks a significant step toward standardized ADS testing, with several future implications:

• Enhanced cross-fertilization of ideas between different research groups owing to a more unified testing environment.
• Potential augmentation with additional simulators and ADS platforms to expand the test coverage.
• Facilitation of advanced research on ADS algorithms by encouraging precise and replicable experimentation on a shared platform.

This endeavor is pivotal as it not only refines the existing processes in ADS testing but also sets a foundational benchmark for subsequent innovations in this domain. As the ADS field progresses, tools like DriveTester will be critical for both academic inquiry and industrial application for safe and reliable autonomous vehicular deployment.