NeuroNCAP: Photorealistic Closed-loop Safety Testing for Autonomous Driving (2404.07762v4)

Abstract: We present a versatile NeRF-based simulator for testing autonomous driving (AD) software systems, designed with a focus on sensor-realistic closed-loop evaluation and the creation of safety-critical scenarios. The simulator learns from sequences of real-world driving sensor data and enables reconfigurations and renderings of new, unseen scenarios. In this work, we use our simulator to test the responses of AD models to safety-critical scenarios inspired by the European New Car Assessment Programme (Euro NCAP). Our evaluation reveals that, while state-of-the-art end-to-end planners excel in nominal driving scenarios in an open-loop setting, they exhibit critical flaws when navigating our safety-critical scenarios in a closed-loop setting. This highlights the need for advancements in the safety and real-world usability of end-to-end planners. By publicly releasing our simulator and scenarios as an easy-to-run evaluation suite, we invite the research community to explore, refine, and validate their AD models in controlled, yet highly configurable and challenging sensor-realistic environments. Code and instructions can be found at https://github.com/atonderski/neuro-ncap

• The paper introduces NeuroNCAP, a photorealistic closed-loop simulation framework using neural radiance fields to dynamically assess AD systems in safety-critical scenarios.
• It presents a novel evaluation protocol focused on collision avoidance metrics, revealing significant limitations in planners like UniAD and VAD.
• The publicly released simulator offers a benchmark tool for advancing AD safety through improved reactive decision-making in complex, real-world conditions.

Overview of "NeuroNCAP: Photorealistic Closed-loop Safety Testing for Autonomous Driving"

The paper "NeuroNCAP: Photorealistic Closed-loop Safety Testing for Autonomous Driving" introduces a novel simulation framework designated as NeuroNCAP, aimed at assessing autonomous driving (AD) systems within complex safety-critical scenarios. The contribution is centered on implementing a simulation environment leveraging neural radiance fields (NeRFs) for replicating photorealistic real-world driving conditions. This approach facilitates closed-loop evaluations where the autonomous vehicle's actions reciprocally influence the driving environment.

Core Contributions

1. Closed-loop Simulation Framework: NeuroNCAP employs NeRFs for neural rendering, enabling the construction and dynamic alteration of driving scenarios derived from authentic driving logs. This infrastructure ensures the creation of sensor-realistic inputs that test AD systems against hypothetical, potentially hazardous circumstances that mimic those from Euro NCAP protocols.
2. Focus on Safety-critical Scenarios: The framework emphasizes scenarios characterized by complex decision-making challenges, such as avoiding stationary, frontal, and side collisions. The evaluation rigorously examines the planners' ability to avert accidents through corrective actions.
3. Comprehensive Evaluation Protocol: The authors devised a new evaluation scheme focusing on collision avoidance metrics rather than traditional displacement measures. This allows for a more direct assessment of model performance in situations where human-like driving decisions are crucial.
4. Public Release and Benchmarking: The authors provide their simulator and associated scenarios publicly, fostering further exploration and refinement in the research community. Researchers can leverage this accessible tool to test various models under controlled, yet diverse, driving conditions.

Results and Discussion

The findings presented are critical in demonstrating the limitations and reliability of state-of-the-art end-to-end planners. NeuroNCAP revealed significant shortcomings in contemporary AD models, specifically with prominent planners such as UniAD and VAD, when exposed to closed-loop safety-critical scenarios. While these models exhibit proficiency in controlled, non-interactive settings, their responses in dynamic, unplanned interactions are less than optimal.

The paper's results showed that both UniAD and VAD struggled with novel safety-critical scenarios, particularly in scenarios requiring complex maneuvers such as avoiding oncoming traffic or unexpected stationary obstacles. Incorporating a trajectory post-processing step, a proposed enhancement for generating safer driving paths, slightly improved outcomes. However, the models still indicate substantial incapability in entirely mitigating all forms of potential collisions, suggesting an evident area for improvement in AD planning algorithms.

Implications and Future Directions

The implications of this paper underscore the need for robust testing frameworks such as NeuroNCAP to genuinely measure AD systems' real-world performance. Despite current advancements in end-to-end learning models for autonomous driving, their deployment in diverse, unconstrained environments demands further innovative measures to ensure safety and reliability fully.

Looking ahead, this work posits several avenues of exploration. Integrating more nuanced and reactive actor behaviors and considering factors such as vehicular dynamics, surface conditions, and additional environmental factors may heighten simulation accuracy and breadth. Moreover, fostering further advancements in neural rendering will not only refine sensor fidelity but will also expand the range of testable scenarios.

In summary, "NeuroNCAP: Photorealistic Closed-loop Safety Testing for Autonomous Driving" provides a comprehensive tool and methodology for assessing AD systems in realistic, safety-critical scenarios. It illustrates both the current potential and limitations of end-to-end autonomous driving technologies, offering valuable insights into areas that demand further attention for the successful deployment of autonomous vehicles in varied driving environments.