NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles (2106.11810v4)

Abstract: In this work, we propose the world's first closed-loop ML-based planning benchmark for autonomous driving. While there is a growing body of ML-based motion planners, the lack of established datasets and metrics has limited the progress in this area. Existing benchmarks for autonomous vehicle motion prediction have focused on short-term motion forecasting, rather than long-term planning. This has led previous works to use open-loop evaluation with L2-based metrics, which are not suitable for fairly evaluating long-term planning. Our benchmark overcomes these limitations by introducing a large-scale driving dataset, lightweight closed-loop simulator, and motion-planning-specific metrics. We provide a high-quality dataset with 1500h of human driving data from 4 cities across the US and Asia with widely varying traffic patterns (Boston, Pittsburgh, Las Vegas and Singapore). We will provide a closed-loop simulation framework with reactive agents and provide a large set of both general and scenario-specific planning metrics. We plan to release the dataset at NeurIPS 2021 and organize benchmark challenges starting in early 2022.

• The paper introduces nuPlan as the first closed-loop ML-based planning benchmark for autonomous vehicles using a 1500-hour dataset from four global cities.
• It details a lightweight simulator and planning-specific metrics that evaluate traffic rule adherence, human-like behavior, vehicle dynamics, and complex driving scenarios.
• The benchmark supports comprehensive evaluation through open-loop and reactive closed-loop tasks, fostering improved integration of planning and perception in AVs.

nuPlan: A Closed-Loop ML-Based Planning Benchmark for Autonomous Vehicles

The paper, "nuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles," introduces a novel benchmark addressing the limitations of existing datasets for autonomous vehicle (AV) planning. The authors highlight the inadequacies of traditional motion prediction datasets which have largely focused on short-term motion forecasting through open-loop evaluation methods, instead of offering robust frameworks for long-term AV planning.

Contributions to Autonomous Vehicle Planning

One of the most significant contributions of this paper is the introduction of the first closed-loop ML-based planning benchmark specifically tailored for autonomous driving. The authors provide a large-scale dataset encompassing 1500 hours of driving data from four cities across different continents: Boston, Pittsburgh, Las Vegas, and Singapore. This dataset is distinct as it captures a wide range of traffic scenarios and cultural driving behaviors, thus offering diverse and rich data for ML training and validation.

Benchmark Framework: The benchmark includes a lightweight closed-loop simulator capable of reactive agent interaction, unique planning-specific metrics, and an evaluation protocol that differentiates between open-loop and closed-loop performance. The emphasis is on planning metrics that consider traffic rule adherence, human-like driving behaviors, vehicle dynamics, and scenario-specific challenges, offering a more comprehensive suite of evaluation criteria than existing systems.

Methodology and Evaluation

The authors delineate three primary challenge tasks within their benchmark: open-loop task for direct human trajectory mimicking, and two closed-loop tasks (non-reactive and reactive) that simulate interactive agent environments. The latter two tasks employ a simulated ego vehicle control based on the planner outputs, allowing for the assessment of maneuver feasibility and safety within dynamic scenarios.

• Data Autolabeling and Semantic Mapping: The dataset is extensively autolabeled using state-of-the-art perception technologies like PointPillars and CenterPoint, enhancing its utility for planning purposes. High-quality sensor data accompanies the autolabeled tracks, albeit with controlled access due to the dataset’s size exceeding 200 TB.
• Comprehensive Scenario Annotations: NuPlan provides detailed scenario annotations, allowing planners to be tested against complex driving situations such as lane changes, pedestrian interactions, and intricate urban driving maneuvers typical to each city. This diversity aids in benchmarking varied ML approaches under conditions analogous to real-world complexities.

Implications and Future Directions

The introduction of nuPlan promises to advance ML-based planning in AVs by supplying a rich dataset and robust evaluation metrics which can serve as standard benchmarks for the community. With this structured environment, researchers can refine ML algorithms in closed-loop settings—an aspect critical for understanding the interplay between planning and perception, thereby fostering innovations in goal-based and reactive agent planning.

In future iterations, nuPlan could evolve with community input to broaden its scope and refine its metrics, ensuring relevance as AV technologies progress. The potential for integration with advancements in sim-to-real transfer methods suggests further explorations in bridging the gap between simulated environments and physical-world deployments. Additionally, continued enhancements in sensor data realism and computational simulation capabilities are vital for nurturing ML models that can confidently navigate the complexities of real-world driving.

In conclusion, nuPlan represents a significant step forward in standardizing the evaluation of AV planning systems, encouraging harmonization of research efforts towards scalable, adaptive, and context-aware autonomous driving solutions.