Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenes (2305.10430v2)

Abstract: Modern autonomous driving systems are typically divided into three main tasks: perception, prediction, and planning. The planning task involves predicting the trajectory of the ego vehicle based on inputs from both internal intention and the external environment, and manipulating the vehicle accordingly. Most existing works evaluate their performance on the nuScenes dataset using the L2 error and collision rate between the predicted trajectories and the ground truth. In this paper, we reevaluate these existing evaluation metrics and explore whether they accurately measure the superiority of different methods. Specifically, we design an MLP-based method that takes raw sensor data (e.g., past trajectory, velocity, etc.) as input and directly outputs the future trajectory of the ego vehicle, without using any perception or prediction information such as camera images or LiDAR. Our simple method achieves similar end-to-end planning performance on the nuScenes dataset with other perception-based methods, reducing the average L2 error by about 20%. Meanwhile, the perception-based methods have an advantage in terms of collision rate. We further conduct in-depth analysis and provide new insights into the factors that are critical for the success of the planning task on nuScenes dataset. Our observation also indicates that we need to rethink the current open-loop evaluation scheme of end-to-end autonomous driving in nuScenes. Codes are available at https://github.com/E2E-AD/AD-MLP.

Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenes

The paper "Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenes" embarks on an analytical investigation concerning the evaluation metrics widely utilized in the assessment of autonomous driving systems. It scrutinizes the efficacy of these metrics in measuring the performance of models on the nuScenes dataset, which is noted for its extensive use in the field of autonomous driving research. The authors propose a method that challenges the prevailing perception-based paradigms by suggesting that accurate trajectory predictions can be achieved through a simple model that focuses on the ego vehicle's physical state rather than its surrounding environment.

Key Contributions and Methodology

The central thesis presented in this paper revolves around re-evaluating the standard metrics applied to determine the superiority of autonomous driving models. The traditional approach inherently emphasizes a multi-stage pipeline integrating perception, prediction, and planning. However, this paper presents a minimalist model that relies solely on the past trajectory, velocity, and acceleration parameters of the ego vehicle, utilizing a multi-layer perceptron (MLP) for predicting future trajectories.

This approach is computationally efficient, operating independently of sensory inputs like camera images and LiDAR point clouds, which are typically used in perception-driven methodologies. It's noteworthy that the presented MLP-based method achieves a reduction in the average L2 error by approximately 20% compared to perception-based counterparts, though it is surpassed by these methods in terms of minimizing collision rates.

Analytical Insights

This paper dedicates significant effort to analyzing the distribution of trajectory points and angles present in the nuScenes dataset. It highlights that a substantial proportion of trajectory movements occur in straight paths or small angular deviations. Such findings suggest that the dataset's inherent properties could facilitate the achieved trajectory prediction accuracy, even in the absence of rich environmental perception data.

Furthermore, the evaluation of ground truth indicates that utilizing occupancy maps to infer collision rates might be inherently flawed. This methodology's reliance on grid-based representations produces inaccuracies, emphasizing a reevaluation in determining collision metrics, especially when the grid size contributes to misrepresented collisions of non-collision scenarios.

Implications and Future Trajectories

By positing that the current metrics could inadequately reflect the potency of perception-rich systems due to inherent data distribution biases, this paper proposes a reassessment of evaluation strategies for end-to-end autonomous driving models. While the minimalist approach proposed is primarily a proof-of-concept, highlighting potential metric-related shortcomings, it underlines the necessity of a more nuanced, discriminative testing framework capable of showcasing the strengths of perception-centered models.

The implications of its findings suggest that ongoing and future research in autonomous driving methodology should incorporate refined evaluation strategies that articulate the complexities and dynamic nature of real-world interactions. Future models could focus on integrating learned insights for perception, prediction, and planning within unified frameworks, maximizing the advantages of each segment while mitigating information loss.

Conclusion

This paper provides a compelling critique of standard evaluation practices, demonstrating that a more straightforward, perceptually devoid model can achieve results comparable to complex perception-driven systems. However, it clearly acknowledges the impracticality of such an approach in real-world driving conditions. The insights discussed encourage a recalibration of assessment techniques and indicate future research directions that could lead to more reliable, real-world applicable autonomous driving systems.