Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving? (2312.03031v2)

Abstract: End-to-end autonomous driving recently emerged as a promising research direction to target autonomy from a full-stack perspective. Along this line, many of the latest works follow an open-loop evaluation setting on nuScenes to study the planning behavior. In this paper, we delve deeper into the problem by conducting thorough analyses and demystifying more devils in the details. We initially observed that the nuScenes dataset, characterized by relatively simple driving scenarios, leads to an under-utilization of perception information in end-to-end models incorporating ego status, such as the ego vehicle's velocity. These models tend to rely predominantly on the ego vehicle's status for future path planning. Beyond the limitations of the dataset, we also note that current metrics do not comprehensively assess the planning quality, leading to potentially biased conclusions drawn from existing benchmarks. To address this issue, we introduce a new metric to evaluate whether the predicted trajectories adhere to the road. We further propose a simple baseline able to achieve competitive results without relying on perception annotations. Given the current limitations on the benchmark and metrics, we suggest the community reassess relevant prevailing research and be cautious whether the continued pursuit of state-of-the-art would yield convincing and universal conclusions. Code and models are available at \url{https://github.com/NVlabs/BEV-Planner}

Summary of "Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?"

The paper in question explores the concept of end-to-end autonomous driving, particularly focusing on open-loop evaluation methods. This research investigates the extent to which ego status — encompassing velocity, acceleration, and yaw angle of the ego vehicle — can suffice for planning decisions in such systems, when evaluated with the existing benchmark datasets and metrics.

The research highlights several key observations:

1. Dataset Limitations: The nuScenes dataset, prevalent in the field, includes predominantly straightforward driving scenarios (approximately 73.9%). This skewed distribution can lead to models overly relying on ego status as a determinant for decision-making due to the simplicity of scenarios involved.
2. Ego Status Dominance: The paper observes that ego status, which contains information on the vehicle's current state, plays a dominating role in open-loop planning evaluations. Leveraging only ego status with a simple MLP network can yield performance on par with more complex methods that incorporate comprehensive sensory information. This raises critical questions about the validity and depth of current planning benchmarks.
3. Metric Shortcomings: Existing metrics, such as L2 distance and collision rate, fail to thoroughly assess a model's planning quality. The authors introduce intersection rates with road boundaries as a new metric, which shifts the evaluation landscape and highlights differences not captured by prior metrics. This new evaluation exposes cases where methods relying heavily on ego status might fall short.
4. Continued Exploration in Intervention-Free Environments: The suggested models and benchmarks call into question the sole reliance on open-loop evaluation due to the lack of feedback from dynamic environments that change in response to the ego vehicle’s actions.

The implications of these findings suggest significant caution when interpreting state-of-the-art results in open-loop end-to-end autonomous driving research. Researchers should reconsider the reliance on constrained datasets and metrics, recognizing the inherent biases they may impose. As a practical implication, it points to the potential oversight in real-world applicability and safety of systems evaluated solely on such benchmarks.

Further, the paper prompts the research community to scrutinize whether striving for superior open-loop performance without addressing these limitations could lead to misplaced confidence in the predictive power of these systems. It argues for a broader, more comprehensive suite of benchmarks, possibly integrating closed-loop settings, to better capture the dynamic and multifaceted nature of autonomous driving tasks.

Overall, the paper challenges the community to envisage more balanced datasets and evaluation criteria, reconstructing the foundation on which future advances should be built, with more holistic and representative coverage of the varied challenges faced in real-world driving scenarios.