DriveGPT: Scaling Autoregressive Behavior Models for Driving (2412.14415v1)

Abstract: We present DriveGPT, a scalable behavior model for autonomous driving. We model driving as a sequential decision making task, and learn a transformer model to predict future agent states as tokens in an autoregressive fashion. We scale up our model parameters and training data by multiple orders of magnitude, enabling us to explore the scaling properties in terms of dataset size, model parameters, and compute. We evaluate DriveGPT across different scales in a planning task, through both quantitative metrics and qualitative examples including closed-loop driving in complex real-world scenarios. In a separate prediction task, DriveGPT outperforms a state-of-the-art baseline and exhibits improved performance by pretraining on a large-scale dataset, further validating the benefits of data scaling.

An Overview of DriveGPT: Scaling Autoregressive Behavior Models for Driving

The paper "DriveGPT: Scaling Autoregressive Behavior Models for Driving" introduces a novel approach in the domain of behavior modeling for autonomous vehicles by leveraging advancements in transformer-based models. This work aims to enhance the capability of autonomous driving systems by applying autoregressive sequence prediction methodologies typically used in LLMs to model complex driving scenarios, thereby exploring the effects of model and data scaling in the context of autonomous driving.

Transformers have set a new standard across various fields of machine learning, particularly in scenarios that require understanding and prediction of sequential data. DriveGPT extends this paradigm to autonomous driving, wherein driving is treated as a sequential decision-making task. The model predicts future states of traffic agents with a significant improvement over previous methodologies by employing an autoregressive fashion for prediction, akin to how LLMs predict sequences of words.

Key Contributions and Findings

• Scaling of Model Parameters and Datasets: One of the core contributions of this paper is the exploration of scaling laws as they pertain to dataset size, model complexity, and computational resources. Through empirical evaluations, it is demonstrated that DriveGPT, comprising over 1.4 billion parameters and trained on more than 100 million samples, exhibits improved predictive performance over smaller, less complex models. The paper highlights that larger, more diverse datasets enable the model to handle infrequent and edge-case scenarios more effectively, which are critical in driving environments.
• Comparison with Existing Models: DriveGPT is put against existing behavior models in the literature, demonstrating superior performance, both quantitatively and qualitatively. When evaluated using various driving-specific metrics such as minimum average displacement error (mADE) and off-road rate, DriveGPT consistently outperforms other models. This is attributed to its ability to leverage both a larger parameter space and a significantly broader dataset.
• Generalizability and Robustness: An additional strength of the DriveGPT model is its capability to generalize across different datasets. It achieves state-of-the-art performance on the Waymo Open Motion Dataset, which underscores the model's robustness and adaptability to unseen driving conditions and diverse environments.

Implications and Future Directions

The implications of this research extend beyond immediate improvements to DriveGPT’s performance in controlled evaluation scenarios. It suggests the potential for autoregressive models, traditionally used in NLP, to be effectively adapted to the spatiotemporal requirements of autonomous driving. This represents a step towards more generalized models that can be applied to different domains requiring sequential decision-making capabilities.

Moreover, the findings on scaling laws provide critical insights into the resource allocations required for future research. The results underline the importance of both data quantity and diversity in capturing complex behaviors, indicating that substantial improvements in autonomous driving systems may be realized through continued investment in collecting and curating larger, high-quality datasets.

While the paper illustrates significant advancements, it also opens avenues for further investigation, particularly in optimizing model architectures for improved efficiency and exploring the integration of multimodal sensory data beyond the structured inputs currently used.

Conclusion

DriveGPT epitomizes a strategic advancement in the domain of autonomous driving, applying the powerful transformer-based autoregressive framework to model driving behaviors with enhanced accuracy and robustness. The paper provides a comprehensive examination of scaling laws, offering valuable empirical evidence to guide future developments in behavior modeling for complex, real-world applications. Its findings reinforce the transformative potential of synergy between data scaling and model complexity, paving the way for more sophisticated and reliable autonomous driving technologies.