Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
80 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Data-Centric Evolution in Autonomous Driving: A Comprehensive Survey of Big Data System, Data Mining, and Closed-Loop Technologies (2401.12888v2)

Published 23 Jan 2024 in cs.RO and cs.CV

Abstract: The aspiration of the next generation's autonomous driving (AD) technology relies on the dedicated integration and interaction among intelligent perception, prediction, planning, and low-level control. There has been a huge bottleneck regarding the upper bound of autonomous driving algorithm performance, a consensus from academia and industry believes that the key to surmount the bottleneck lies in data-centric autonomous driving technology. Recent advancement in AD simulation, closed-loop model training, and AD big data engine have gained some valuable experience. However, there is a lack of systematic knowledge and deep understanding regarding how to build efficient data-centric AD technology for AD algorithm self-evolution and better AD big data accumulation. To fill in the identified research gaps, this article will closely focus on reviewing the state-of-the-art data-driven autonomous driving technologies, with an emphasis on the comprehensive taxonomy of autonomous driving datasets characterized by milestone generations, key features, data acquisition settings, etc. Furthermore, we provide a systematic review of the existing benchmark closed-loop AD big data pipelines from the industrial frontier, including the procedure of closed-loop frameworks, key technologies, and empirical studies. Finally, the future directions, potential applications, limitations and concerns are discussed to arouse efforts from both academia and industry for promoting the further development of autonomous driving. The project repository is available at: https://github.com/LincanLi98/Awesome-Data-Centric-Autonomous-Driving.

PDF HTML Abstract

Data-Centric Evolution in Autonomous Driving: An Analytical Synopsis

The paper presents a meticulous review of the paradigm shift towards data-centric methodologies in the field of autonomous driving (AD). Drawing on the latest advancements, the survey expounds on the integration of big data systems, data mining, and closed-loop technologies, elucidating the evolution towards a more data-driven approach in AD technology. This shift addresses the constraints associated with the algorithms' performance ceiling by pivoting towards comprehensive data-centric technologies.

The paper meticulously classifies and explores the progression of autonomous driving datasets into generational milestones, reflecting technology's rapid advancement. It emphasizes intricate details on the dataset's acquisition, settings, and key characteristics, offering a nuanced view of the landscape. For instance, the transition from the early, more basic datasets like KITTI to newer, more complex datasets such as DriveLM underscores an evolving emphasis on multi-modal data integration and enhanced scenario variety. DriveLM represents a notable example where Generative AI models utilize large-scale language and vision models to improve scenario understanding, addressing challenges like data Long-Tail Distribution and out-of-distribution detection.

Central to the paper is its examination of state-of-the-art closed-loop systems. It delineates the procedural frameworks from data collection to model deployment, typical in pioneering systems like NVIDIA's MagLev and Tesla's robust data platforms. These platforms exemplify closed-loop paradigms, incorporating comprehensive data ingestion, intelligent selection, dynamic labeling, model training, and iterative feedback through real-world deployment loops. This systematic feedback mechanism demonstrates a shift from static to dynamic model training and deployment, providing insightful implications for continued academic and industrial exploration.

Moreover, the paper explores high-fidelity data generation and simulation technologies employing generative AI, spotlighting breakthroughs like CARLA simulator and world models such as GAIA-1 and DriveDreamer. These technologies showcase a novel capability to generate realistic driving scenarios from synthetic sources, addressing the scarcity of rare and challenging driving data scenarios.

The discussion concerning auto-labeling technologies marks another focal point, emphasizing efficiency and scalability in annotating vast data volumes. The transformation from manual annotation to sophisticated auto-labeling systems, including 3D dynamic and 3D static scene labeling methodologies, reflects a crucial advancement minimizing labor-intensive processes.

The paper concludes with an articulation of the prospects and challenges ahead. It anticipates an augmentation in dataset maturity and infrastructure hardware to support expansive AI models while addressing data security and privacy concerns. Sustaining trustworthy autonomous systems through explainability and developing personalized autonomous driving recommendations based on user behavior data are underscored as future research avenues.

In essence, the paper underscores the imperative of an evolved, integrated ecosystem for autonomous driving, marrying technological sophistication with practical deployment considerations. It sets a foundational roadmap, encouraging further academic inquiry and industrial collaboration to transcend existing constraints and holistically enhance autonomous driving technologies. This progression towards a more data-centric framework in autonomous driving holds significant promise for shaping the forefront of intelligent transportation systems.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (65)
  1. Semantickitti: A dataset for semantic scene understanding of lidar sequences. In CVPR, pages 9297–9307, 2019.
  2. Also: Automotive lidar self-supervision by occupancy estimation. In CVPR, pages 13455–13465, 2023.
  3. nuscenes: A multimodal dataset for autonomous driving. In CVPR, pages 11621–11631, 2020.
  4. nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles. arXiv preprint arXiv:2106.11810, 2021.
  5. Argoverse: 3d tracking and forecasting with rich maps. In CVPR, pages 8748–8757, 2019.
  6. Scene recognition with prototype-agnostic scene layout. IEEE Transactions on Image Processing, 29:5877–5888, 2020.
  7. End-to-end autonomous driving: Challenges and frontiers, 2023.
  8. End-to-end autonomous driving: Challenges and frontiers. arXiv, 2306.16927, 2023.
  9. Vma: Divide-and-conquer vectorized map annotation system for large-scale driving scene. arXiv preprint arXiv:2304.09807, 2023.
  10. The cityscapes dataset for semantic urban scene understanding. In CVPR, pages 3213–3223, 2016.
  11. A survey on multimodal large language models for autonomous driving. In WACV, pages 958–979, 2024.
  12. Jean-Emmanuel Deschaud. Kitti-carla: a kitti-like dataset generated by carla simulator. arXiv preprint arXiv:2109.00892, 2021.
  13. Exploratory analysis of injury severity under different levels of driving automation (sae level 2-5) using multi-source data, 2023.
  14. Object based scene representations using fisher scores of local subspace projections. In NeurIPS, volume 29, 2016.
  15. Carla: An open urban driving simulator. In Conference on robot learning, pages 1–16, 2017.
  16. Ashok Elluswamy. Keynote speaking: Foundation models for autonomy (tesla), 2023.
  17. Keynote speaking: Inside nvidia’s ai infrastructure for self-driving cars, 2020.
  18. Mit advanced vehicle technology study: Large-scale naturalistic driving study of driver behavior and interaction with automation. IEEE Access, 7:102021–102038, 2019.
  19. Jiyang Gao. Keynote speaking: How data-driven flywheel enables scalable path to full autonomy (momenta), 2023.
  20. Are we ready for autonomous driving? the kitti vision benchmark suite. In CVPR, pages 3354–3361, 2012.
  21. World models. arXiv preprint arXiv:1803.10122, 2018.
  22. Query-based temporal fusion with explicit motion for 3d object detection. In NeurIPS, 2023.
  23. One thousand and one hours: Self-driving motion prediction dataset. In Conference on Robot Learning, pages 409–418. PMLR, 2021.
  24. Gaia-1: A generative world model for autonomous driving. arXiv preprint arXiv:2309.17080, 2023.
  25. A survey on trajectory-prediction methods for autonomous driving. IEEE Transactions on Intelligent Vehicles, 7(3):652–674, 2022.
  26. Differentiable integrated motion prediction and planning with learnable cost function for autonomous driving. IEEE Transactions on Neural Networks and Learning Systems, pages 1–15, 2023.
  27. Level-5 autonomous driving—are we there yet? a review of research literature. ACM Computing Surveys (CSUR), 55(2):1–38, 2022.
  28. Point cloud forecasting as a proxy for 4d occupancy forecasting. In CVPR, pages 1116–1124, 2023.
  29. End-to-end deep learning-based autonomous driving control for high-speed environment. The Journal of Supercomputing, 78(2):1961–1982, 2022.
  30. Sample efficient deep reinforcement learning with online state abstraction and causal transformer model prediction. IEEE Transactions on Neural Networks and Learning Systems, pages 1–15, 2023.
  31. Time3d: End-to-end joint monocular 3d object detection and tracking for autonomous driving. In CVPR, pages 3885–3894, 2022.
  32. Open-sourced data ecosystem in autonomous driving: the present and future. arXiv preprint arXiv:2312.03408, 2023.
  33. Delving into the devils of bird’s-eye-view perception: A review, evaluation and recipe. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–20, 2023.
  34. Improving generative imagination in object-centric world models. In ICML, pages 6140–6149, 2020.
  35. A novel scene classification model combining resnet based transfer learning and data augmentation with a filter. Neurocomputing, 338:191–206, 2019.
  36. A survey on autonomous driving datasets: Data statistic, annotation, and outlook. arXiv preprint arXiv:2401.01454, 2024.
  37. Retrieval augmented classification for long-tail visual recognition. In CVPR, pages 6959–6969, 2022.
  38. Harris M. The radical scope of tesla’s data hoard: Every tesla is providing reams of sensitive data about its driver’s life. IEEE Spectrum, 59(10):40–45, 2022.
  39. Verification and validation methods for decision-making and planning of automated vehicles: A review. IEEE Transactions on Intelligent Vehicles, 2022.
  40. 3d object detection from images for autonomous driving: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–20, 2023.
  41. Occ-bev: Multi-camera unified pre-training via 3d scene reconstruction. arXiv preprint arXiv:2305.18829, 2023.
  42. Vision-based semantic segmentation in scene understanding for autonomous driving: Recent achievements, challenges, and outlooks. IEEE Transactions on Intelligent Transportation Systems, 23(12):22694–22715, 2022.
  43. Offboard 3d object detection from point cloud sequences. In CVPR, pages 6134–6144, 2021.
  44. Nuscenes-qa: A multi-modal visual question answering benchmark for autonomous driving scenario. arXiv preprint arXiv:2305.14836, 2023.
  45. Automated vehicle control developments in the path program. IEEE Transactions on Vehicular Technology, 40(1):114–130, 1991.
  46. Ride-hailing service aware electric taxi fleet management using reinforcement learning. In ICUFN, pages 427–432, 2022.
  47. Drivelm: Driving with graph visual question answering, 2023.
  48. Scalability in perception for autonomous driving: Waymo open dataset. In CVPR, pages 2446–2454, 2020.
  49. A comprehensive review of yolo architectures in computer vision: From yolov1 to yolov8 and yolo-nas. Machine Learning and Knowledge Extraction, 5(4):1680–1716, 2023.
  50. Scene as occupancy. In CVPR, pages 8406–8415, 2023.
  51. Ltp: Lane-based trajectory prediction for autonomous driving. In CVPR, pages 17134–17142, 2022.
  52. Drivedreamer: Towards real-world-driven world models for autonomous driving. arXiv preprint arXiv:2309.09777, 2023.
  53. On the road with gpt-4v (ision): Early explorations of visual-language model on autonomous driving. arXiv preprint arXiv:2311.05332, 2023.
  54. Argoverse 2: Next generation datasets for self-driving perception and forecasting. In NeurIPS, 2021.
  55. Language prompt for autonomous driving. arXiv preprint arXiv:2309.04379, 2023.
  56. Transformation-equivariant 3d object detection for autonomous driving. In AAAI, pages 2795–2802, 2023.
  57. Mv-map: Offboard hd-map generation with multi-view consistency. In CVPR, pages 8658–8668, 2023.
  58. Auto4d: Learning to label 4d objects from sequential point clouds. arXiv preprint arXiv:2101.06586, 2021.
  59. Ad-pt: Autonomous driving pre-training with large-scale point cloud dataset. arXiv preprint arXiv:2306.00612, 2023.
  60. Rethinking closed-loop training for autonomous driving. In ECCV, pages 264–282, 2022.
  61. Ai-tp: Attention-based interaction-aware trajectory prediction for autonomous driving. IEEE Transactions on Intelligent Vehicles, 8(1):73–83, 2023.
  62. Spatiotemporal adaptive attention 3d multiobject tracking for autonomous driving. Knowledge-Based Systems, 267:110442, 2023.
  63. Occworld: Learning a 3d occupancy world model for autonomous driving, 2023.
  64. Dynamically conservative self-driving planner for long-tail cases. IEEE Transactions on Intelligent Transportation Systems, 24(3):3476–3488, 2023.
  65. Nemo: Neural map growing system for spatiotemporal fusion in bird’s-eye-view and bdd-map benchmark. arXiv preprint arXiv:2306.04540, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (7)
  1. Lincan Li (8 papers)
  2. Wei Shao (95 papers)
  3. Wei Dong (106 papers)
  4. Yijun Tian (29 papers)
  5. Kaixiang Yang (18 papers)
  6. Wenjie Zhang (138 papers)
  7. Qiming Zhang (31 papers)
Citations (6)
View on Semantic Scholar