Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Open-sourced Data Ecosystem in Autonomous Driving: the Present and Future (2312.03408v4)

Published 6 Dec 2023 in cs.CV

Abstract: With the continuous maturation and application of autonomous driving technology, a systematic examination of open-source autonomous driving datasets becomes instrumental in fostering the robust evolution of the industry ecosystem. Current autonomous driving datasets can broadly be categorized into two generations. The first-generation autonomous driving datasets are characterized by relatively simpler sensor modalities, smaller data scale, and is limited to perception-level tasks. KITTI, introduced in 2012, serves as a prominent representative of this initial wave. In contrast, the second-generation datasets exhibit heightened complexity in sensor modalities, greater data scale and diversity, and an expansion of tasks from perception to encompass prediction and control. Leading examples of the second generation include nuScenes and Waymo, introduced around 2019. This comprehensive review, conducted in collaboration with esteemed colleagues from both academia and industry, systematically assesses over seventy open-source autonomous driving datasets from domestic and international sources. It offers insights into various aspects, such as the principles underlying the creation of high-quality datasets, the pivotal role of data engine systems, and the utilization of generative foundation models to facilitate scalable data generation. Furthermore, this review undertakes an exhaustive analysis and discourse regarding the characteristics and data scales that future third-generation autonomous driving datasets should possess. It also delves into the scientific and technical challenges that warrant resolution. These endeavors are pivotal in advancing autonomous innovation and fostering technological enhancement in critical domains. For further details, please refer to https://github.com/OpenDriveLab/DriveAGI.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (168)
  1. Alibaba. Opentrek. https://www.aliyun.com/product/iovcc, 2022.
  2. Mohamed Aly. Real time detection of lane markers in urban streets. In 2008 IEEE intelligent vehicles symposium, pages 7–12, 2008.
  3. Amazon. Amazon sagemaker data labeling. https://aws.amazon.com/sagemaker/data-labeling.
  4. Amazon. Autonomous driving data framework. https://docs.aws.amazon.com/prescriptive-guidance/latest/addf-security-and-operations/welcome.html, 2022.
  5. Learning Spatial Regularization Correlation Filters with the Hilbert-Schmidt Independence Criterion in RKHS for UAV Tracking. IEEE Transactions on Instrumentation and Measurement, 72:1–12, 2023. doi: 10.1109/TIM.2023.3265106.
  6. The blackbird UAV dataset. The International Journal of Robotics Research, 39(10-11):1346–1364, 2020.
  7. Baidu. Baidu cloud. https://cloud.baidu.com.
  8. SemanticKITTI: A dataset for semantic scene understanding of lidar sequences. In 2019 IEEE/CVF International Conference on Computer Vision, pages 9296–9306, 2019. doi: 10.1109/ICCV.2019.00939.
  9. Unsupervised Labeled Lane Markers Using Maps. In Proceedings of the IEEE/CVF international conference on computer vision workshops, pages 0–0, 2019.
  10. An application of stereo matching algorithm based on transfer learning on robots in multiple scenarios. Scientific Reports, 13(1):12739, 2023.
  11. DDD17: End-to-end davis driving dataset. arXiv preprint arXiv:1711.01458, 2017.
  12. The MaSTr1325 dataset for training deep USV obstacle detection models. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 3431–3438, 2019.
  13. MODS—a USV-oriented object detection and obstacle segmentation benchmark. IEEE Transactions on Intelligent Transportation Systems, 23(8):13403–13418, 2021.
  14. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023.
  15. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems, volume 33, pages 1877–1901, 2020.
  16. NuScenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621–11631, 2020.
  17. NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles. arXiv preprint arXiv:2106.11810, 2022.
  18. CARLA. CARLA Autonomous Driving Challenge. https://leaderboard.carla.org, 2023.
  19. METEOR: A dense, heterogeneous, and unstructured traffic dataset with rare behaviors. In 2023 IEEE International Conference on Robotics and Automation, pages 9169–9175, 2023.
  20. Argoverse: 3D Tracking and Forecasting with Rich Maps. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8748–8757, 2019.
  21. Persformer: 3D Lane Detection via Perspective Transformer and the OpenLane Benchmark. In European Conference on Computer Vision, pages 550–567, 2022.
  22. PaLI-X: On Scaling up a Multilingual Vision and Language Model. arXiv preprint arXiv:2305.18565, 2023.
  23. Are we ready for unmanned surface vehicles in inland waterways? the USVInland multisensor dataset and benchmark. IEEE Robotics and Automation Letters, 6(2):3964–3970, 2021.
  24. NEAT: Neural attention fields for end-to-end autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15793–15803, 2021.
  25. A review of driving style recognition methods from short-term and long-term perspectives. IEEE Transactions on Intelligent Vehicles, pages 1–15, 2023. doi: 10.1109/TIV.2023.3279425.
  26. HDVIO: Improving Localization and Disturbance Estimation with Hybrid Dynamics VIO. arXiv preprint arXiv:2306.11429, 2023.
  27. The Cityscapes Dataset for Semantic Urban Scene Understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213–3223, 2016.
  28. Robonet: Large-scale multi-robot learning. arXiv preprint arXiv:1910.11215, 2019.
  29. Datatang. Datatang. https://www.datatang.com, 2011.
  30. Parting with Misconceptions about Learning-based Vehicle Motion Planning. arXiv preprint arXiv:2306.07962, 2023.
  31. Real-time path planning to prevent traffic jam through an intelligent transportation system. In 2016 IEEE symposium on computers and communication, pages 726–731, 2016.
  32. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning, pages 7480–7512, 2023.
  33. Talk2Car: Taking control of your self-driving car. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2088–2098, 2019.
  34. Divvy Bike. Chicago Divvy bicycle sharing data. https://www.divvybikes.com/system-data, 2015.
  35. Brian Donovan and DB Work. New York City Taxi Data (2010-2013). http://dx.doi.org/10.13012/J8PN93H8, 2014.
  36. CARLA: An open urban driving simulator. In Conference on robot learning, pages 1–16, 2017.
  37. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  38. PaLM-E: An Embodied Multimodal Language Model. arXiv preprint arXiv:2303.03378, 2023a.
  39. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023b.
  40. DriveLM Contributors. DriveLM: Drive on Language. https://github.com/OpenDriveLab/DriveLM, 2023.
  41. European Commission. White Paper on Artificial Intelligence: a European approach to excellence and trust, 2020. URL https://commission.europa.eu/publications/white-paper-artificial-intelligence-european-approach-excellence-and-trust_en.
  42. LiDAR-CS Dataset: Lidar point cloud dataset with cross-sensors for 3d object detection. arXiv preprint arXiv:2301.12515, 2023.
  43. MacFormer: Map-Agent Coupled Transformer for Real-Time and Robust Trajectory Prediction. IEEE Robotics and Automation Letters, 8(10):6795–6802, 2023.
  44. Visual exploration of big spatio-temporal urban data: A study of new york city taxi trips. IEEE transactions on visualization and computer graphics, 19(12):2149–2158, 2013.
  45. Transportation Networks for Research Core Team. Transportation Networks for Research. https://github.com/bstabler/TransportationNetworks, 2023.
  46. MUAD: Multiple uncertainties for autonomous driving, a benchmark for multiple uncertainty types and tasks. arXiv preprint arXiv:2203.01437, 2022.
  47. A New Performance Measure and Evaluation Benchmark for Road Detection Algorithms. In 16th International IEEE Conference on Intelligent Transportation Systems, pages 1693–1700, 2013.
  48. Incremental path planning: Reservation system in V2X environment. Physica A: Statistical Mechanics and its Applications, 623:128914, 2023.
  49. Vision meets robotics: The KITTI dataset. The International Journal of Robotics Research, 32(11):1231–1237, 2013.
  50. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition, pages 3354–3361, 2012.
  51. A2D2: Audi autonomous driving dataset. arXiv preprint arXiv:2004.06320, 2020.
  52. Scaling and benchmarking self-supervised visual representation learning. In Proceedings of the ieee/cvf International Conference on computer vision, pages 6391–6400, 2019.
  53. GP3: Gaussian process path planning for reliable shortest path in transportation networks. IEEE Transactions on Intelligent Transportation Systems, 23(8):11575–11590, 2021.
  54. Gen-lanenet: A generalized and scalable approach for 3d lane detection. In European Conference on Computer Vision, pages 666–681, 2020.
  55. Openstreetmap: User-generated street maps. IEEE Pervasive computing, 7(4):12–18, 2008.
  56. Towards Smaller and Stronger: An Edge-Aware Lightweight Segmentation Approach for Unmanned Surface Vehicles in Water Scenarios. Sensors, 2023.
  57. The Reasonable Crowd: Towards evidence-based and interpretable models of driving behavior. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 6708–6715, 2021.
  58. Autonomous robot navigation based on openstreetmap geodata. In 13th International IEEE Conference on Intelligent Transportation Systems, pages 1645–1650, 2010.
  59. One thousand and one hours: Self-driving motion prediction dataset. In Conference on Robot Learning, pages 409–418, 2021.
  60. Where2comm: Communication-efficient collaborative perception via spatial confidence maps. In Advances in neural information processing systems, volume 35, pages 4874–4886, 2022.
  61. The ApolloScape dataset for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition workshops, pages 954–960, 2018.
  62. The apolloscape open dataset for autonomous driving and its application. IEEE transactions on pattern analysis and machine intelligence, 42(10):2702–2719, 2019.
  63. Multi-agent trajectory prediction by combining egocentric and allocentric views. In Conference on Robot Learning, pages 1434–1443, 2022.
  64. Towards capturing the temporal dynamics for trajectory prediction: a coarse-to-fine approach. In Conference on Robot Learning, pages 910–920, 2023a.
  65. Hdgt: Heterogeneous driving graph transformer for multi-agent trajectory prediction via scene encoding. IEEE transactions on pattern analysis and machine intelligence, 45(11):13860–13875, 2023b.
  66. Jittor. Jittor Artificial Intelligence Challenge. https://cg.cs.tsinghua.edu.cn/jittor/news/2021-1-22-13-14-comp, 2021.
  67. DAWN: Vehicle detection in adverse weather nature dataset. arXiv preprint arXiv:2008.05402, 2020.
  68. Real-time traffic state estimation with connected vehicles. IEEE Transactions on Intelligent Transportation Systems, 18(7):1687–1699, 2017.
  69. Textual explanations for self-driving vehicles. In European Conference on Computer Vision, pages 563–578, 2018.
  70. Chances of interpretable transfer learning for human activity recognition in warehousing. In Computational Logistics: 12th International Conference, pages 163–177, 2021.
  71. Robotic grasp detection using deep convolutional neural networks. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 769–776, 2017.
  72. Vpgnet: Vanishing point guided network for lane and road marking detection and recognition. In Proceedings of the IEEE international conference on computer vision, pages 1947–1955, 2017.
  73. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023.
  74. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. IEEE transactions on pattern analysis and machine intelligence, 45(3):3461–3475, 2022a.
  75. AADS: Augmented autonomous driving simulation using data-driven algorithms. Science robotics, 4(28):e863–e863, 2019.
  76. 2DSegFormer: 2-D Transformer Model for Semantic Segmentation on Aerial Images. IEEE Transactions on Geoscience and Remote Sensing, 60:1–13, 2022b.
  77. V2X-Sim: Multi-agent collaborative perception dataset and benchmark for autonomous driving. IEEE Robotics and Automation Letters, 7(4):10914–10921, 2022c.
  78. KITTI-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d. IEEE transactions on pattern analysis and machine intelligence, 2022.
  79. Brno Urban Dataset – the new data for self-driving agents and mapping tasks. In 2020 IEEE International Conference on Robotics and Automation, pages 3284–3290, 2020.
  80. How To Guide Your Learner: Imitation Learning with Active Adaptive Expert Involvement. arXiv preprint arXiv:2303.02073, 2023.
  81. Real-time routing with OpenStreetMap data. In Proceedings of the 19th ACM SIGSPATIAL international conference on advances in geographic information systems, pages 513–516, 2011.
  82. UAVid: A semantic segmentation dataset for UAV imagery. ISPRS journal of photogrammetry and remote sensing, 165:108–119, 2020.
  83. TITAN: Future forecast using action priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11186–11196, 2020.
  84. Roboturk: A crowdsourcing platform for robotic skill learning through imitation. In Conference on Robot Learning, pages 879–893, 2018.
  85. Object Scene Flow for Autonomous Vehicles. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3061–3070, 2015.
  86. Microsoft. Visual object tagging tool: An electron app for building end to end object detection models from images and videos. https://github.com/microsoft/VoTT, 2017.
  87. TAS-NIR: A VIS+NIR dataset for fine-grained semantic segmentation in unstructured outdoor environments. arXiv preprint arXiv:2212.09368, 2022.
  88. A benchmark and simulator for UAV tracking. In European Conference on Computer Vision, pages 445–461, 2016.
  89. What’s There in the Dark. In 2019 IEEE International Conference on Image Processing, pages 2996–3000, 2019.
  90. National Science and Technology Council of USA. NATIONAL ARTIFICIAL INTELLIGENCE RESEARCH AND DEVELOPMENT STRATEGIC PLAN 2023 UPDATE, 2023. URL https://www.nitrd.gov/national-artificial-intelligence-research-and-development-strategic-plan-2023-update.
  91. NEO Research Group. NEO benchmark datasets. https://neo.lcc.uma.es/vrp/vehicle-routingproblem, 2013.
  92. The Mapillary Vistas Dataset for Semantic Understanding of Street Scenes. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4990–4999, 2017.
  93. Solving Capacitated Vehicle Routing Problem Using Chicken Swarm Optimization with Genetic Algorithm. International Journal of Intelligent Engineering Systems, 13(5), 2020.
  94. LARa: Creating a dataset for human activity recognition in logistics using semantic attributes. Sensors, 2020.
  95. Nividia. How maglev speeds autonomous vehicles to superhuman levels of safety. https://blogs.nvidia.com/blog/2018/09/13/how-maglev-speeds-autonomous-vehicles-to-superhuman-levels-of-safety, 2018.
  96. Caltrans: California Department of Transportation. Caltrans Performance Measurement System (PeMS). https://pems.dot.ca.gov, 2017.
  97. OpenAI. GPT-4 Technical Report. 2303.08774, 2023.
  98. OpenDriveLab. Autonomous Driving Challenge. https://opendrivelab.com/AD23Challenge.html, 2023a.
  99. OpenDriveLab. DriveAGI: Embracing foundation models into autonomous agent and system. https://github.com/OpenDriveLab/DriveAGI, 2023b.
  100. CARRADA dataset: Camera and automotive radar with range-angle-doppler annotations. In 2020 25th International Conference on Pattern Recognition, pages 5068–5075, 2021.
  101. Iso-dream: Isolating and leveraging noncontrollable visual dynamics in world models. In Advances in neural information processing systems, volume 35, pages 23178–23191, 2022.
  102. Spatial As Deep: Spatial CNN for Traffic Scene Understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
  103. Semanticposs: A point cloud dataset with large quantity of dynamic instances. In 2020 IEEE Intelligent Vehicles Symposium, pages 687–693, 2020.
  104. Leveraging Future Relationship Reasoning for Vehicle Trajectory Prediction. In The Eleventh International Conference on Learning Representations, 2022.
  105. The H3D Dataset for Full-Surround 3D Multi-Object Detection and Tracking in Crowded Urban Scenes. In 2019 International Conference on Robotics and Automation, pages 9552–9557, 2019.
  106. Convergence behavior for traffic assignment characterization metrics. Transportmetrica A: Transport Science, 17(4):1244–1271, 2021.
  107. Pazhou Lab. International Algorithm Case Competition. https://iacc.pazhoulab-huangpu.com, 2023.
  108. A*3D Dataset: Towards Autonomous Driving in Challenging Environments. In 2020 IEEE International conference on Robotics and Automation, pages 2267–2273, 2020.
  109. DriveIRL: Drive in real life with inverse reinforcement learning. In 2023 IEEE International Conference on Robotics and Automation, pages 1544–1550, 2023.
  110. Modeling perception errors towards robust decision making in autonomous vehicles. arXiv preprint arXiv:2001.11695, 2020.
  111. Lost and found: detecting small road hazards for self-driving vehicles. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 1099–1106, 2016.
  112. Toward Driving Scene Understanding: A Dataset for Learning Driver Behavior and Causal Reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7699–7707, 2018.
  113. PIE: A large-scale dataset and models for pedestrian intention estimation and trajectory prediction. In Proceedings of the IEEE international conference on computer vision, pages 6262–6271, 2019.
  114. Interruption-Aware Cooperative Perception for V2X Communication-Aided Autonomous Driving. arXiv preprint arXiv:2304.11821, 2023.
  115. FJMP: Factorized Joint Multi-Agent Motion Prediction over Learned Directed Acyclic Interaction Graphs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13745–13755, 2023.
  116. Scale AI. Scale data engine. https://scale.com/data-engine.
  117. Segments.ai. Multi-sensor Labeling Platform. https://segments.ai/.
  118. Reasonnet: End-to-end driving with temporal and global reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13723–13733, 2023.
  119. MTR++: Multi-agent motion prediction with symmetric scene modeling and guided intention querying. arXiv preprint arXiv:2306.17770, 2023.
  120. Scene as Occupancy. In Proceedings of the IEEE international conference on computer vision, pages 8406–8415, 2023.
  121. ROAD: The road event awareness dataset for autonomous driving. IEEE transactions on pattern analysis and machine intelligence, 45(1):1036–1054, 2022.
  122. The effectiveness of mae pre-pretraining for billion-scale pretraining. arXiv preprint arXiv:2303.13496, 2023.
  123. Towards expert-level medical question answering with large language models. arXiv preprint arXiv:2305.09617, 2023.
  124. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2446–2454, 2020.
  125. M2I: From factored marginal trajectory prediction to interactive prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6543–6552, 2022.
  126. Toronto-3D: A large-scale mobile lidar dataset for semantic segmentation of urban roadways. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 202–203, 2020.
  127. Tesla. Full self-driving. https://www.tesla.com/support/autopilot.
  128. Tesla. Building foundation models for autonomy. https://www.youtube.com/watch?v=6x-Xb_uT7ts, 2023.
  129. The State Council of the People’s Republic of China. Next Generation Artificial Intelligence Development Plan, 2017. URL https://www.gov.cn/zhengce/content/2017-07/20/content_5211996.htm.
  130. Open-source public transportation mobility simulation engine dtalite-s: A discretized space–time network-based modeling framework for bridging multi-agent simulation and optimization. Urban Rail Transit, 5:1–16, 2019.
  131. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
  132. Adaptive passenger-finding recommendation system for taxi drivers with load balancing problem. Computers & Industrial Engineering, 169:108187, 2022.
  133. TuSimple. https://github.com/TuSimple/tusimple-benchmark, 2017.
  134. IDD: A dataset for exploring problems of autonomous navigation in unconstrained environments. In 2019 IEEE Winter Conference on Applications of Computer Vision, pages 1743–1751, 2019.
  135. Talk2Nav: Long-range vision-and-language navigation with dual attention and spatial memory. International Journal of Computer Vision, 129:246–266, 2021.
  136. Improving Multi-Agent Trajectory Prediction Using Traffic States on Interactive Driving Scenarios. IEEE Robotics and Automation Letters, 8(5):2708–2715, 2023.
  137. OpenLane-V2: A Topology Reasoning Benchmark for Scene Understanding in Autonomous Driving. arXiv preprint arXiv:2304.10440, 2023a.
  138. Transportation origin-destination demand estimation with quasi-sparsity. Transportation science, 57(2):289–312, 2023b.
  139. Efficient Reinforcement Learning for Autonomous Driving with Parameterized Skills and Priors. arXiv preprint arXiv:2305.04412, 2023c.
  140. Autonomous Driving Based on Approximate Safe Action. IEEE Transactions on Intelligent Transportation Systems, pages 1–9, 2023d.
  141. Waymo. Waymo Open Dataset Challenges. https://waymo.com/open/challenges, 2023.
  142. Evaluating contraflow high-occupancy vehicle lane designs for mitigating high-occupancy vehicle lane performance degradation. Transportation research record, 2677(5):707–719, 2023.
  143. UrbanLoco: A full sensor suite dataset for mapping and localization in urban scenes. In 2020 IEEE international conference on robotics and automation, pages 2310–2316, 2020.
  144. 4Seasons: A cross-season dataset for multi-weather slam in autonomous driving. arXiv preprint arXiv:2009.06364, 2020.
  145. Argoverse 2: Next Generation Datasets for Self-driving Perception and Forecasting. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
  146. ParallelNet: Multi-mode trajectory prediction by multi-mode trajectory fusion. arXiv preprint arXiv:2212.10203, 2022.
  147. An Imitation Learning Method with Data Augmentation and Post Processing for Planning in Autonomous Driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2023.
  148. Pandaset: Advanced sensor suite dataset for autonomous driving. In 2021 IEEE International Intelligent Transportation Systems Conference, pages 3095–3101, 2021.
  149. AgBM-DTALite: An integrated modelling system of agent-based travel behaviour and transportation network dynamics. Travel Behaviour and Society, 12:141–150, 2018.
  150. Curvelane-nas: Unifying lane-sensitive architecture search and adaptive point blending. In European Conference on Computer Vision, pages 689–704, 2020.
  151. Opv2v: An open benchmark dataset and fusion pipeline for perception with vehicle-to-vehicle communication. In 2022 International Conference on Robotics and Automation, pages 2583–2589, 2022.
  152. ONCE-3DLanes: Building Monocular 3D Lane Detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17143–17152, 2022.
  153. DrivingStereo: A large-scale dataset for stereo matching in autonomous driving scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 899–908, 2019.
  154. Exploring spatial variation of bike sharing trip production and attraction: A study based on Chicago’s Divvy system. Applied geography, 115:102130, 2020.
  155. ShorelineNet: An efficient deep learning approach for shoreline semantic segmentation for unmanned surface vehicles. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5403–5409, 2021.
  156. Improved PSPNet-based water shoreline detection in complex inland river scenarios. Complex & Intelligent Systems, 9(1):233–245, 2023.
  157. WoodScape: A multi-task, multi-camera fisheye dataset for autonomous driving. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9308–9318, 2019.
  158. Dair-V2X: A large-scale dataset for vehicle-infrastructure cooperative 3D object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21361–21370, 2022.
  159. WildDash - creating hazard-aware benchmarks. In European Conference on Computer Vision, pages 402–416, 2018.
  160. Multi-task federated learning for traffic prediction and its application to route planning. In 2021 IEEE Intelligent Vehicles Symposium, pages 451–457, 2021.
  161. Interaction dataset: An international, adversarial and cooperative motion dataset in interactive driving scenarios with semantic maps. arXiv preprint arXiv:1910.03088, 2019.
  162. 3D Harmonic Loss: Towards Task-consistent and Time-friendly 3D Object Detection on Edge for V2X Orchestration. IEEE transactions on vehicular technology, pages 1–12, 2023.
  163. VIL-100: a new dataset and a baseline model for video instance lane detection. In Proceedings of the IEEE international conference on computer vision, pages 15681–15690, 2021a.
  164. BEVerse: Unified Perception and Prediction in Birds-Eye-View for Vision-Centric Autonomous Driving. arXiv preprint arXiv:2205.09743, 2022.
  165. End-to-end urban driving by imitating a reinforcement learning coach. In Proceedings of the IEEE international conference on computer vision, pages 15222–15232, 2021b.
  166. DTALite: A queue-based mesoscopic traffic simulator for fast model evaluation and calibration. Cogent Engineering, 1(1):961345, 2014.
  167. QCNeXt: A Next-Generation Framework For Joint Multi-Agent Trajectory Prediction. arXiv preprint arXiv:2306.10508, 2023.
  168. An online ride-sharing path-planning strategy for public vehicle systems. IEEE Transactions on Intelligent Transportation Systems, 20(2):616–627, 2018.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (19)
  1. Hongyang Li (99 papers)
  2. Yang Li (1142 papers)
  3. Huijie Wang (8 papers)
  4. Jia Zeng (45 papers)
  5. Pinlong Cai (28 papers)
  6. Huilin Xu (7 papers)
  7. Dahua Lin (336 papers)
  8. Junchi Yan (241 papers)
  9. Feng Xu (180 papers)
  10. Lu Xiong (23 papers)
  11. Jingdong Wang (236 papers)
  12. Futang Zhu (1 paper)
  13. Chunjing Xu (66 papers)
  14. Tiancai Wang (48 papers)
  15. Beipeng Mu (7 papers)
  16. Zhihui Peng (15 papers)
  17. Yu Qiao (563 papers)
  18. Li Chen (590 papers)
  19. Fei Xia (111 papers)
Citations (17)
View on Semantic Scholar

Summary

Open-sourced Data Ecosystem in Autonomous Driving: The Present and Future

The paper under review offers a rigorous examination of the open-sourced data ecosystem essential for the evolution of autonomous driving technology. It meticulously categorizes and assesses over seventy open-source datasets, while addressing both existing challenges and future trajectories in dataset development. With an emphasis on the two generations of datasets, the paper outlines how these datasets support the growing demands of tasks in autonomous driving.

Overview of Autonomous Driving Datasets

The paper categorizes datasets into two main generations. The first generation, epitomized by the KITTI dataset introduced in 2012, focuses on simpler sensor modalities and perception-level tasks. Conversely, the second generation, including nuScenes and Waymo, expands to more complex sensor data and a broader range of tasks beyond perception to cover prediction and control. These datasets exemplify the industry's progression in handling increased data scale and complexity to accommodate sophisticated autonomous driving systems.

Data Engine System and Dataset Annotation

A significant portion of the paper is dedicated to data engine systems, which are crucial to constructing massive, high-quality datasets. The paper highlights prevalent labeling strategies and tools, such as Amazon's Sagemaker Ground Truth and Scale AI's Data Engine, which are compared based on capabilities and cost. Furthermore, it addresses the role of automation in data labeling—contributing to improved efficiency and reduced costs.

Next-generation Datasets

The authors argue for the development of a third-generation dataset to meet the demands of modern autonomous driving technology. They propose criteria, such as comprehensive sensor coverage, high-quality annotations, and support for innovative paradigms like end-to-end frameworks. By integrating advancements from foundation models, these datasets aim to enhance both the scale and quality, aligning with industrial and academic developments.

Implications and Future Directions

The implications of this research are profound, as it provides a foundational understanding required to facilitate the evolution of autonomous driving technology. Practically, the review underscores the necessity of collaborative efforts between academia and industry to advance dataset quality and accessibility. Theoretically, the insights derived from evaluating numerous datasets set a clear trajectory for further research focused on long-tail scenarios and sophisticated simulation techniques.

In summary, the paper provides an essential roadmap for the continued development of open-source datasets in autonomous driving. By highlighting current practices, challenges, and future needs, it serves as a critical resource for researchers and practitioners aiming to further innovate and refine autonomous technologies. Future directions could involve leveraging artificial intelligence-generated content to efficiently simulate rare driving scenarios, ensuring the robustness and reliability of autonomous systems in real-world settings.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - OpenDriveLab/DriveAGI: Embracing Foundation Models into Autonomous Agent and System (732 stars)