Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
144 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

The Waymo Open Sim Agents Challenge (2305.12032v4)

Published 19 May 2023 in cs.CV, cs.LG, cs.MA, and cs.RO

Abstract: Simulation with realistic, interactive agents represents a key task for autonomous vehicle software development. In this work, we introduce the Waymo Open Sim Agents Challenge (WOSAC). WOSAC is the first public challenge to tackle this task and propose corresponding metrics. The goal of the challenge is to stimulate the design of realistic simulators that can be used to evaluate and train a behavior model for autonomous driving. We outline our evaluation methodology, present results for a number of different baseline simulation agent methods, and analyze several submissions to the 2023 competition which ran from March 16, 2023 to May 23, 2023. The WOSAC evaluation server remains open for submissions and we discuss open problems for the task.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (82)
  1. A survey of statistical model checking. ACM Transactions on Modeling and Computer Simulation (TOMACS), 28(1):1–39, 2018.
  2. ChauffeurNet: Learning to drive by imitating the best and synthesizing the worst. In Robotics: Science and Systems XV, 2019.
  3. SimNet: Learning reactive self-driving simulations from real-world observations. In ICRA, 2021.
  4. BARK: Open behavior benchmarking in multi-agent environments. In IROS, 2020.
  5. Multi-agent imitation learning for driving simulation. In IROS, 2018.
  6. nuScenes: A multimodal dataset for autonomous driving. In CVPR, pages 11621–11631, 2020.
  7. Nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles. In CVPR ADP3 workshop, 2021.
  8. SpAGNN: Spatially-aware graph neural networks for relational behavior forecasting from sensor data. In ICRA, pages 9491–9497, 2020a.
  9. Implicit latent variable model for scene-consistent motion forecasting. In ECCV, 2020b.
  10. Argoverse: 3d tracking and forecasting with rich maps. In CVPR, June 2019.
  11. Learning by cheating. In Conference on Robot Learning, pages 66–75. PMLR, 2020.
  12. Learning to drive from a world on rails. In ICCV, pages 15590–15599, 2021a.
  13. Geosim: Realistic video simulation via geometry-aware composition for self-driving. In CVPR, pages 7230–7240, June 2021b.
  14. Collision avoidance detour: A solution for 2023 waymo open dataset challenge - sim agents. Technical report, Carnegie Mellon University, 2023.
  15. End-to-end driving via conditional imitation learning. In ICRA, pages 4693–4700. IEEE, 2018.
  16. A survey of algorithms for black-box safety validation of cyber-physical systems. Journal of Artificial Intelligence Research, 72:377–428, 2021.
  17. Convolutional social pooling for vehicle trajectory prediction. In CVPR Workshops. Computer Vision Foundation / IEEE Computer Society, 2018.
  18. Diffusion models beat gans on image synthesis. In NeurIPS, 2021.
  19. CARLA: An open urban driving simulator. In CoRL, 2017.
  20. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
  21. Nick Roy Ethan Pronovost, Kai Wang. Generating driving scenes with diffusion. In ICRA Workshop on Scalable Autonomous Driving, June 2023.
  22. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. In ICCV, 2021.
  23. Trafficgen: Learning to generate diverse and realistic traffic scenarios. In ICRA, 2023.
  24. Are we ready for autonomous driving? the kitti vision benchmark suite. In CVPR, 2012.
  25. A fast procedure for computing the distance between complex objects in three-dimensional space. IEEE Journal on Robotics and Automation, 4(2):193–203, 1988.
  26. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017.
  27. Denoising diffusion probabilistic models. In NeurIPS, 2020.
  28. Symphony: Learning realistic and diverse agents for autonomous driving simulation. In ICRA, 2022.
  29. The trajectron: Probabilistic multi-agent trajectory modeling with dynamic spatiotemporal graphs. In ICCV, October 2019.
  30. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
  31. Analyzing and improving the image quality of stylegan. In CVPR, 2020.
  32. Drivergym: Democratising reinforcement learning for autonomous driving. arXiv preprint arXiv:2111.06889, 2021.
  33. Sumo (simulation of urban mobility)-an open-source traffic simulation. In Proceedings of the 4th middle East Symposium on Simulation and Modelling (MESM20002), pages 183–187, 2002.
  34. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. TPAMI, 2022.
  35. Imitation is not enough: Robustifying imitation with reinforcement learning for challenging driving scenarios. In IROS, 2023.
  36. Jfp: Joint future prediction with interactive multi-agent modeling for autonomous driving. In CoRL, 2023.
  37. Lidarsim: Realistic lidar simulation by leveraging the real world. In CVPR, June 2020.
  38. Simulating behaviors of traffic agents for autonomous driving via interactive autoregression. Technical report, Nanyang Technological University,, 2023a.
  39. Map-adaptive multimodal trajectory prediction using hierarchical graph neural networks. IEEE Robotics and Automation Letters, 8(6):3685–3692, 2023b.
  40. Wayformer: Motion forecasting via simple & efficient attention networks. In ICRA, 2023.
  41. f-gan: Training generative neural samplers using variational divergence minimization. Advances in neural information processing systems, 29, 2016.
  42. Emanuel Parzen. On estimation of a probability density function and mode. The annals of mathematical statistics, 33(3):1065–1076, 1962.
  43. Dean A. Pomerleau. Alvinn: An autonomous land vehicle in a neural network. In Advances in Neural Information Processing Systems, 1988.
  44. A simple yet effective method for simulating realistic multi-agent behaviors. Technical report, 2023.
  45. Zero-shot text-to-image generation. In ICML, 2021.
  46. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
  47. Generating useful accident-prone driving scenarios via a learned traffic prior. In CVPR, June 2022.
  48. Precog: Prediction conditioned on goals in visual multi-agent settings. In ICCV, October 2019.
  49. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
  50. Murray Rosenblatt. Remarks on some nonparametric estimates of a density function. The annals of mathematical statistics, pages 832–837, 1956.
  51. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, 2011.
  52. Photorealistic text-to-image diffusion models with deep language understanding. In NeurIPS, 2022.
  53. Improved techniques for training gans. In Proceedings of the 30th International Conference on Neural Information Processing Systems, 2016.
  54. Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data. In ECCV, 2020.
  55. Mtr-a: 1st place solution for 2022 waymo open dataset challenge – motion prediction, 2022a.
  56. Motion transformer with global intention localization and local movement refinement. In Advances in Neural Information Processing Systems, 2022b.
  57. Narrowing the coordinate-frame gap in behavior prediction models: Distillation for efficient and accurate scene-centric motion forecasting. arXiv:2206.03970, 2022.
  58. Scalability in perception for autonomous driving: Waymo open dataset. In CVPR, 2020.
  59. InterSim: Interactive traffic simulation via explicit relation modeling. In IROS, 2022.
  60. Trafficsim: Learning to simulate realistic multi-agent behaviors. In CVPR, 2021.
  61. Scenegen: Learning to generate realistic traffic scenes. In CVPR, June 2021.
  62. Block-nerf: Scalable large scene neural view synthesis. In CVPR, 2022.
  63. Multiple futures prediction. In NeurIPS, 2019.
  64. Analyzing the variety loss in the context of probabilistic trajectory prediction. In ICCV, October 2019.
  65. Congested traffic states in empirical observations and microscopic simulations. Physical review E, 62(2):1805, 2000.
  66. Multipath++: Efficient information fusion and trajectory aggregation for behavior prediction. In ICRA, 2022.
  67. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.
  68. Nocturne: a scalable driving benchmark for bringing multi-agent learning one step closer to the real world. In NeurIPS Datasets and Benchmarks Track, 2022.
  69. Advsim: Generating safety-critical scenarios for self-driving vehicles. In CVPR, 2021.
  70. Joint-multipath++ for simulation agents. Technical report, 2023.
  71. Multiverse transformer: 1st place solution for waymo open sim agents challenge 2023. Technical report, Pegasus, 2023.
  72. Argoverse 2: Next generation datasets for self-driving perception and forecasting. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS Datasets and Benchmarks 2021), 2021.
  73. Flow: Architecture and benchmarking for reinforcement learning in traffic control. arXiv preprint arXiv:1710.05465, 10, 2017.
  74. Bits: Bi-level imitation for traffic simulation. In ICRA, 2023.
  75. Learning naturalistic driving environment with statistical realism. Nature Communications, 14(1):2037, 2023.
  76. Surfelgan: Synthesizing realistic sensor data for autonomous driving. In CVPR, June 2020.
  77. Scaling autoregressive models for content-rich text-to-image generation. Transactions on Machine Learning Research, 2022.
  78. INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps. arXiv:1910.03088 [cs, eess], 2019.
  79. Trafficbots: Towards world models for autonomous driving simulation and motion prediction. In ICRA, 2023.
  80. Language-guided traffic simulation via scene-level diffusion. In CoRL, 2023a.
  81. Guided conditional diffusion for controllable traffic simulation. In ICRA, 2023b.
  82. Smarts: An open-source scalable multi-agent rl training school for autonomous driving. In CoRL, 2020.
Citations (34)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now