Papers
Topics
Authors
Recent
Detailed Answer
Quick Answer
Concise responses based on abstracts only
Detailed Answer
Well-researched responses based on abstracts and relevant paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses
Gemini 2.5 Flash
Gemini 2.5 Flash 45 tok/s
Gemini 2.5 Pro 49 tok/s Pro
GPT-5 Medium 11 tok/s Pro
GPT-5 High 19 tok/s Pro
GPT-4o 88 tok/s Pro
Kimi K2 214 tok/s Pro
GPT OSS 120B 460 tok/s Pro
Claude Sonnet 4 38 tok/s Pro
2000 character limit reached

ScenicNL: Generating Probabilistic Scenario Programs from Natural Language (2405.03709v3)

Published 3 May 2024 in cs.SE, cs.AI, cs.LG, and cs.PL

Abstract: For cyber-physical systems (CPS), including robotics and autonomous vehicles, mass deployment has been hindered by fatal errors that occur when operating in rare events. To replicate rare events such as vehicle crashes, many companies have created logging systems and employed crash reconstruction experts to meticulously recreate these valuable events in simulation. However, in these methods, "what if" questions are not easily formulated and answered. We present ScenarioNL, an AI System for creating scenario programs from natural language. Specifically, we generate these programs from police crash reports. Reports normally contain uncertainty about the exact details of the incidents which we represent through a Probabilistic Programming Language (PPL), Scenic. By using Scenic, we can clearly and concisely represent uncertainty and variation over CPS behaviors, properties, and interactions. We demonstrate how commonplace prompting techniques with the best LLMs (LLM) are incapable of reasoning about probabilistic scenario programs and generating code for low-resource languages such as Scenic. Our system is comprised of several LLMs chained together with several kinds of prompting strategies, a compiler, and a simulator. We evaluate our system on publicly available autonomous vehicle crash reports in California from the last five years and share insights into how we generate code that is both semantically meaningful and syntactically correct.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (44)
  1. URL https://api.semanticscholar.org/CorpusID:268232499.
  2. Archive of Formal Proofs. URL https://www.isa-afp.org/.
  3. Big data and deep learning in smart cities: A comprehensive dataset for ai-driven traffic accident detection and computer vision systems, 2024.
  4. Uncertainty-based traffic accident anticipation with spatio-temporal relational learning. In Proceedings of the 28th ACM International Conference on Multimedia, MM ’20. ACM, October 2020. doi: 10.1145/3394171.3413827. URL http://dx.doi.org/10.1145/3394171.3413827.
  5. Prompting is programming: A query language for large language models. Proceedings of the ACM on Programming Languages, 7(PLDI):1946–1969, June 2023. ISSN 2475-1421. doi: 10.1145/3591300. URL http://dx.doi.org/10.1145/3591300.
  6. nuScenes: A multimodal dataset for autonomous driving, May 2020. URL http://arxiv.org/abs/1903.11027. arXiv:1903.11027 [cs, stat].
  7. Anticipating Accidents in Dashcam Videos. In Shang-Hong Lai, Vincent Lepetit, Ko Nishino, and Yoichi Sato (eds.), Computer Vision – ACCV 2016, Lecture Notes in Computer Science, pp.  136–153, Cham, 2017. Springer International Publishing. ISBN 978-3-319-54190-7. doi: 10.1007/978-3-319-54190-7˙9.
  8. Evaluating Large Language Models Trained on Code, July 2021. URL https://arxiv.org/abs/2107.03374v2.
  9. TARGET: Automated Scenario Generation from Traffic Rules for Testing Autonomous Vehicles, October 2023. URL http://arxiv.org/abs/2305.06018. arXiv:2305.06018 [cs].
  10. California DMV. Autonomous vehicle collision reports. https://www.dmv.ca.gov/portal/vehicle-industry-services/autonomous-vehicles/autonomous-vehicle-collision-reports/, 2023. Accessed: 2023-10-24.
  11. Carla: An open urban driving simulator, 2017.
  12. VerifAI: A toolkit for the formal design and analysis of artificial intelligence-based systems. In 31st International Conference on Computer Aided Verification (CAV), July 2019.
  13. Scenic: A Language for Scenario Specification and Scene Generation. In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation, pp.  63–78, June 2019. doi: 10.1145/3314221.3314633. URL http://arxiv.org/abs/1809.09310. arXiv:1809.09310 [cs].
  14. Scenic: A language for scenario specification and data generation. Machine Learning Journal, 2023.
  15. Precise Zero-Shot Dense Retrieval without Relevance Labels, December 2022. URL http://arxiv.org/abs/2212.10496. arXiv:2212.10496 [cs].
  16. Waymax: An accelerated, data-driven simulator for large-scale autonomous driving research, 2023.
  17. Andrew J. Hawkins. Welcome to Simulation City, the virtual world where Waymo tests its autonomous vehicles, July 2021. URL https://www.theverge.com/2021/7/6/22565448/waymo-simulation-city-autonomous-vehicle-testing-virtual.
  18. HuggingFace. sentence-transformers/all-mpnet-base-v2, 2023. URL https://huggingface.co/sentence-transformers/all-mpnet-base-v2.
  19. Proving the correctness of disk paxos. Archive of Formal Proofs, June 2005. ISSN 2150-914x. https://isa-afp.org/entries/DiskPaxos.html, Formal proof development.
  20. Evan Jones. Llama.cpp. https://github.com/ggerganov/llama.cpp/pull/1773, 2023.
  21. Driving to Safety: How Many Miles of Driving Would It Take to Demonstrate Autonomous Vehicle Reliability? URL https://www.autosafety.org/wp-content/uploads/2016/04/RAND-AV-Report.pdf.
  22. Gregory Kamradt. LLMTest_needleinahaystack/README.md at main · gkamradt/LLMTest_needleinahaystack. URL https://github.com/gkamradt/LLMTest_NeedleInAHaystack/blob/main/README.md.
  23. The Shift from Models to Compound AI Systems. URL http://bair.berkeley.edu/blog/2024/02/18/compound-ai-systems/.
  24. Simulation-based reconstruction of traffic incidents from moving vehicle mono-camera. Science & Justice, 62(1):94–109, 2022. ISSN 1355-0306. doi: https://doi.org/10.1016/j.scijus.2021.11.001. URL https://www.sciencedirect.com/science/article/pii/S1355030621001362.
  25. Citywide reconstruction of traffic flow using the vehicle-mounted moving camera in the carla driving simulator. In 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC), pp.  2292–2299, 2022. doi: 10.1109/ITSC55140.2022.9921927.
  26. StarCoder: may the source be with you!, December 2023. URL http://arxiv.org/abs/2305.06161. arXiv:2305.06161 [cs].
  27. Adaptive physics-informed trajectory reconstruction exploiting driver behavior and car dynamics. Scientific Reports, 13(1):1121, January 2023. ISSN 2045-2322. doi: 10.1038/s41598-023-28202-1. URL https://www.nature.com/articles/s41598-023-28202-1. Number: 1 Publisher: Nature Publishing Group.
  28. The Lean 4 Theorem Prover and Programming Language. In André Platzer and Geoff Sutcliffe (eds.), Automated Deduction – CADE 28, volume 12699, pp.  625–635. Springer International Publishing, Cham, 2021. ISBN 978-3-030-79875-8 978-3-030-79876-5. doi: 10.1007/978-3-030-79876-5˙37. URL https://link.springer.com/10.1007/978-3-030-79876-5_37. Series Title: Lecture Notes in Computer Science.
  29. CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis, February 2023. URL http://arxiv.org/abs/2203.13474. arXiv:2203.13474 [cs].
  30. Generating context-aware api calls from natural language description using neural embeddings and machine translation, 2021.
  31. A new methodology for vehicle trajectory reconstruction based on wavelet analysis. Transportation Research Part C: Emerging Technologies, 74:150–167, January 2017. ISSN 0968-090X. doi: 10.1016/j.trc.2016.11.010. URL https://www.sciencedirect.com/science/article/pii/S0968090X16302261.
  32. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024.
  33. Ethical considerations and moral implications of autonomous vehicles and unavoidable collisions. Theoretical Issues in Ergonomics Science, 23:435 – 452, 2021. URL https://api.semanticscholar.org/CorpusID:244591637.
  34. Code Llama: Open Foundation Models for Code, August 2023. URL https://arxiv.org/abs/2308.12950v2.
  35. Waymo simulated driving behavior in reconstructed fatal crashes within an autonomous vehicle operating domain. Accident Analysis & Prevention, 163:106454, 2021. ISSN 0001-4575. doi: https://doi.org/10.1016/j.aap.2021.106454. URL https://www.sciencedirect.com/science/article/pii/S0001457521004851.
  36. Mpnet: Masked and permuted pre-training for language understanding, 2020.
  37. CADSim: Robust and Scalable in-the-wild 3D Reconstruction for Controllable Sensor Simulation. In Proceedings of The 6th Conference on Robot Learning, pp.  630–642. PMLR, March 2023a. URL https://proceedings.mlr.press/v205/wang23b.html. ISSN: 2640-3498.
  38. Adept: A testing platform for simulated autonomous driving. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, ASE ’22, New York, NY, USA, 2023b. Association for Computing Machinery. ISBN 9781450394758. doi: 10.1145/3551349.3559528. URL https://doi.org/10.1145/3551349.3559528.
  39. Chain-of-thought prompting elicits reasoning in large language models, 2023.
  40. EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervision, November 2023a. URL http://arxiv.org/abs/2311.02077. arXiv:2311.02077 [cs].
  41. LeanDojo: Theorem Proving with Retrieval-Augmented Language Models, October 2023b. URL http://arxiv.org/abs/2306.15626. arXiv:2306.15626 [cs, stat].
  42. Tree of thoughts: Deliberate problem solving with large language models, 2023.
  43. BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning, April 2020. URL http://arxiv.org/abs/1805.04687. arXiv:1805.04687 [cs].
  44. Language-guided traffic simulation via scene-level diffusion, 2023.
Citations (5)
View on Semantic Scholar
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Follow-Up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now