Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
80 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Conformal Temporal Logic Planning using Large Language Models (2309.10092v4)

Published 18 Sep 2023 in cs.RO and cs.AI

Abstract: This paper addresses planning problems for mobile robots. We consider missions that require accomplishing multiple high-level sub-tasks, expressed in natural language (NL), in a temporal and logical order. To formally define the mission, we treat these sub-tasks as atomic predicates in a Linear Temporal Logic (LTL) formula. We refer to this task specification framework as LTL-NL. Our goal is to design plans, defined as sequences of robot actions, accomplishing LTL-NL tasks. This action planning problem cannot be solved directly by existing LTL planners because of the NL nature of atomic predicates. To address it, we propose HERACLEs, a hierarchical neuro-symbolic planner that relies on a novel integration of (i) existing symbolic planners generating high-level task plans determining the order at which the NL sub-tasks should be accomplished; (ii) pre-trained LLMs to design sequences of robot actions based on these task plans; and (iii) conformal prediction acting as a formal interface between (i) and (ii) and managing uncertainties due to LLM imperfections. We show, both theoretically and empirically, that HERACLEs can achieve user-defined mission success rates. Finally, we provide comparative experiments demonstrating that HERACLEs outperforms LLM-based planners that require the mission to be defined solely using NL. Additionally, we present examples demonstrating that our approach enhances user-friendliness compared to conventional symbolic approaches.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (60)
  1. H. Kress-Gazit, G. E. Fainekos, and G. J. Pappas, “Where’s waldo? sensor-based temporal logic motion planning,” in IEEE International Conference on Robotics and Automation, 2007, pp. 3116–3121.
  2. E. Plaku, L. E. Kavraki, and M. Y. Vardi, “Motion planning with dynamics by a synergistic combination of layers of planning,” IEEE Transactions on Robotics, vol. 26, no. 3, pp. 469–482, 2010.
  3. S. L. Smith, J. Tumova, C. Belta, and D. Rus, “Optimal path planning for surveillance with temporal-logic constraints,” The International Journal of Robotics Research, vol. 30, no. 14, pp. 1695–1708, 2011.
  4. J. Tumova and D. V. Dimarogonas, “Multi-agent planning under local ltl specifications and event-based synchronization,” Automatica, vol. 70, pp. 239–248, 2016.
  5. Y. Chen, X. C. Ding, A. Stefanescu, and C. Belta, “Formal approach to the deployment of distributed robotic teams,” IEEE Transactions on Robotics, vol. 28, no. 1, pp. 158–171, 2012.
  6. A. Ulusoy, S. L. Smith, and C. Belta, “Optimal multi-robot path planning with ltl constraints: guaranteeing correctness through synchronization,” in Distributed Autonomous Robotic Systems.   Springer, 2014, pp. 337–351.
  7. E. Plaku and S. Karaman, “Motion planning with temporal-logic specifications: Progress and challenges,” AI communications, vol. 29, no. 1, pp. 151–162, 2016.
  8. Y. Shoukry, P. Nuzzo, A. Balkan, I. Saha, A. L. Sangiovanni-Vincentelli, S. A. Seshia, G. J. Pappas, and P. Tabuada, “Linear temporal logic motion planning for teams of underactuated robots using satisfiability modulo convex programming,” in IEEE 56th Conference on Decision and Control, December 2017, pp. 1132–1137.
  9. X. Sun and Y. Shoukry, “Neurosymbolic motion and task planning for linear temporal logic tasks,” arXiv preprint arXiv:2210.05180, 2022.
  10. Y. Kantaros and M. M. Zavlanos, “Stylus*: A temporal logic optimal control synthesis algorithm for large-scale multi-robot systems,” The International Journal of Robotics Research, vol. 39, no. 7, pp. 812–836, 2020.
  11. C. I. Vasile and C. Belta, “Sampling-based temporal logic path planning,” in IEEE/RSJ International Conference on Intelligent Robots and Systems, Tokyo, Japan, November 2013, pp. 4817–4822.
  12. Q. H. Ho, Z. N. Sunberg, and M. Lahijanian, “Planning with simba: Motion planning under uncertainty for temporal goals using simplified belief guides,” in IEEE International Conference on Robotics and Automation (ICRA), 2023, pp. 5723–5729.
  13. D. Kamale, S. Haesaert, and C.-I. Vasile, “Cautious planning with incremental symbolic perception: Designing verified reactive driving maneuvers,” in 2023 IEEE International Conference on Robotics and Automation (ICRA), 2023, pp. 1652–1658.
  14. J. Wang, S. Kalluraya, and Y. Kantaros, “Verified compositions of neural network controllers for temporal logic control objectives,” in 2022 IEEE 61st Conference on Decision and Control (CDC).   IEEE, 2022, pp. 4004–4009.
  15. L. Lindemann, J. Nowak, L. Schönbächler, M. Guo, J. Tumova, and D. V. Dimarogonas, “Coupled multi-robot systems under linear temporal logic and signal temporal logic tasks,” IEEE Transactions on Control Systems Technology, vol. 29, no. 2, pp. 858–865, 2019.
  16. I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg, “Progprompt: Generating situated robot task plans using large language models,” in IEEE International Conference on Robotics and Automation (ICRA), 2023, pp. 11 523–11 530.
  17. A. Z. Ren, A. Dixit, A. Bodrova, S. Singh, S. Tu, N. Brown, P. Xu, L. Takayama, F. Xia, J. Varley, Z. Xu, D. Sadigh, A. Zeng, and A. Majumdar, “Robots that ask for help: Uncertainty alignment for large language model planners,” 2023.
  18. J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng, “Code as policies: Language model programs for embodied control,” in IEEE International Conference on Robotics and Automation (ICRA), 2023, pp. 9493–9500.
  19. D. Shah, B. Osiński, S. Levine et al., “Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action,” in Conference on Robot Learning.   PMLR, 2023, pp. 492–504.
  20. Y. Xie, C. Yu, T. Zhu, J. Bai, Z. Gong, and H. Soh, “Translating natural language to planning goals with large-language models,” arXiv preprint arXiv:2302.05128, 2023.
  21. Y. Ding, X. Zhang, C. Paxton, and S. Zhang, “Task and motion planning with large language models for object rearrangement,” arXiv preprint arXiv:2303.06247, 2023.
  22. B. Liu, Y. Jiang, X. Zhang, Q. Liu, S. Zhang, J. Biswas, and P. Stone, “Llm+ p: Empowering large language models with optimal planning proficiency,” arXiv preprint arXiv:2304.11477, 2023.
  23. J. Wu, R. Antonova, A. Kan, M. Lepert, A. Zeng, S. Song, J. Bohg, S. Rusinkiewicz, and T. Funkhouser, “Tidybot: Personalized robot assistance with large language models,” arXiv preprint arXiv:2305.05658, 2023.
  24. A. Zeng, M. Attarian, B. Ichter, K. Choromanski, A. Wong, S. Welker, F. Tombari, A. Purohit, M. Ryoo, V. Sindhwani et al., “Socratic models: Composing zero-shot multimodal reasoning with language,” arXiv preprint arXiv:2204.00598, 2022.
  25. S. Stepputtis, J. Campbell, M. Phielipp, S. Lee, C. Baral, and H. Ben Amor, “Language-conditioned imitation learning for robot manipulation tasks,” Advances in Neural Information Processing Systems, vol. 33, pp. 13 139–13 150, 2020.
  26. S. Li, X. Puig, C. Paxton, Y. Du, C. Wang, L. Fan, T. Chen, D.-A. Huang, E. Akyürek, A. Anandkumar et al., “Pre-trained language models for interactive decision-making,” Advances in Neural Information Processing Systems, vol. 35, pp. 31 199–31 212, 2022.
  27. D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu et al., “Palm-e: An embodied multimodal language model,” arXiv preprint arXiv:2303.03378, 2023.
  28. W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y. Chebotar et al., “Inner monologue: Embodied reasoning through planning with language models,” arXiv preprint arXiv:2207.05608, 2022.
  29. M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman et al., “Do as i can, not as i say: Grounding language in robotic affordances,” arXiv preprint arXiv:2204.01691, 2022.
  30. J. Ruan, Y. Chen, B. Zhang, Z. Xu, T. Bao, G. Du, S. Shi, H. Mao, X. Zeng, and R. Zhao, “Tptu: Task planning and tool usage of large language model-based ai agents,” arXiv preprint arXiv:2308.03427, 2023.
  31. Y. Chen, J. Arkin, Y. Zhang, N. Roy, and C. Fan, “Autotamp: Autoregressive task and motion planning with llms as translators and checkers,” arXiv preprint arXiv:2306.06531, 2023.
  32. Y. Kantaros, M. Malencia, V. Kumar, and G. Pappas, “Reactive temporal logic planning for multiple robots in unknown environments,” in IEEE International Conference on Robotics and Automation (ICRA), June 2020, pp. 11 479–11 485.
  33. V. Vasilopoulos, Y. Kantaros, G. J. Pappas, and D. E. Koditschek, “Reactive planning for mobile manipulation tasks in unexplored semantic environments,” in IEEE International Conference on Robotics and Automation (ICRA), 2021, pp. 6385–6392.
  34. K. Leahy, A. Jones, and C. I. Vasile, “Fast decomposition of temporal logic specifications for heterogeneous teams,” IEEE Robotics and Automation Letters, 2022.
  35. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017.
  36. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018.
  37. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” The Journal of Machine Learning Research, vol. 21, no. 1, pp. 5485–5551, 2020.
  38. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020.
  39. J. W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young et al., “Scaling language models: Methods, analysis & insights from training gopher,” arXiv preprint arXiv:2112.11446, 2021.
  40. R. Thoppilan, D. De Freitas, J. Hall, N. Shazeer, A. Kulshreshtha, H.-T. Cheng, A. Jin, T. Bos, L. Baker, Y. Du et al., “Lamda: Language models for dialog applications,” arXiv preprint arXiv:2201.08239, 2022.
  41. J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le, “Finetuned language models are zero-shot learners,” arXiv preprint arXiv:2109.01652, 2021.
  42. A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann et al., “Palm: Scaling language modeling with pathways,” arXiv preprint arXiv:2204.02311, 2022.
  43. A. N. Angelopoulos, S. Bates et al., “Conformal prediction: A gentle introduction,” Foundations and Trends® in Machine Learning, vol. 16, no. 4, pp. 494–591, 2023.
  44. B. Kumar, C. Lu, G. Gupta, A. Palepu, D. Bellamy, R. Raskar, and A. Beam, “Conformal prediction with large language models for multi-choice question answering,” arXiv preprint arXiv:2305.18404, 2023.
  45. M. Cleaveland, I. Lee, G. J. Pappas, and L. Lindemann, “Conformal prediction regions for time series using linear complementarity programming,” arXiv preprint arXiv:2304.01075, 2023.
  46. Y. LeCun, “Do large language models need sensory grounding for meaning and understanding?” Workshop on Philosophy of Deep Learning, NYU Center for Mind, Brain and Consciousness and the Columbia Center for Science and Society, 2023. [Online]. Available: https://drive.google.com/file/d/1BU5bV3X5w65DwSMapKcsr0ZvrMRU_Nbi/view
  47. A. Angelopoulos, S. Bates, J. Malik, and M. I. Jordan, “Uncertainty sets for image classifiers using conformal prediction,” arXiv preprint arXiv:2009.14193, 2020.
  48. G. E. Fainekos, H. Kress-Gazit, and G. J. Pappas, “Temporal logic motion planning for mobile robots,” in IEEE International Conference on Robotics and Automation (ICRA), Barcelona, Spain, April 2005, pp. 2020–2025.
  49. K. Leahy, D. Zhou, C.-I. Vasile, K. Oikonomopoulos, M. Schwager, and C. Belta, “Persistent surveillance for unmanned aerial vehicles subject to charging and temporal logic constraints,” Autonomous Robots, vol. 40, no. 8, pp. 1363–1378, 2016.
  50. M. Guo and M. M. Zavlanos, “Distributed data gathering with buffer constraints and intermittent communication,” in IEEE International Conference on Robotics and Automation (ICRA), 2017.
  51. F. Fuggitti, “Ltlf2dfa,” March 2019. [Online]. Available: https://github.com/whitemech/LTLf2DFA
  52. J. White, Q. Fu, S. Hays, M. Sandborn, C. Olea, H. Gilbert, A. Elnashar, J. Spencer-Smith, and D. C. Schmidt, “A prompt pattern catalog to enhance prompt engineering with chatgpt,” arXiv preprint arXiv:2302.11382, 2023.
  53. V. Vovk, “Conditional validity of inductive conformal predictors,” in Asian conference on machine learning.   PMLR, 2012, pp. 475–490.
  54. M. Marian, F. Stîngă, M.-T. Georgescu, H. Roibu, D. Popescu, and F. Manta, “A ros-based control application for a robotic platform using the gazebo 3d simulator,” in 2020 21th International Carpathian Control Conference (ICCC), 2020, pp. 1–5.
  55. R. Amsters and P. Slaets, “Turtlebot 3 as a robotics education platform,” in Robotics in Education, M. Merdan, W. Lepuschitz, G. Koppensteiner, R. Balogh, and D. Obdržálek, Eds.   Cham: Springer International Publishing, 2020, pp. 170–181.
  56. OpenRobotics. (2022) Turtlebot e-manual. [Online]. Available: https://emanual.robotis.com/docs/en/platform/turtlebot3/overview/#overview
  57. A. Dadbin, A. Kalhor, and M. T. Masouleh, “A comparison study on the dynamic control of openmanipulator-x by pd with gravity compensation tuned by oscillation damping based on the phase-trajectory-length concept,” in 2022 8th International Conference on Control, Instrumentation and Automation (ICCIA), 2022, pp. 1–7.
  58. A. Pajaziti, “Slam–map building and navigation via ros,” International Journal of Intelligent Systems and Applications in Engineering, vol. 2, no. 4, pp. 71–75, 2014.
  59. D. Coleman, I. Sucan, S. Chitta, and N. Correll, “Reducing the barrier to entry of complex robotic software: a moveit! case study,” arXiv preprint arXiv:1404.3785, 2014.
  60. A. N. Angelopoulos and S. Bates, “A gentle introduction to conformal prediction and distribution-free uncertainty quantification,” arXiv preprint arXiv:2107.07511, 2021.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Jun Wang (990 papers)
  2. Jiaming Tong (1 paper)
  3. Kaiyuan Tan (8 papers)
  4. Yevgeniy Vorobeychik (123 papers)
  5. Yiannis Kantaros (39 papers)
Citations (16)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets