Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
153 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

A Survey of Optimization-based Task and Motion Planning: From Classical To Learning Approaches (2404.02817v5)

Published 3 Apr 2024 in cs.RO and cs.AI

Abstract: Task and Motion Planning (TAMP) integrates high-level task planning and low-level motion planning to equip robots with the autonomy to effectively reason over long-horizon, dynamic tasks. Optimization-based TAMP focuses on hybrid optimization approaches that define goal conditions via objective functions and are capable of handling open-ended goals, robotic dynamics, and physical interaction between the robot and the environment. Therefore, optimization-based TAMP is particularly suited to solve highly complex, contact-rich locomotion and manipulation problems. This survey provides a comprehensive review on optimization-based TAMP, covering (i) planning domain representations, including action description languages and temporal logic, (ii) individual solution strategies for components of TAMP, including AI planning and trajectory optimization (TO), and (iii) the dynamic interplay between logic-based task planning and model-based TO. A particular focus of this survey is to highlight the algorithm structures to efficiently solve TAMP, especially hierarchical and distributed approaches. Additionally, the survey emphasizes the synergy between the classical methods and contemporary learning-based innovations such as LLMs. Furthermore, the future research directions for TAMP is discussed in this survey, highlighting both algorithmic and application-specific challenges.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (242)
  1. A. Adu-Bredu, N. Devraj, and O. C. Jenkins, “Optimal constrained task planning as mixed integer programming,” in 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2022, pp. 12 029–12 036.
  2. C. R. Garrett, R. Chitnis, R. Holladay, B. Kim, T. Silver, L. P. Kaelbling, and T. Lozano-Pérez, “Integrated task and motion planning,” Annual review of control, robotics, and autonomous systems, vol. 4, pp. 265–293, 2021.
  3. J. Envall, R. Poranne, and S. Coros, “Differentiable task assignment and motion planning,” in 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2023, pp. 2049–2056.
  4. N. T. Dantam, Z. K. Kingston, S. Chaudhuri, and L. E. Kavraki, “An incremental constraint-based framework for task and motion planning,” The International Journal of Robotics Research, vol. 37, no. 10, pp. 1134–1151, 2018.
  5. T. Lozano-Pérez and L. P. Kaelbling, “A constraint-based method for solving sequential manipulation planning problems,” in 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.   IEEE, 2014, pp. 3684–3691.
  6. C. R. Garrett, T. Lozano-Pérez, and L. P. Kaelbling, “Pddlstream: Integrating symbolic planners and blackbox samplers via optimistic adaptive planning,” in Proceedings of the International Conference on Automated Planning and Scheduling, vol. 30, 2020, pp. 440–448.
  7. A. Krontiris and K. E. Bekris, “Efficiently solving general rearrangement tasks: A fast extension primitive for an incremental sampling-based planner,” in 2016 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2016, pp. 3924–3931.
  8. M. Toussaint, “Logic-geometric programming: An optimization-based approach to combined task and motion planning.” in IJCAI, 2015, pp. 1930–1936.
  9. R. Takano, H. Oyama, and M. Yamakita, “Continuous optimization-based task and motion planning with signal temporal logic specifications for sequential manipulation,” in 2021 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2021, pp. 8409–8415.
  10. H. Guo, F. Wu, Y. Qin, R. Li, K. Li, and K. Li, “Recent trends in task and motion planning for robotics: A survey,” ACM Computing Surveys, 2023.
  11. A. Orthey, C. Chamzas, and L. E. Kavraki, “Sampling-based motion planning: A comparative review,” Annual Review of Control, Robotics, and Autonomous Systems, vol. 7, 2023.
  12. S. Karaman and E. Frazzoli, “Sampling-based algorithms for optimal motion planning,” The international journal of robotics research, vol. 30, no. 7, pp. 846–894, 2011.
  13. P. S. Schmitt, W. Neubauer, W. Feiten, K. M. Wurm, G. V. Wichert, and W. Burgard, “Optimal, sampling-based manipulation planning,” in 2017 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2017, pp. 3426–3432.
  14. M. A. Toussaint, K. R. Allen, K. A. Smith, and J. B. Tenenbaum, “Differentiable physics and stable modes for tool-use and manipulation planning,” in Robotics: Science and Systems Foundation, 2018.
  15. T. Migimatsu and J. Bohg, “Object-centric task and motion planning in dynamic environments,” IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 844–851, 2020.
  16. T. Stouraitis, I. Chatzinikolaidis, M. Gienger, and S. Vijayakumar, “Online hybrid motion planning for dyadic collaborative manipulation via bilevel optimization,” IEEE Transactions on Robotics, vol. 36, no. 5, pp. 1452–1471, 2020.
  17. B. Aceituno-Cabezas, C. Mastalli, H. Dai, M. Focchi, A. Radulescu, D. G. Caldwell, J. Cappelletto, J. C. Grieco, G. Fernández-López, and C. Semini, “Simultaneous contact, gait, and motion planning for robust multilegged locomotion via mixed-integer convex optimization,” IEEE Robotics and Automation Letters, vol. 3, no. 3, pp. 2531–2538, 2017.
  18. J.-P. Sleiman, F. Farshidian, and M. Hutter, “Versatile multicontact planning and control for legged loco-manipulation,” Science Robotics, vol. 8, no. 81, p. eadg5014, 2023.
  19. Y. Zhao, Y. Li, L. Sentis, U. Topcu, and J. Liu, “Reactive task and motion planning for robust whole-body dynamic locomotion in constrained environments,” The International Journal of Robotics Research, vol. 41, no. 8, pp. 812–847, 2022.
  20. M. Asselmeier, J. Ivanova, Z. Zhou, P. A. Vela, and Y. Zhao, “Hierarchical experience-informed navigation for multi-modal quadrupedal rebar grid traversal,” IEEE International Conference on Robotics and Automation, 2024.
  21. D. Meli, H. Nakawala, and P. Fiorini, “Logic programming for deliberative robotic task planning,” Artificial Intelligence Review, pp. 1–39, 2023.
  22. P. M. Wensing, M. Posa, Y. Hu, A. Escande, N. Mansard, and A. Del Prete, “Optimization-based control for dynamic legged robots,” IEEE Transactions on Robotics, 2023.
  23. M. Posa, C. Cantu, and R. Tedrake, “A direct method for trajectory optimization of rigid bodies through contact,” The International Journal of Robotics Research, vol. 33, no. 1, pp. 69–81, 2014.
  24. Y. Tassa, N. Mansard, and E. Todorov, “Control-limited differential dynamic programming,” in 2014 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2014, pp. 1168–1175.
  25. L. Xu, T. Ren, G. Chalvatzaki, and J. Peters, “Accelerating integrated task and motion planning with neural feasibility checking,” arXiv preprint arXiv:2203.10568, 2022.
  26. Z. Yang, C. Garrett, T. Lozano-Perez, L. Kaelbling, and D. Fox, “Sequence-based plan feasibility prediction for efficient task and motion planning,” in Robotics science and systems, 2023.
  27. D. Driess, J.-S. Ha, and M. Toussaint, “Deep visual reasoning: Learning to predict action sequences for task and motion planning from an initial scene image,” in Robotics: Science and Systems 2020 (RSS 2020).   RSS Foundation, 2020.
  28. K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg, “Text2motion: From natural language instructions to feasible plans,” Autonomous Robots, vol. 47, no. 8, pp. 1345–1365, 2023.
  29. T. Silver, S. Dan, K. Srinivas, J. B. Tenenbaum, L. Kaelbling, and M. Katz, “Generalized planning in pddl domains with pretrained large language models,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 18, 2024, pp. 20 256–20 264.
  30. W. Huang, P. Abbeel, D. Pathak, and I. Mordatch, “Language models as zero-shot planners: Extracting actionable knowledge for embodied agents,” in International Conference on Machine Learning.   PMLR, 2022, pp. 9118–9147.
  31. S. Cheng and D. Xu, “League: Guided skill learning and abstraction for long-horizon manipulation,” IEEE Robotics and Automation Letters, vol. 8, no. 10, pp. 6451–6458, 2023.
  32. M. J. McDonald and D. Hadfield-Menell, “Guided imitation of task and motion planning,” in Conference on Robot Learning.   PMLR, 2022, pp. 630–640.
  33. M. Mansouri, F. Pecora, and P. Schüller, “Combining task and motion planning: Challenges and guidelines,” Frontiers in Robotics and AI, vol. 8, p. 637888, 2021.
  34. L. Antonyshyn, J. Silveira, S. Givigi, and J. Marshall, “Multiple mobile robot task and motion planning: A survey,” ACM Computing Surveys, vol. 55, no. 10, pp. 1–35, 2023.
  35. C. Belta and S. Sadraddini, “Formal methods for control synthesis: An optimization perspective,” Annual Review of Control, Robotics, and Autonomous Systems, vol. 2, pp. 115–140, 2019.
  36. O. Shorinwa, T. Halsted, J. Yu, and M. Schwager, “Distributed optimization methods for multi-robot systems: Part i–a tutorial,” arXiv preprint arXiv:2301.11313, 2023.
  37. T. Silver, R. Chitnis, J. Tenenbaum, L. P. Kaelbling, and T. Lozano-Pérez, “Learning symbolic operators for task and motion planning,” in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2021, pp. 3182–3189.
  38. A. Pnueli, “The temporal logic of programs,” in 18th Annual Symposium on Foundations of Computer Science (sfcs 1977).   ieee, 1977, pp. 46–57.
  39. G. De Giacomo and M. Y. Vardi, “Linear temporal logic and linear dynamic logic on finite traces,” in IJCAI’13 Proceedings of the Twenty-Third international joint conference on Artificial Intelligence.   Association for Computing Machinery, 2013, pp. 854–860.
  40. O. Maler and D. Nickovic, “Monitoring temporal properties of continuous signals,” in International Symposium on Formal Techniques in Real-Time and Fault-Tolerant Systems.   Springer, 2004, pp. 152–166.
  41. R. Koymans, “Specifying real-time properties with metric temporal logic,” Real-time systems, vol. 2, no. 4, pp. 255–299, 1990.
  42. C. Aeronautiques, A. Howe, C. Knoblock, I. D. McDermott, A. Ram, M. Veloso, D. Weld, D. W. SRI, A. Barrett, D. Christianson et al., “Pddl— the planning domain definition language,” Technical Report, Tech. Rep., 1998.
  43. P. Haslum, N. Lipovetzky, D. Magazzeni, and C. Muise, “An introduction to the planning domain definition language,” Synthesis Lectures on Artificial Intelligence and Machine Learning, vol. 13, no. 2, pp. 1–187, 2019.
  44. M. Fox and D. Long, “Pddl2. 1: An extension to pddl for expressing temporal planning domains,” Journal of artificial intelligence research, vol. 20, pp. 61–124, 2003.
  45. D. L. Kovacs, “Bnf definition of pddl 3.1,” Unpublished manuscript from the IPC-2011 website, vol. 15, 2011.
  46. M. Fox and D. Long, “Modelling mixed discrete-continuous domains for planning,” Journal of Artificial Intelligence Research, vol. 27, pp. 235–297, 2006.
  47. H. L. Younes and M. L. Littman, “Ppddl1. 0: An extension to pddl for expressing planning domains with probabilistic effects,” Techn. Rep. CMU-CS-04-162, vol. 2, p. 99, 2004.
  48. D. L. Kovács, “A multi-agent extension of pddl3.1,” in ICAPS 2012 Proceedings of the 3rd Workshop on the International Planning Competition (WS-IPC 2012), 2012, pp. 19–37.
  49. E. A. Emerson and E. M. Clarke, “Using branching time temporal logic to synthesize synchronization skeletons,” Science of Computer programming, vol. 2, no. 3, pp. 241–266, 1982.
  50. A. Donzé and O. Maler, “Robust satisfaction of temporal logic over real-valued signals,” in International Conference on Formal Modeling and Analysis of Timed Systems.   Springer, 2010, pp. 92–106.
  51. V. Raman, A. Donzé, M. Maasoumy, R. M. Murray, A. Sangiovanni-Vincentelli, and S. A. Seshia, “Model predictive control with signal temporal logic specifications,” in 53rd IEEE Conference on Decision and Control.   IEEE, 2014, pp. 81–87.
  52. V. Kurtz and H. Lin, “Mixed-integer programming for signal temporal logic with fewer binary variables,” IEEE Control Systems Letters, vol. 6, pp. 2635–2640, 2022.
  53. T. Silver, R. Chitnis, N. Kumar, W. McClinton, T. Lozano-Perez, L. P. Kaelbling, and J. Tenenbaum, “Predicate invention for bilevel planning,” in AAAI Conference on Artificial Intelligence (AAAI), 2023.
  54. R. Chitnis, T. Silver, J. B. Tenenbaum, T. Lozano-Perez, and L. P. Kaelbling, “Learning neuro-symbolic relational transition models for bilevel planning,” in 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2022, pp. 4166–4173.
  55. R. Chitnis, T. Silver, B. Kim, L. Kaelbling, and T. Lozano-Perez, “Camps: Learning context-specific abstractions for efficient planning in factored mdps,” in Conference on Robot Learning.   PMLR, 2021, pp. 64–79.
  56. T. Silver, R. Chitnis, A. Curtis, J. B. Tenenbaum, T. Lozano-Pérez, and L. P. Kaelbling, “Planning with learned object importance in large problem instances using graph neural networks,” in Proceedings of the AAAI conference on artificial intelligence, vol. 35, no. 13, 2021, pp. 11 962–11 971.
  57. Y. Zhu, J. Tremblay, S. Birchfield, and Y. Zhu, “Hierarchical planning for long-horizon manipulation with geometric and symbolic scene graphs,” in 2021 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2021, pp. 6541–6548.
  58. C. Wang, D. Xu, and L. Fei-Fei, “Generalizable task planning through representation pretraining,” IEEE Robotics and Automation Letters, vol. 7, no. 3, pp. 8299–8306, 2022.
  59. Y. Ding, X. Zhang, S. Amiri, N. Cao, H. Yang, A. Kaminski, C. Esselink, and S. Zhang, “Integrating action knowledge and llms for task planning and situation handling in open worlds,” Autonomous Robots, vol. 47, no. 8, pp. 981–997, 2023.
  60. B. Liu, Y. Jiang, X. Zhang, Q. Liu, S. Zhang, J. Biswas, and P. Stone, “Llm+p: Empowering large language models with optimal planning proficiency,” arXiv preprint arXiv:2304.11477, 2023.
  61. I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg, “Progprompt: Generating situated robot task plans using large language models,” in 2023 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2023, pp. 11 523–11 530.
  62. Z. Zhao, W. S. Lee, and D. Hsu, “Large language models as commonsense knowledge for large-scale task planning,” Advances in Neural Information Processing Systems, vol. 36, 2024.
  63. A. Z. Ren, A. Dixit, A. Bodrova, S. Singh, S. Tu, N. Brown, P. Xu, L. Takayama, F. Xia, J. Varley et al., “Robots that ask for help: Uncertainty alignment for large language model planners,” in Conference on Robot Learning.   PMLR, 2023, pp. 661–682.
  64. Y. Xie, C. Yu, T. Zhu, J. Bai, Z. Gong, and H. Soh, “Translating natural language to planning goals with large-language models,” arXiv preprint arXiv:2302.05128, 2023.
  65. J. Pan, G. Chou, and D. Berenson, “Data-efficient learning of natural language to linear temporal logic translators for robot task specification,” in 2023 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2023, pp. 11 554–11 561.
  66. Y. Chen, J. Arkin, Y. Zhang, N. Roy, and C. Fan, “Autotamp: Autoregressive task and motion planning with llms as translators and checkers,” arXiv preprint arXiv:2306.06531, 2023.
  67. J. Hoffmann, “Ff: The fast-forward planning system,” AI magazine, vol. 22, no. 3, pp. 57–57, 2001.
  68. J. A. Baier, F. Bacchus, and S. A. McIlraith, “A heuristic search approach to planning with temporally extended preferences,” Artificial Intelligence, vol. 173, no. 5-6, pp. 593–618, 2009.
  69. L. Zhu and R. Givan, “Simultaneous heuristic search for conjunctive subgoals,” in Proceedings of the 20th national conference on Artificial intelligence-Volume 3, 2005, pp. 1235–1240.
  70. M. Helmert, “The fast downward planning system,” Journal of Artificial Intelligence Research, vol. 26, pp. 191–246, 2006.
  71. S. Richter, M. Westphal, and M. Helmert, “Lama 2008 and 2011,” in International Planning Competition, 2011, pp. 117–124.
  72. I. Georgievski and M. Aiello, “An overview of hierarchical task network planning,” arXiv preprint arXiv:1403.7426, 2014.
  73. J. E. Hopcroft, R. Motwani, and J. D. Ullman, “Introduction to automata theory, languages, and computation,” Acm Sigact News, vol. 32, no. 1, pp. 60–65, 2001.
  74. A. Pnueli and R. Rosner, “On the synthesis of a reactive module,” in Proceedings of the 16th ACM SIGPLAN-SIGACT symposium on Principles of programming languages, 1989, pp. 179–190.
  75. S. Maoz and J. O. Ringert, “Gr (1) synthesis for ltl specification patterns,” in Proceedings of the 2015 10th joint meeting on foundations of software engineering, 2015, pp. 96–106.
  76. R. Ehlers and V. Raman, “Slugs: Extensible gr (1) synthesis,” in Computer Aided Verification: 28th International Conference, CAV 2016, Toronto, ON, Canada, July 17-23, 2016, Proceedings, Part II 28.   Springer, 2016, pp. 333–339.
  77. H. M. Pasula, L. S. Zettlemoyer, and L. P. Kaelbling, “Learning symbolic models of stochastic domains,” Journal of Artificial Intelligence Research, vol. 29, pp. 309–352, 2007.
  78. E. Amir and A. Chang, “Learning partially observable deterministic action models,” Journal of Artificial Intelligence Research, vol. 33, pp. 349–402, 2008.
  79. G. Konidaris, L. P. Kaelbling, and T. Lozano-Perez, “Symbol acquisition for probabilistic high-level planning.”   AAAI Press/International Joint Conferences on Artificial Intelligence, 2015.
  80. G. Konidaris, L. P. Kaelbling, and T. Lozano-Perez, “From skills to symbols: Learning symbolic representations for abstract high-level planning,” Journal of Artificial Intelligence Research, vol. 61, pp. 215–289, 2018.
  81. B. Ames, A. Thackston, and G. Konidaris, “Learning symbolic representations for planning with parameterized skills,” in 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2018, pp. 526–533.
  82. D. Xu, S. Nair, Y. Zhu, J. Gao, A. Garg, L. Fei-Fei, and S. Savarese, “Neural task programming: Learning to generalize across hierarchical tasks,” in 2018 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2018, pp. 3795–3802.
  83. D.-A. Huang, S. Nair, D. Xu, Y. Zhu, A. Garg, L. Fei-Fei, S. Savarese, and J. C. Niebles, “Neural task graphs: Generalizing to unseen tasks from a single video demonstration,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 8565–8574.
  84. D. Xu, R. Martín-Martín, D.-A. Huang, Y. Zhu, S. Savarese, and L. F. Fei-Fei, “Regression planning networks,” Advances in Neural Information Processing Systems, vol. 32, 2019.
  85. F. Ceola, E. Tosello, L. Tagliapietra, G. Nicola, and S. Ghidoni, “Robot task planning via deep reinforcement learning: a tabletop object sorting application,” in 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC).   IEEE, 2019, pp. 486–492.
  86. D. Xu, A. Mandlekar, R. Martín-Martín, Y. Zhu, S. Savarese, and L. Fei-Fei, “Deep affordance foresight: Planning through what can be done in the future,” in 2021 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2021, pp. 6206–6213.
  87. J. Liang, M. Sharma, A. LaGrassa, S. Vats, S. Saxena, and O. Kroemer, “Search-based task planning with learned skill effect models for lifelong robotic manipulation,” in 2022 International Conference on Robotics and Automation (ICRA).   IEEE, 2022, pp. 6351–6357.
  88. OpenAI, “Chatgpt,” Accessed: 2023-02-08, 2023, cit. on pp. 1, 16. [Online]. Available: https://openai.com/blog/chatgpt/
  89. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, “Llama: Open and efficient foundation language models,” 2023.
  90. P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig, “Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing,” ACM Computing Surveys, vol. 55, no. 9, pp. 1–35, 2023.
  91. S. Imani, L. Du, and H. Shrivastava, “Mathprompter: Mathematical reasoning using large language models,” in ICLR 2023 Workshop on Trustworthy and Reliable Large-Scale Machine Learning Models, 2023.
  92. V. Gaur and N. Saunshi, “Reasoning in large language models through symbolic math word problems,” in The 61st Annual Meeting Of The Association For Computational Linguistics, 2023.
  93. A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julian et al., “Do as i can, not as i say: Grounding language in robotic affordances,” in Conference on Robot Learning, 2023, pp. 287–318.
  94. W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y. Chebotar et al., “Inner monologue: Embodied reasoning through planning with language models,” in Conference on Robot Learning.   PMLR, 2023, pp. 1769–1782.
  95. Y. Hu, Q. Xie, V. Jain, J. Francis, J. Patrikar, N. Keetha, S. Kim, Y. Xie, T. Zhang, Z. Zhao et al., “Toward general-purpose robots via foundation models: A survey and meta-analysis,” arXiv preprint arXiv:2312.08782, 2023.
  96. Y. Ding, X. Zhang, C. Paxton, and S. Zhang, “Task and motion planning with large language models for object rearrangement,” in 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2023, pp. 2086–2092.
  97. J. T. Betts, “Survey of numerical methods for trajectory optimization,” Journal of guidance, control, and dynamics, vol. 21, no. 2, pp. 193–207, 1998.
  98. S. Boyd, N. Parikh, E. Chu, B. Peleato, J. Eckstein et al., “Distributed optimization and statistical learning via the alternating direction method of multipliers,” Foundations and Trends® in Machine learning, vol. 3, no. 1, pp. 1–122, 2011.
  99. I. Mordatch and E. Todorov, “Combining the benefits of function approximation and trajectory optimization.” in Robotics: Science and Systems, vol. 4, 2014, p. 23.
  100. M. Janner, Q. Li, and S. Levine, “Offline reinforcement learning as one big sequence modeling problem,” Advances in neural information processing systems, vol. 34, pp. 1273–1286, 2021.
  101. M. Kelly, “An introduction to trajectory optimization: How to do your own direct collocation,” SIAM Review, vol. 59, no. 4, pp. 849–904, 2017.
  102. D. Pardo, L. Möller, M. Neunert, A. W. Winkler, and J. Buchli, “Evaluating direct transcription and nonlinear optimization methods for robot motion planning,” IEEE Robotics and Automation Letters, vol. 1, no. 2, pp. 946–953, 2016.
  103. A. Wächter and L. T. Biegler, “On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programming,” Mathematical programming, vol. 106, pp. 25–57, 2006.
  104. P. E. Gill, W. Murray, and M. A. Saunders, “Snopt: An sqp algorithm for large-scale constrained optimization,” SIAM review, vol. 47, no. 1, pp. 99–131, 2005.
  105. D. Mayne, “A second-order gradient method for determining optimal trajectories of non-linear discrete-time systems,” International Journal of Control, vol. 3, no. 1, pp. 85–95, 1966.
  106. Z. Xie, C. K. Liu, and K. Hauser, “Differential dynamic programming with nonlinear constraints,” in 2017 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2017, pp. 695–702.
  107. B. Plancher, Z. Manchester, and S. Kuindersma, “Constrained unscented dynamic programming,” in 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2017, pp. 5674–5680.
  108. T. A. Howell, B. E. Jackson, and Z. Manchester, “Altro: A fast solver for constrained trajectory optimization,” in 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2019, pp. 7674–7679.
  109. J.-P. Sleiman, F. Farshidian, and M. Hutter, “Constraint handling in continuous-time ddp-based model predictive control,” in 2021 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2021, pp. 8209–8215.
  110. W. Jallet, A. Bambade, N. Mansard, and J. Carpentier, “Constrained differential dynamic programming: A primal-dual augmented lagrangian approach,” in 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2022, pp. 13 371–13 378.
  111. Y. Wang, H. Li, Y. Zhao, X. Chen, X. Huang, and Z. Jiang, “A fast coordinated motion planning method for dual-arm robot based on parallel constrained ddp,” IEEE/ASME Transactions on Mechatronics (Early Access), 2023.
  112. J. Eckstein and D. P. Bertsekas, “On the douglas—rachford splitting method and the proximal point algorithm for maximal monotone operators,” Mathematical programming, vol. 55, pp. 293–318, 1992.
  113. B. Wohlberg, “Admm penalty parameter selection by residual balancing,” arXiv preprint arXiv:1704.06209, 2017.
  114. T. Goldstein, B. O’Donoghue, S. Setzer, and R. Baraniuk, “Fast alternating direction optimization methods,” SIAM Journal on Imaging Sciences, vol. 7, no. 3, pp. 1588–1623, 2014.
  115. L. Ferranti, L. Lyons, R. R. Negenborn, T. Keviczky, and J. Alonso-Mora, “Distributed nonlinear trajectory optimization for multi-robot motion planning,” IEEE Transactions on Control Systems Technology, 2022.
  116. O. Shorinwa, T. Halsted, J. Yu, and M. Schwager, “Distributed optimization methods for multi-robot systems: Part ii–a survey,” arXiv preprint arXiv:2301.11361, 2023.
  117. R. Ni, Z. Pan, and X. Gao, “Robust multi-robot trajectory optimization using alternating direction method of multiplier,” IEEE Robotics and Automation Letters, vol. 7, no. 3, pp. 5950–5957, 2022.
  118. A. Aydinoglu and M. Posa, “Real-time multi-contact model predictive control via admm,” in 2022 International Conference on Robotics and Automation (ICRA).   IEEE, 2022, pp. 3414–3421.
  119. S. Le Cleac’h and Z. Manchester, “Fast solution of optimal control problems with l1 cost,” in AAS/AIAA Astrodynamics Specialist Conference, vol. 904, 2019, pp. 1–11.
  120. Z. Zhao, Z. Zhou, M. Park, and Y. Zhao, “Sydebo: Symbolic-decision-embedded bilevel optimization for long-horizon manipulation in dynamic environments,” IEEE Access, vol. 9, pp. 128 817–128 826, 2021.
  121. L. Wijayarathne, Z. Zhou, Y. Zhao, and F. L. Hammond, “Real-time deformable-contact-aware model predictive control for force-modulated manipulation,” IEEE Transactions on Robotics, pp. 1–18, 2023.
  122. H. Li, R. J. Frei, and P. M. Wensing, “Model hierarchy predictive control of robotic systems,” IEEE Robotics and Automation Letters, vol. 6, no. 2, pp. 3373–3380, 2021.
  123. C. Khazoom, S. Heim, D. Gonzalez-Diaz, and S. Kim, “Optimal scheduling of models and horizons for model hierarchy predictive control,” in 2023 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2023, pp. 9952–9958.
  124. A. Herzog, S. Schaal, and L. Righetti, “Structured contact force optimization for kino-dynamic motion generation,” in 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2016, pp. 2703–2710.
  125. Z. Zhou, B. Wingo, N. Boyd, S. Hutchinson, and Y. Zhao, “Momentum-aware trajectory optimization and control for agile quadrupedal locomotion,” IEEE Robotics and Automation Letters, vol. 7, no. 3, pp. 7755–7762, 2022.
  126. R. Budhiraja, J. Carpentier, and N. Mansard, “Dynamics consensus between centroidal and whole-body models for locomotion of legged robots,” in 2019 International Conference on Robotics and Automation (ICRA).   IEEE, 2019, pp. 6727–6733.
  127. A. Meduri, P. Shah, J. Viereck, M. Khadiv, I. Havoutis, and L. Righetti, “Biconmp: A nonlinear model predictive control framework for whole body motion planning,” IEEE Transactions on Robotics, 2023.
  128. Z. Zhou and Y. Zhao, “Accelerated admm based trajectory optimization for legged locomotion with coupled rigid body dynamics,” in 2020 American Control Conference (ACC).   IEEE, 2020, pp. 5082–5089.
  129. C. Finn, S. Levine, and P. Abbeel, “Guided cost learning: Deep inverse optimal control via policy optimization,” in International conference on machine learning.   PMLR, 2016, pp. 49–58.
  130. P. Englert, N. A. Vien, and M. Toussaint, “Inverse kkt: Learning cost functions of manipulation tasks from demonstrations,” The International Journal of Robotics Research, vol. 36, no. 13-14, pp. 1474–1488, 2017.
  131. P. Sharma, B. Sundaralingam, V. Blukis, C. Paxton, T. Hermans, A. Torralba, J. Andreas, and D. Fox, “Correcting robot plans with natural language feedback,” arXiv preprint arXiv:2204.05186, 2022.
  132. W. Yu, N. Gileadi, C. Fu, S. Kirmani, K.-H. Lee, M. G. Arenas, H.-T. L. Chiang, T. Erez, L. Hasenclever, J. Humplik et al., “Language to rewards for robotic skill synthesis,” in Conference on Robot Learning.   PMLR, 2023, pp. 374–404.
  133. W. Huang, C. Wang, R. Zhang, Y. Li, J. Wu, and L. Fei-Fei, “Voxposer: Composable 3d value maps for robotic manipulation with language models,” in Conference on Robot Learning.   PMLR, 2023, pp. 540–562.
  134. M. Parmar, M. Halm, and M. Posa, “Fundamental challenges in deep learning for stiff contact dynamics,” in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2021, pp. 5181–5188.
  135. S. Pfrommer, M. Halm, and M. Posa, “Contactnets: Learning discontinuous contact dynamics with smooth, implicit representations,” in Conference on Robot Learning.   PMLR, 2021, pp. 2279–2291.
  136. B. Bianchini, M. Halm, and M. Posa, “Simultaneous learning of contact and continuous dynamics,” in Conference on Robot Learning.   PMLR, 2023, pp. 3966–3978.
  137. S. Le Cleac’h, H.-X. Yu, M. Guo, T. Howell, R. Gao, J. Wu, Z. Manchester, and M. Schwager, “Differentiable physics simulation of dynamics-augmented neural objects,” IEEE Robotics and Automation Letters, vol. 8, no. 5, pp. 2780–2787, 2023.
  138. D. Driess, J.-S. Ha, M. Toussaint, and R. Tedrake, “Learning models as functionals of signed-distance fields for manipulation planning,” in Conference on robot learning.   PMLR, 2022, pp. 245–255.
  139. S. Levine and V. Koltun, “Guided policy search,” in International conference on machine learning.   PMLR, 2013, pp. 1–9.
  140. S. Levine and V. Koltun, “Learning complex neural network policies with trajectory optimization,” in International Conference on Machine Learning.   PMLR, 2014, pp. 829–837.
  141. A. Duburcq, Y. Chevaleyre, N. Bredeche, and G. Boéris, “Online trajectory planning through combined trajectory optimization and function approximation: Application to the exoskeleton atalante,” in 2020 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2020, pp. 3756–3762.
  142. Z. Zhao, S. Zuo, T. Zhao, and Y. Zhao, “Adversarially regularized policy learning guided by trajectory optimization,” in Learning for Dynamics and Control Conference.   PMLR, 2022, pp. 844–857.
  143. M. J. Bency, A. H. Qureshi, and M. C. Yip, “Neural path planning: Fixed time, near-optimal path generation via oracle imitation,” in 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2019, pp. 3965–3972.
  144. A. H. Qureshi, J. Dong, A. Choe, and M. C. Yip, “Neural manipulation planning on constraint manifolds,” IEEE Robotics and Automation Letters, vol. 5, no. 4, pp. 6089–6096, 2020.
  145. I. Radosavovic, T. Xiao, B. Zhang, T. Darrell, J. Malik, and K. Sreenath, “Learning humanoid locomotion with transformers,” arXiv preprint arXiv:2303.03381, 2023.
  146. C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song, “Diffusion policy: Visuomotor policy learning via action diffusion,” in Proceedings of Robotics: Science and Systems (RSS), 2023.
  147. J. Viereck and L. Righetti, “Learning a centroidal motion planner for legged locomotion,” in 2021 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2021, pp. 4905–4911.
  148. X. Lin, G. I. Fernandez, Y. Liu, T. Zhu, Y. Shirai, and D. Hong, “Multi-modal multi-agent optimization for limms, a modular robotics approach to delivery automation,” in 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2022, pp. 12 674–12 681.
  149. Z. Gu, R. Guo, W. Yates, Y. Chen, and Y. Zhao, “Walking-by-logic: Signal temporal logic-guided model predictive control for bipedal locomotion resilient to external perturbations,” arXiv preprint arXiv:2309.13172, 2023.
  150. M. Toussaint and M. Lopes, “Multi-bound tree search for logic-geometric programming in cooperative manipulation domains,” in 2017 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2017, pp. 4044–4051.
  151. S.-Y. Lo, S. Zhang, and P. Stone, “The petlon algorithm to plan efficiently for task-level-optimal navigation,” Journal of Artificial Intelligence Research, vol. 69, pp. 471–500, 2020.
  152. E. M. Wolff, U. Topcu, and R. M. Murray, “Optimization-based trajectory generation with linear temporal logic specifications,” in 2014 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2014, pp. 5319–5325.
  153. J. Chen, B. C. Williams, and C. Fan, “Optimal mixed discrete-continuous planning for linear hybrid systems,” in Proceedings of the 24th International Conference on Hybrid Systems: Computation and Control, 2021, pp. 1–12.
  154. T. Kogo, K. Takaya, and H. Oyama, “Fast milp-based task and motion planning for pick-and-place with hard/soft constraints of collision-free route,” in 2021 IEEE International Conference on Systems, Man, and Cybernetics (SMC).   IEEE, 2021, pp. 1020–1027.
  155. M. Katayama, S. Tokuda, M. Yamakita, and H. Oyama, “Fast ltl-based flexible planning for dual-arm manipulation,” in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2020, pp. 6605–6612.
  156. S. Saha and A. A. Julius, “Task and motion planning for manipulator arms with metric temporal logic specifications,” IEEE robotics and automation letters, vol. 3, no. 1, pp. 379–386, 2017.
  157. Y. V. Pant, H. Abbas, R. A. Quaye, and R. Mangharam, “Fly-by-logic: Control of multi-drone fleets with temporal logic objectives,” in 2018 ACM/IEEE 9th International Conference on Cyber-Physical Systems (ICCPS).   IEEE, 2018, pp. 186–197.
  158. C. Sun, N. Kingry, and R. Dai, “A unified formulation and nonconvex optimization method for mixed-type decision-making of robotic systems,” IEEE Transactions on Robotics, vol. 37, no. 3, pp. 831–846, 2020.
  159. F. Bacchus and Q. Yang, “Downward refinement and the efficiency of hierarchical problem solving,” Artificial Intelligence, vol. 71, no. 1, pp. 43–100, 1994.
  160. A. Akbari, F. Lagriffoul, and J. Rosell, “Combined heuristic task and motion planning for bi-manual robots,” Autonomous robots, vol. 43, no. 6, pp. 1575–1590, 2019.
  161. A. Agostini and J. Piater, “Unified task and motion planning using object-centric abstractions of motion constraints,” arXiv preprint arXiv:2312.17605, 2023.
  162. S. Srivastava, E. Fang, L. Riano, R. Chitnis, S. Russell, and P. Abbeel, “Combined task and motion planning through an extensible planner-independent interface layer,” in 2014 IEEE international conference on robotics and automation (ICRA).   IEEE, 2014, pp. 639–646.
  163. K. Hauser and J.-C. Latombe, “Multi-modal motion planning in non-expansive spaces,” The International Journal of Robotics Research, vol. 29, no. 7, pp. 897–915, 2010.
  164. Z. Kingston and L. E. Kavraki, “Scaling multimodal planning: Using experience and informing discrete search,” IEEE Transactions on Robotics, vol. 39, no. 1, pp. 128–146, 2022.
  165. M. Toussaint, “A tutorial on newton methods for constrained trajectory optimization and relations to slam, gaussian process smoothing, optimal control, and probabilistic inference,” Geometric and numerical foundations of movements, pp. 361–392, 2017.
  166. M. Toussaint, J.-S. Ha, and D. Driess, “Describing physics for physical reasoning: Force-based sequential manipulation planning,” IEEE Robotics and Automation Letters, vol. 5, no. 4, pp. 6209–6216, 2020.
  167. S. Zimmermann, G. Hakimifard, M. Zamora, R. Poranne, and S. Coros, “A multi-level optimization framework for simultaneous grasping and motion planning,” IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 2966–2972, 2020.
  168. M. S. Phoon, P. S. Schmitt, and G. v. Wichert, “Constraint-based task specification and trajectory optimization for sequential manipulation,” in 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2022, pp. 197–202.
  169. E. L. Lawler and D. E. Wood, “Branch-and-bound methods: A survey,” Operations research, vol. 14, no. 4, pp. 699–719, 1966.
  170. L. Huang, X. Chen, W. Huo, J. Wang, F. Zhang, B. Bai, and L. Shi, “Branch and bound in mixed integer linear programming problems: A survey of techniques and trends,” arXiv preprint arXiv:2111.06257, 2021.
  171. T. Berthold, “Primal heuristics for mixed integer programs,” Ph.D. dissertation, Zuse Institute Berlin (ZIB), 2006.
  172. S. S. Dey, Y. Dubey, M. Molinaro, and P. Shah, “A theoretical and computational analysis of full strong-branching,” Mathematical Programming, pp. 1–34, 2023.
  173. M. Fischetti and A. Lodi, “Local branching,” Mathematical programming, vol. 98, pp. 23–47, 2003.
  174. Gurobi Optimization, LLC, “Gurobi Optimizer Reference Manual,” 2023. [Online]. Available: https://www.gurobi.com
  175. T. M. Inc., “Matlab version: 9.13.0 (r2022b),” Natick, Massachusetts, United States, 2022. [Online]. Available: https://www.mathworks.com
  176. N. Funk, S. Menzenbach, G. Chalvatzaki, and J. Peters, “Graph-based reinforcement learning meets mixed integer programs: An application to 3d robot assembly discovery,” in 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2022, pp. 10 215–10 222.
  177. A. Shamsah, Z. Gu, J. Warnke, S. Hutchinson, and Y. Zhao, “Integrated task and motion planning for safe legged navigation in partially observable environments,” IEEE Transactions on Robotics, 2023.
  178. J. Warnke, A. Shamsah, Y. Li, and Y. Zhao, “Towards safe locomotion navigation in partially observable environments with uneven terrain,” in 2020 59th IEEE Conference on Decision and Control (CDC).   IEEE, 2020, pp. 958–965.
  179. Y. Shirai, X. Lin, A. Schperberg, Y. Tanaka, H. Kato, V. Vichathorn, and D. Hong, “Simultaneous contact-rich grasping and locomotion via distributed optimization enabling free-climbing for multi-limbed robots,” in 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2022, pp. 13 563–13 570.
  180. Y. V. Pant, H. Abbas, and R. Mangharam, “Smooth operator: Control using the smooth robustness of temporal logic,” in 2017 IEEE Conference on Control Technology and Applications (CCTA).   IEEE, 2017, pp. 1235–1240.
  181. N. Mehdipour, C.-I. Vasile, and C. Belta, “Arithmetic-geometric mean robustness for control from signal temporal logic specifications,” in 2019 American Control Conference (ACC).   IEEE, 2019, pp. 1690–1695.
  182. Y. Gilpin, V. Kurtz, and H. Lin, “A smooth robustness measure of signal temporal logic for symbolic control,” IEEE Control Systems Letters, vol. 5, no. 1, pp. 241–246, 2020.
  183. Z. Gu, Y. Zhao, Y. Chen, R. Guo, J. K. Leestma, G. S. Sawicki, and Y. Zhao, “Robust-locomotion-by-logic: Perturbation-resilient bipedal locomotion via signal temporal logic guided model predictive control,” arXiv preprint arXiv:2403.15993, 2024.
  184. A. M. Wells, N. T. Dantam, A. Shrivastava, and L. E. Kavraki, “Learning feasibility for task and motion planning in tabletop environments,” IEEE robotics and automation letters, vol. 4, no. 2, pp. 1255–1262, 2019.
  185. M. Noseworthy, C. Moses, I. Brand, S. Castro, L. Kaelbling, T. Lozano-Pérez, and N. Roy, “Active learning of abstract plan feasibility,” arXiv preprint arXiv:2107.00683, 2021.
  186. A. Mandlekar, C. Garrett, D. Xu, and D. Fox, “Human-in-the-loop task and motion planning for imitation learning,” in 7th Annual Conference on Robot Learning, 2023.
  187. T. Silver, A. Athalye, J. B. Tenenbaum, T. Lozano-Perez, and L. P. Kaelbling, “Learning neuro-symbolic skills for bilevel planning,” in Conference on Robot Learning (CoRL), 2022.
  188. A. Cauligi, P. Culbertson, B. Stellato, D. Bertsimas, M. Schwager, and M. Pavone, “Learning mixed-integer convex optimization strategies for robot planning and control,” in 2020 59th IEEE Conference on Decision and Control (CDC).   IEEE, 2020, pp. 1698–1705.
  189. Y. Sung, Z. Wang, and P. Stone, “Learning to correct mistakes: Backjumping in long-horizon task and motion planning,” in Conference on Robot Learning.   PMLR, 2023, pp. 2115–2124.
  190. A. Curtis, X. Fang, L. P. Kaelbling, T. Lozano-Pérez, and C. R. Garrett, “Long-horizon manipulation of unknown objects via task and motion planning with estimated affordances,” in 2022 International Conference on Robotics and Automation (ICRA).   IEEE, 2022, pp. 1940–1946.
  191. Z. Wang, C. R. Garrett, L. P. Kaelbling, and T. Lozano-Pérez, “Active model learning and diverse action sampling for task and motion planning,” in 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2018, pp. 4107–4114.
  192. Z. Wang, C. R. Garrett, L. P. Kaelbling, and T. Lozano-Pérez, “Learning compositional models of robot skills for task and motion planning,” The International Journal of Robotics Research, vol. 40, no. 6-7, pp. 866–894, 2021.
  193. B. Kim, L. Kaelbling, and T. Lozano-Pérez, “Guiding search in continuous state-action spaces by learning an action sampler from off-target search experience,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, no. 1, 2018.
  194. B. Kim, Z. Wang, L. P. Kaelbling, and T. Lozano-Pérez, “Learning to guide task and motion planning using score-space representation,” The International Journal of Robotics Research, vol. 38, no. 7, pp. 793–812, 2019.
  195. R. Chitnis, D. Hadfield-Menell, A. Gupta, S. Srivastava, E. Groshev, C. Lin, and P. Abbeel, “Guided search for task and motion plans using learned heuristics,” in 2016 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2016, pp. 447–454.
  196. B. Kim and L. Shimanuki, “Learning value functions with relational state representations for guiding task-and-motion planning,” in Conference on Robot Learning.   PMLR, 2020, pp. 955–968.
  197. J. Ortiz-Haro, V. N. Hartmann, O. S. Oguz, and M. Toussaint, “Learning efficient constraint graph sampling for robotic sequential manipulation,” in 2021 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2021, pp. 4606–4612.
  198. J. Ortiz-Haro, J.-S. Ha, D. Driess, and M. Toussaint, “Structured deep generative models for sampling on constraint manifolds in sequential manipulation,” in Conference on Robot Learning.   PMLR, 2022, pp. 213–223.
  199. Z. Li, K. Yu, S. Cheng, and D. Xu, “League++: Empowering continual robot learning through guided skill acquisition with large language models,” in ICLR 2024 Workshop on Large Language Model (LLM) Agents.
  200. Y. Meng and C. Fan, “Signal temporal logic neural predictive control,” IEEE Robotics and Automation Letters, 2023.
  201. T. Pan, A. M. Wells, R. Shome, and L. E. Kavraki, “Failure is an option: Task and motion planning with failing executions,” in 2022 International Conference on Robotics and Automation (ICRA).   IEEE, 2022, pp. 1947–1953.
  202. A. S. Wang and O. Kroemer, “Learning robust manipulation strategies with multimodal state transition models and recovery heuristics,” in 2019 International Conference on Robotics and Automation (ICRA).   IEEE, 2019, pp. 1309–1315.
  203. S. Luo, H. Wu, S. Duan, Y. Lin, and J. Rojas, “Endowing robots with longer-term autonomy by recovering from external disturbances in manipulation through grounded anomaly classification and recovery policies,” Journal of Intelligent & Robotic Systems, vol. 101, pp. 1–40, 2021.
  204. X. Zhang, Y. Ding, S. Amiri, H. Yang, A. Kaminski, C. Esselink, and S. Zhang, “Grounding classical task planners via vision-language models,” arXiv preprint arXiv:2304.08587, 2023.
  205. Z. Liu, A. Bahety, and S. Song, “Reflect: Summarizing robot experiences for failure explanation and correction,” in Conference on Robot Learning.   PMLR, 2023, pp. 3468–3484.
  206. K. Rana, J. Haviland, S. Garg, J. Abou-Chakra, I. Reid, and N. Suenderhauf, “Sayplan: Grounding large language models using 3d scene graphs for scalable robot task planning,” in 7th Annual Conference on Robot Learning, 2023.
  207. J. Zhang, C. Liu, X. Li, H.-L. Zhen, M. Yuan, Y. Li, and J. Yan, “A survey for solving mixed integer programming via machine learning,” Neurocomputing, vol. 519, pp. 205–217, 2023.
  208. Y. Bengio, A. Lodi, and A. Prouvost, “Machine learning for combinatorial optimization: a methodological tour d’horizon,” European Journal of Operational Research, vol. 290, no. 2, pp. 405–421, 2021.
  209. A. Marcos Alvarez, Q. Louveaux, and L. Wehenkel, “A supervised machine learning approach to variable branching in branch-and-bound,” Technical Report, 2014.
  210. E. Khalil, P. Le Bodic, L. Song, G. Nemhauser, and B. Dilkina, “Learning to branch in mixed integer programming,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 30, no. 1, 2016.
  211. M. Gasse, D. Chételat, N. Ferroni, L. Charlin, and A. Lodi, “Exact combinatorial optimization with graph convolutional neural networks,” Advances in neural information processing systems, vol. 32, 2019.
  212. J. Song, Y. Yue, B. Dilkina et al., “A general large neighborhood search framework for solving integer linear programs,” Advances in Neural Information Processing Systems, vol. 33, pp. 20 012–20 023, 2020.
  213. N. Sonnerat, P. Wang, I. Ktena, S. Bartunov, and V. Nair, “Learning a large neighborhood search algorithm for mixed integer programs,” arXiv preprint arXiv:2107.10201, 2021.
  214. D. Liu, M. Fischetti, and A. Lodi, “Learning to search in local branching,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 4, 2022, pp. 3796–3803.
  215. M. Srinivasan, A. Chakrabarty, R. Quirynen, N. Yoshikawa, T. Mariyama, and S. Di Cairano, “Fast multi-robot motion planning via imitation learning of mixed-integer programs,” IFAC-PapersOnLine, vol. 54, no. 20, pp. 598–604, 2021.
  216. A. Cauligi, P. Culbertson, E. Schmerling, M. Schwager, B. Stellato, and M. Pavone, “Coco: Online mixed-integer control via supervised learning,” IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 1447–1454, 2021.
  217. R. Deits, T. Koolen, and R. Tedrake, “Lvis: Learning from value function intervals for contact-aware robot controllers,” in 2019 International Conference on Robotics and Automation (ICRA).   IEEE, 2019, pp. 7762–7768.
  218. A. Ajay, S. Han, Y. Du, S. Li, A. Gupta, T. Jaakkola, J. Tenenbaum, L. Kaelbling, A. Srivastava, and P. Agrawal, “Compositional foundation models for hierarchical planning,” Advances in Neural Information Processing Systems, vol. 36, 2024.
  219. Y. Qiu, Z. Zhao, Y. Ziser, A. Korhonen, E. M. Ponti, and S. B. Cohen, “Are large language models temporally grounded?” arXiv preprint arXiv:2311.08398, 2023.
  220. X. L. Li, A. Kuncoro, J. Hoffmann, C. de Masson d’Autume, P. Blunsom, and A. Nematzadeh, “A systematic investigation of commonsense knowledge in large language models,” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022, pp. 11 838–11 855.
  221. J. Chen, W. Shi, Z. Fu, S. Cheng, L. Li, and Y. Xiao, “Say what you mean! large language models speak too positively about negative commonsense knowledge,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023, pp. 9890–9908.
  222. B. Chen, Z. Xu, S. Kirmani, B. Ichter, D. Driess, P. Florence, D. Sadigh, L. Guibas, and F. Xia, “Spatialvlm: Endowing vision-language models with spatial reasoning capabilities,” arXiv preprint arXiv:2401.12168, 2024.
  223. J. Rocamonde, V. Montesinos, E. Nava, E. Perez, and D. Lindner, “Vision-language models are zero-shot reward models for reinforcement learning,” arXiv preprint arXiv:2310.12921, 2023.
  224. L. Chen, B. Li, S. Shen, J. Yang, C. Li, K. Keutzer, T. Darrell, and Z. Liu, “Large language models are visual reasoning coordinators,” arXiv preprint arXiv:2310.15166, 2023.
  225. L. Liu, X. Yang, Y. Shen, B. Hu, Z. Zhang, J. Gu, and G. Zhang, “Think-in-memory: Recalling and post-thinking enable llms with long-term memory,” arXiv preprint arXiv:2311.08719, 2023.
  226. W. Wang, L. Dong, H. Cheng, X. Liu, X. Yan, J. Gao, and F. Wei, “Augmenting language models with long-term memory,” Advances in Neural Information Processing Systems, vol. 36, 2024.
  227. S. S. Kannan, V. L. Venkatesh, and B.-C. Min, “Smart-llm: Smart multi-agent robot task planning using large language models,” arXiv preprint arXiv:2309.10062, 2023.
  228. J. Yang, H. Zhang, F. Li, X. Zou, C. Li, and J. Gao, “Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v,” arXiv preprint arXiv:2310.11441, 2023.
  229. B. Chen, F. Xia, B. Ichter, K. Rao, K. Gopalakrishnan, M. S. Ryoo, A. Stone, and D. Kappler, “Open-vocabulary queryable scene representations for real world planning,” in 2023 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2023, pp. 11 509–11 522.
  230. L. Wang, Y. Ling, Z. Yuan, M. Shridhar, C. Bao, Y. Qin, B. Wang, H. Xu, and X. Wang, “Gensim: Generating robotic simulation tasks via large language models,” in The Twelfth International Conference on Learning Representations, 2023.
  231. M. T. Mason, “Toward robotic manipulation,” Annual Review of Control, Robotics, and Autonomous Systems, vol. 1, pp. 1–28, 2018.
  232. Z. Zhang, J. Yan, X. Kong, G. Zhai, and Y. Liu, “Efficient motion planning based on kinodynamic model for quadruped robots following persons in confined spaces,” IEEE/ASME Transactions on Mechatronics, vol. 26, no. 4, pp. 1997–2006, 2021.
  233. C. McGreavy and Z. Li, “Reachability map for diverse and energy efficient stepping of humanoids,” IEEE/ASME Transactions on Mechatronics, vol. 27, no. 6, pp. 5307–5317, 2022.
  234. C. Sferrazza, D.-M. Huang, X. Lin, Y. Lee, and P. Abbeel, “Humanoidbench: Simulated humanoid benchmark for whole-body locomotion and manipulation,” arXiv preprint arXiv:2403.10506, 2024.
  235. W. Liu, X. Liang, and M. Zheng, “Task-constrained motion planning considering uncertainty-informed human motion prediction for human–robot collaborative disassembly,” IEEE/ASME Transactions on Mechatronics, vol. 28, no. 4, pp. 2056–2063, 2023.
  236. Y. Cheng, L. Sun, and M. Tomizuka, “Human-aware robot task planning based on a hierarchical task model,” IEEE Robotics and Automation Letters, vol. 6, no. 2, pp. 1136–1143, 2021.
  237. K. Darvish, E. Simetti, F. Mastrogiovanni, and G. Casalino, “A hierarchical architecture for human–robot cooperation processes,” IEEE Transactions on Robotics, vol. 37, no. 2, pp. 567–586, 2020.
  238. M. Faroni, A. Umbrico, M. Beschi, A. Orlandini, A. Cesta, and N. Pedrocchi, “Optimal task and motion planning and execution for multiagent systems in dynamic environments,” IEEE Transactions on Cybernetics, 2023.
  239. A. T. Le, P. Kratzer, S. Hagenmayer, M. Toussaint, and J. Mainprice, “Hierarchical human-motion prediction and logic-geometric programming for minimal interference human-robot tasks,” in 2021 30th IEEE International Conference on Robot & Human Interactive Communication (RO-MAN).   IEEE, 2021, pp. 7–14.
  240. C. Zhang, J. Chen, J. Li, Y. Peng, and Z. Mao, “Large language models for human-robot interaction: A review,” Biomimetic Intelligence and Robotics, p. 100131, 2023.
  241. A. Kshirsagar, H. Kress-Gazit, and G. Hoffman, “Specifying and synthesizing human-robot handovers,” in 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2019, pp. 5930–5936.
  242. A. Mörtl, M. Lawitzky, A. Kucukyilmaz, M. Sezgin, C. Basdogan, and S. Hirche, “The role of roles: Physical cooperation between humans and robots,” The International Journal of Robotics Research, vol. 31, no. 13, pp. 1656–1674, 2012.
Citations (5)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now