Thought of Search: Planning with Language Models Through The Lens of Efficiency
Abstract: Among the most important properties of algorithms investigated in computer science are soundness, completeness, and complexity. These properties, however, are rarely analyzed for the vast collection of recently proposed methods for planning with LLMs. In this work, we alleviate this gap. We analyse these properties of using LLMs for planning and highlight that recent trends abandon both soundness and completeness for the sake of inefficiency. We propose a significantly more efficient approach that can, at the same time, maintain both soundness and completeness. We exemplify on four representative search problems, comparing to the LLM-based solutions from the literature that attempt to solve these problems. We show that by using LLMs to produce the code for the search components we can solve the entire datasets with 100\% accuracy with only a few calls to the LLM. We argue for a responsible use of compute resources; urging research community to investigate sound and complete LLM-based approaches that uphold efficiency.
- Graph of Thoughts: Solving Elaborate Problems with Large Language Models. In AAAI, 17682–17690. AAAI Press.
- Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planning. In Proceedings of the Thirty-Seventh Annual Conference on Neural Information Processing Systems (NeurIPS 2023).
- DeepSeek-Coder: When the Large Language Model Meets Programming – The Rise of Code Intelligence. arXiv:2401.14196 [cs.SE].
- Reasoning with Language Model is Planning with World Model. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023).
- Hart, P. E.; et al. 1968. A Formal Basis for the Heuristic Determination of Minimum Cost Paths. IEEE Transactions on Systems Science and Cybernetics, 4(2): 100–107.
- Helmert, M. 2006. The Fast Downward Planning System. 26: 191–246.
- OpenAI Dev. Forum. 2024. Performance analysis of Assistants versus Chat completion. https://community.openai.com/t/performance-analysis-of-assistants-versus-chat-completion-chat-completion-seems-somewhat-faster-for-complete-message-generation-streaming-taken-into-account/628368.
- Large Language Models as Planning Domain Generators. In Bernardini, S.; and Muise, C., eds., Proceedings of the Thirty-Fourth International Conference on Automated Planning and Scheduling (ICAPS 2024). AAAI Press.
- Generalized Planning in PDDL Domains with Pretrained Large Language Models. In Dy, J.; and Natarajan, S., eds., Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence (AAAI 2024). AAAI Press.
- Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the Thirty-Sixth Annual Conference on Neural Information Processing Systems (NeurIPS 2022), 24824–24837.
- Rewoo: Decoupling reasoning from observations for efficient augmented language models. arXiv:2305.18323 [cs.CL].
- Tree of thoughts: Deliberate problem solving with large language models. In Proceedings of the Thirty-Seventh Annual Conference on Neural Information Processing Systems (NeurIPS 2023).
- ReAct: Synergizing Reasoning and Acting in Language Models. In Proceedings of the Eleventh International Conference on Learning Representations (ICLR 2023). OpenReview.net.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.