Thought of Search: Planning with Language Models Through The Lens of Efficiency

Abstract: Among the most important properties of algorithms investigated in computer science are soundness, completeness, and complexity. These properties, however, are rarely analyzed for the vast collection of recently proposed methods for planning with LLMs. In this work, we alleviate this gap. We analyse these properties of using LLMs for planning and highlight that recent trends abandon both soundness and completeness for the sake of inefficiency. We propose a significantly more efficient approach that can, at the same time, maintain both soundness and completeness. We exemplify on four representative search problems, comparing to the LLM-based solutions from the literature that attempt to solve these problems. We show that by using LLMs to produce the code for the search components we can solve the entire datasets with 100\% accuracy with only a few calls to the LLM. We argue for a responsible use of compute resources; urging research community to investigate sound and complete LLM-based approaches that uphold efficiency.

• The paper introduces a symbolic representation approach that minimizes repeated LLM evaluations and reduces computational complexity.
• It compares various LLM-based planning methods, highlighting efficiency gains over traditional token-based strategies.
• Experimental results on the 24 Game demonstrate improved success rates and reduced costs, emphasizing broader application potential.

Thought of Search: Planning with LLMs Through The Lens of Efficiency

Introduction

This paper titled "Thought of Search: Planning with LLMs Through The Lens of Efficiency" (2404.11833) presents a study on the use of LLMs for planning tasks and challenges current practices by highlighting inefficiencies in computational resource utilization. The authors critically assess the complexity of various planning methods incorporating LLMs, arguing that despite their promising capabilities for natural language processing, employing LLMs for planning needs a methodological shift towards efficiency and deterministic soundness.

Complexity of Current Approaches

The paper delineates several existing strategies for employing LLMs in planning and categorizes them based upon the complexity of LLM evaluations and the resultant computational demands. The methods analyzed include:

• Input-Output (IO) Prompting: Evaluated once per input, resulting in $O(1)$ complexity.
• Chain-of-Thought (CoT): Uses predefined examples for generating thought sequences; similarly maintains $O(1)$ complexity.
• ReAct: Interleaves reasoning with action steps, invoking LLM repeatedly for policy-like evaluations, leading to $O(n)$ complexity due to step-wise rollouts.
• ReWOO: Optimizes evaluation frequency via initial broad planning, potentially reducing to $O(1)$ evaluations though the worst case remains $O(n)$.
• Reasoning via Planning (RAP) and Tree of Thoughts (ToT): Implement search-based expansions using LLMs, incurring $O(N \times d \times L)$ and $O(b \times k \times T)$ complexities respectively.
• Graph of Thoughts (GoT): Extends the complexity with graph-based thought aggregation, similar to ToT in $O(N)$ complexity.

Additionally, some methods attempt a more generalized approach by deriving policies directly using LLMs with problems solvable through symbolic representations, optimizing complexity to $O(1)$ per domain.

Proposed Approach

The authors advocate for a more practical and sustainable utilization of LLMs in planning by suggesting a symbolic representation approach. This involves pre-processing search components using LLM-generated models rather than continuous evaluations during the search. By implementing a search mechanism like STRIPS or PDDL, planners can efficiently derive successors, goal tests, and heuristics without redundant LLM queries.

The work emphasizes utilizing LLMs to code generation components such as the successor state function and goal recognition in the example of the 24 Game. This method reduces unnecessary computational burden by validating and checking correctness pre-search, thereby ensuring soundness and completeness in the planning process.

Experimentation and Results

An experiment with the Python-based 24 Game showcases the practical efficacy of minimal LLM interactions, requiring fewer evaluations and yielding consistent results across 1362 test instances with varying difficulties. The solution rates, up to 45–75% with ToT, reveal the comparative advantage of symbolic search and programming integration facilitated by LLMs against purely token-based expansions. It is underscored that these principles can significantly cut down operational costs and improve accuracy in broader planning applications.

Conclusion

The paper effectively proposes an alternative approach to LLM utilization in planning tasks by demonstrating that symbolic models and search structures might offer more efficient and feasible solutions. The potential of deriving key planning components using LLM-generated models establishes prospects for further refining AI-driven search algorithms. The authors invite the research community to explore these opportunities, focusing on both the fidelity of planning outputs and optimization of computing resources.