ADaPT: As-Needed Decomposition and Planning with Language Models (2311.05772v2)

Abstract: LLMs are increasingly being used for interactive decision-making tasks requiring planning and adapting to the environment. Recent works employ LLMs-as-agents in broadly two ways: iteratively determining the next action (iterative executors) or generating plans and executing sub-tasks using LLMs (plan-and-execute). However, these methods struggle with task complexity, as the inability to execute any sub-task may lead to task failure. To address these shortcomings, we introduce As-Needed Decomposition and Planning for complex Tasks (ADaPT), an approach that explicitly plans and decomposes complex sub-tasks as-needed, i.e., when the LLM is unable to execute them. ADaPT recursively decomposes sub-tasks to adapt to both task complexity and LLM capability. Our results demonstrate that ADaPT substantially outperforms established strong baselines, achieving success rates up to 28.3% higher in ALFWorld, 27% in WebShop, and 33% in TextCraft -- a novel compositional dataset that we introduce. Through extensive analysis, we illustrate the importance of multilevel decomposition and establish that ADaPT dynamically adjusts to the capabilities of the executor LLM as well as to task complexity.

Evaluation of ADaPT: As-Needed Decomposition and Planning with LLMs

The paper "ADaPT: As-Needed Decomposition and Planning with LLMs" presents a nuanced approach to employing LLMs as interactive agents for decision-making tasks. ADaPT, standing for "As-Needed Decomposition and Planning with LLMs," proposes a recursive strategy for task execution that dynamically decomposes complex tasks into manageable sub-tasks based on the capabilities of the executing LLM. This methodology marks a departure from extant "plan-and-execute" and iterative execution strategies, which often falter under task complexity and sub-task failure scenarios.

The experimental evaluation showcases the robustness of ADaPT across a set of challenging environments—ALFWorld, WebShop, and TextCraft—each embodying different facets of interactive decision-making tasks. In ALFWorld, a simulated household environment, ADaPT improves task success by up to 28.3% over standard baselines such as ReAct and Plan-and-Solve. The WebShop platform, embodying the challenges of navigating e-commerce websites, sees a 27% boost in task success with ADaPT, while TextCraft, a new task environment requiring the crafting of items based on Minecraft recipes, reports a standout performance increase of 33%.

This performance is attributed to ADaPT's ability to dynamically decompose tasks with recursive logic when the executor, a LLM, fails to achieve sub-task objectives. By combining a planner module that creates high-level plans and an executor's self-assessment of task completion, ADaPT facilitates a flexible approach to managing task complexity and LLM execution capability divergence. Such dynamic adaptability is pivotal when confronting intricate and novel task requirements, which may not be easily anticipated at the planning stage alone.

Methodologically, ADaPT leverages separate LLM-based modules for planning and execution within a controlling framework, enabling iterative task adaptation and the recursive decomposition of sub-tasks. The successful orchestration of these modules is operationalized in a controller LLM program that handles communication between modules, aligns execution with task complexity, and enforces termination when maximal recursive depth is reached. This scheme allows ADaPT to manage more efficaciously the task complexity inherent in environments such as ALFWorld and WebShop.

A notable contribution of this work is its demonstration of improved task execution via recursive decomposition without incurring extensive planning overhead or dependence on extensive feedback and memory systems. The real-world applicability of ADaPT is underscored by its ability to seamlessly integrate with diverse environments—demonstrating not only a theoretical advance but also promising practical applicability in fields such as assistive robotics, web-based automation, and interactive gaming engines.

Furthermore, the analysis extends to illustrate how varying execution capabilities of different LLMs can be accommodated within the ADaPT structure. This adaptability not only supports scaling across LLM levels but also underpins the methodological integration of different LLMs for enhanced performance, thus adding significant versatility to ADaPT's usability.

In conclusion, ADaPT stands as a significant advancement in LLM-based task execution frameworks. By enabling as-needed task decomposition, it presents a scalable, adaptive solution applicable across complex, multi-layered environments. Future explorations could refine this methodology further by integrating additional feedback mechanisms and exploring even broader application domains, thereby extending the capabilities and performance of automated decision-making systems aligned with human-like reasoning and task execution.