Planning in the Dark: LLM-Symbolic Planning Pipeline without Experts (2409.15915v1)

Abstract: LLMs have shown promise in solving natural language-described planning tasks, but their direct use often leads to inconsistent reasoning and hallucination. While hybrid LLM-symbolic planning pipelines have emerged as a more robust alternative, they typically require extensive expert intervention to refine and validate generated action schemas. It not only limits scalability but also introduces a potential for biased interpretation, as a single expert's interpretation of ambiguous natural language descriptions might not align with the user's actual intent. To address this, we propose a novel approach that constructs an action schema library to generate multiple candidates, accounting for the diverse possible interpretations of natural language descriptions. We further introduce a semantic validation and ranking module that automatically filter and rank the generated schemas and plans without expert-in-the-loop. The experiments showed our pipeline maintains superiority in planning over the direct LLM planning approach. These findings demonstrate the feasibility of a fully automated end-to-end LLM-symbolic planner that requires no expert intervention, opening up the possibility for a broader audience to engage with AI planning with less prerequisite of domain expertise.

• The paper introduces a robust planning pipeline that leverages multiple LLM instances to generate diverse action schemas, boosting solvability from under 0.0001% to over 95%.
• It employs semantic coherence filtering using sentence encoders and a conformal prediction framework to ensure accurate alignment with intended task descriptions.
• The pipeline ranks generated plans based on cumulative similarity scores, resulting in high-quality, expert-independent action plans for complex tasks.

Planning in the Dark: LLM-Symbolic Planning Pipeline without Experts

The paper "Planning in the Dark: LLM-Symbolic Planning Pipeline without Experts" by Sukai Huang, Nir Lipovetzky and Trevor Cohn tackles the problem of generating robust action schemas from natural language descriptions using LLMs, without relying on expert intervention. This research addresses critical issues in the existing LLM-based planning frameworks, particularly the need for expert intervention and the risk of biased interpretations.

Key Contributions

1. Diverse Action Schema Library Construction: The authors propose leveraging multiple LLM instances to generate diverse action schema candidates. This diversity captures the various possible interpretations inherent in natural language descriptions. The approach is theoretically grounded in the observation that relying on a single LLM instance significantly reduces the likelihood of generating a solvable action schema set. By generating multiple candidate schemas, the probability of obtaining at least one solvable set increases dramatically (e.g., from less than 0.0001% to over 95% with 10 LLM instances).
2. Semantic Coherence Filtering: A sentence encoder is employed to validate and filter generated action schemas based on semantic similarity scores. This step ensures that the generated schemas align closely with the intended task descriptions, addressing semantic errors that LLMs might introduce. The process is statistically reinforced using a Conformal Prediction (CP) framework to set a confidence threshold, thus maintaining high-quality schema sets.
3. Plan Generation and Ranking: Using the filtered action schemas, the LLM-symbolic planner generates multiple action plans. These plans are then ranked based on the cumulative semantic similarity scores of their constituent action schemas. This method ensures that the generated plans are not only feasible but also aligned with the user's descriptions and intentions.

Experimental Validation

The experiments validate several hypotheses:

• Semantic Equivalence: Pre-trained sentence encoders demonstrated high, yet improvable, precision in distinguishing semantically aligned action schema-description pairs from misaligned ones. Finetuning the encoders with manipulated, hard-negative samples further improved this ability.
• Ambiguity Handling: Testing with both detailed and layman descriptions showed that non-expert inputs led to more diverse, solvable action schema sets, underscoring the system's ability to navigate the inherent ambiguity in natural language.
• Efficiency and Effectiveness: The implementation of CP significantly reduced the computational overhead by filtering out low-quality schemas early on, yet still maintained a high ratio of solvable sets.
• Plan Quality: The pipeline produced high-quality plans without expert intervention, favorably compared to direct LLM-based planning approaches, including in complex tasks like the Sussman Anomaly.

Comparisons and Insights

Comparisons with other approaches (e.g., \citet{guan2023leveraging} and \citet{yao2024tree}) highlight the strengths and weaknesses of each method:

• Tree-of-Thought (ToT) models, while generating multiple plans, fail in guaranteeing soundness due to inherent limitations in long-term probabilistic planning.
• Expert-dependent models, despite producing sound schemas, suffer from scalability and potential biases, making them less practical for broader applications.

In contrast, the proposed pipeline combines the advantages of both worlds, generating multiple, ranked, and semantically coherent plans without requiring extensive expert input. This marks a significant improvement in both the accessibility and efficiency of AI planning systems.

Implications and Future Developments

Practical Implications:

• This research opens the door for broader engagement with AI planning technologies, allowing users without extensive domain expertise to generate robust plans.
• It significantly reduces the need for iterative expert feedback, thus saving resources and time, particularly crucial in large-scale applications.

Theoretical Implications:

• The method demonstrates the feasibility of fully automated, end-to-end LLM-symbolic planning systems, which can adapt to diverse task domains and user inputs.
• This approach importantly maintains the balance between flexibility and correctness by leveraging the complementary strengths of LLMs and symbolic planners.

Future Directions:

• Further advancements in sentence encoding models could lead to even more accurate filtering and ranking mechanisms.
• Exploring methods to dynamically adjust predicate lists and schema forms based on real-time feedback from symbolic planners could enhance the adaptability of the model.
• Developing evaluation metrics tailored to dynamically generated schema models would provide better benchmarks for future research.

In conclusion, the paper's methodologies and insights significantly advance the field of AI planning, providing a scalable, efficient, and user-friendly framework for generating precise action schemas and plans from natural language descriptions.