Formal-LLM: Integrating Formal Language and Natural Language for Controllable LLM-based Agents (2402.00798v4)

Abstract: Recent advancements on LLMs enable AI Agents to automatically generate and execute multi-step plans to solve complex tasks. However, since LLM's content generation process is hardly controllable, current LLM-based agents frequently generate invalid or non-executable plans, which jeopardizes the performance of the generated plans and corrupts users' trust in LLM-based agents. In response, this paper proposes a novel "Formal-LLM" framework for LLM-based agents by integrating the expressiveness of natural language and the precision of formal language. Specifically, the framework allows agent developers to express their requirements or constraints for the planning process as an automaton. A stack-based LLM plan generation process is then conducted under the supervision of the automaton to ensure that the generated plan satisfies the constraints, making the planning process controllable. We conduct experiments on both benchmark tasks and practical real-life tasks, and our framework achieves over 50% overall performance increase, which validates the feasibility and effectiveness of employing Formal-LLM to guide the plan generation of agents, preventing the agents from generating invalid and unsuccessful plans. Further, more controllable LLM-based agents can facilitate the broader utilization of LLM in application scenarios where high validity of planning is essential. The source code of this work is available at https://github.com/agiresearch/Formal-LLM.

Integrating Formal Language with Natural Language for Enhanced LLM-Based Agent Control

Introduction

The integration of formal language and natural language offers a promising approach to enhance the controllability of LLMs in generating and executing multi-step plans. This paper introduces a novel framework, Formal-LLM, designed to amalgamate the expressiveness of natural language with the precision inherent to formal languages. The primary challenge addressed is the frequent generation of invalid or non-executable plans by current LLM-based agents, a critical issue that undermines the effectiveness and reliability of these models.

The Formal-LLM Framework

Motivation and Challenges

The necessity for a structured and controllable planning mechanism in LLM-based agents has become increasingly apparent, given the propensity of these models to generate plans that are often invalid or unexecutable in practical scenarios. The introduction of the Formal-LLM framework is motivated by the need to leverage the strengths of both formal and natural languages to mitigate these issues. The framework aims to ensure that the plans generated are both valid and executable by enforcing constraints derived from formal languages while maintaining the fluidity and adaptability provided by natural language processing.

Framework Components

The core of the Formal-LLM framework comprises the conversion of natural language constraints into a formalized structure that utilizes Context-Free Grammar (CFG) and Pushdown Automaton (PDA). These components are essential for representing and enforcing planning constraints, thereby guiding the LLM in generating valid plans. The framework introduces a mechanism for translating these formal constraints into a PDA, which then oversees the planning process, ensuring compliance with the predefined constraints.

• Context-Free Grammar (CFG): Utilized to define the constraints in a structured and precise manner, enabling the delineation of valid planning sequences and actions.
• Pushdown Automaton (PDA): Serves as the executional backbone of the framework, directing the LLM’s planning process in accordance with the constraints specified by the CFG.
• Backtracking Mechanism: To address potential dead ends in plan generation, a backtracking mechanism is incorporated, allowing for the revisitation of previous steps and the exploration of alternative pathways.
• Integration with Reinforcement Learning (RL): The framework enhances its application by incorporating reinforcement learning from task feedback (RLTF), leveraging valid plan execution as a basis for model fine-tuning.

Experimental Insights

The application of the Formal-LLM framework across various tasks demonstrated substantial improvements in the quality and validity of the generated plans. Experiments conducted on benchmark tasks revealed over a 50% increase in overall performance metrics compared to baseline approaches. Notably, the framework exhibited a remarkable ability to consistently generate executable plans across diverse scenarios, validating its practical efficacy and versatility.

Implications and Future Directions

The Formal-LLM framework presents a significant advancement in the domain of LLM-based agents, offering enhanced control over the planning process through the integration of formal and natural languages. This approach not only addresses the current limitations regarding the generation of valid and executable plans but also opens new avenues for research and application in complex task environments.

Future work could explore automated mechanisms for translating natural language constraints into formal language structures, further streamlining the process. Additionally, extending the framework to accommodate more complex planning scenarios, including those requiring multi-faceted decision-making and adaptation, represents a promising direction for advancing the capabilities of LLM-based agents.

Conclusion

The introduction of the Formal-LLM framework signifies a pivotal step forward in the development of more reliable and efficient LLM-based agents. By bridging the gap between the expressiveness of natural language and the precision of formal language, this approach enhances the controllability and effectiveness of automated planning processes, paving the way for broader applications and innovations in artificial intelligence.