Enhancing LLMs for Power System Simulations: A Feedback-driven Multi-agent Framework (2411.16707v3)

Abstract: The integration of experimental technologies with LLMs is transforming scientific research. It positions AI as a versatile research assistant rather than a mere problem-solving tool. In the field of power systems, however, managing simulations -- one of the essential experimental technologies -- remains a challenge for LLMs due to their limited domain-specific knowledge, restricted reasoning capabilities, and imprecise handling of simulation parameters. To address these limitations, this paper proposes a feedback-driven, multi-agent framework. It incorporates three proposed modules: an enhanced retrieval-augmented generation (RAG) module, an improved reasoning module, and a dynamic environmental acting module with an error-feedback mechanism. Validated on 69 diverse tasks from Daline and MATPOWER, this framework achieves success rates of 93.13% and 96.85%, respectively. It significantly outperforms ChatGPT 4o, o1-preview, and the fine-tuned GPT-4o, which all achieved a success rate lower than 30% on complex tasks. Additionally, the proposed framework also supports rapid, cost-effective task execution, completing each simulation in approximately 30 seconds at an average cost of 0.014 USD for tokens. Overall, this adaptable framework lays a foundation for developing intelligent LLM-based assistants for human researchers, facilitating power system research and beyond.

• The paper introduces a robust multi-agent framework that integrates enhanced retrieval, reasoning, and environmental feedback to improve LLM simulation accuracy.
• The framework employs adaptive query planning, chain-of-thought prompting, and dynamic error correction to significantly outperform state-of-the-art models.
• The approach paves the way for intelligent research assistants and automated power system simulations, advancing both theoretical and practical applications.

Enhancing LLMs for Power System Simulations: A Feedback-driven Multi-agent Framework

This paper presents a novel approach for enhancing the capabilities of LLMs in conducting power system simulations. Addressing the inherent limitations in LLMs such as restricted domain-specific knowledge, limited reasoning capabilities, and imprecise handling of simulation parameters, the authors propose a feedback-driven, multi-agent framework. The proposed framework consists of three key modules: an enhanced retrieval-augmented generation (RAG) module, an improved reasoning module, and a dynamic environmental acting module with a sophisticated error-feedback mechanism. These elements work collaboratively to enable LLMs to execute power system simulations more accurately and efficiently.

Framework Architecture

1. Enhanced RAG Module: The authors introduce a refined RAG module featuring adaptive query planning and the use of a triple-based structure for the knowledge base. This approach enhances retrieval accuracy by allowing LLMs to identify and interpret simulation functions and options more effectively than standard RAG methods. The triple-based structure, in particular, assists in capturing the nuanced logical relationships necessary for complex simulation tasks, therefore expanding the knowledge accessibility for the LLMs in a cost-efficient manner.
2. Enhanced Reasoning Module: To strengthen the reasoning capacity of LLMs, the module employs chain-of-thought prompting and few-shot prompting. This structured reasoning involves breaking down tasks into smaller, manageable parts, allowing LLMs to follow logical pathways and generate accurate simulation codes. The integration of dynamic retrieval knowledge and static basic knowledge ensures the LLMs maintain coherence and accuracy in code generation.
3. Environmental Acting Module with Feedback: This module allows the LLMs to interact directly with the simulation environment, receiving feedback to iteratively refine its outputs. By incorporating an error-correction mechanism, the framework enables LLMs to autonomously identify and rectify errors, significantly improving the reliability of simulation results.

Results and Evaluation

The framework was validated using two power system simulators, Daline, and MATPOWER, covering both familiar and novel tools for the LLMs. The framework achieved remarkable success rates of 93.13% for Daline and 96.85% for MATPOWER across various tasks, considerably outperforming state-of-the-art models like GPT-4 and o1-preview, which showed success rates as low as 0% on complex tasks. These results underscore the substantial improvement in simulation capability brought about by the proposed framework.

Implications and Future Directions

This research demonstrates the transformative potential of enhancing LLMs with a feedback-driven multi-agent framework in power system simulations. Practically, the ability to automate complex simulation tasks promises to significantly boost the productivity of researchers by allowing them to focus on innovative and conceptually demanding aspects of research. Theoretically, it highlights a significant step towards developing LLM-based intelligent research assistants equipped for domain-specific complex operations.

Future work could expand on automatic evaluation methods for unbenchmarked results and synchronize simulations across multiple tools, enhancing the framework's scalability and reliability. Additionally, integrating error-detection mechanisms could further refine the framework, flagging potential inaccuracies for researcher review to complement the high level of accuracy already demonstrated. Overall, the paper lays a robust foundation for continued advancements in AI-assisted research methodologies in the field of power systems and beyond.