PlanRAG: A Plan-then-Retrieval Augmented Generation for Generative Large Language Models as Decision Makers (2406.12430v1)

Abstract: In this paper, we conduct a study to utilize LLMs as a solution for decision making that requires complex data analysis. We define Decision QA as the task of answering the best decision, $d_{best}$, for a decision-making question $Q$, business rules $R$ and a database $D$. Since there is no benchmark that can examine Decision QA, we propose Decision QA benchmark, DQA. It has two scenarios, Locating and Building, constructed from two video games (Europa Universalis IV and Victoria 3) that have almost the same goal as Decision QA. To address Decision QA effectively, we also propose a new RAG technique called the iterative plan-then-retrieval augmented generation (PlanRAG). Our PlanRAG-based LM generates the plan for decision making as the first step, and the retriever generates the queries for data analysis as the second step. The proposed method outperforms the state-of-the-art iterative RAG method by 15.8% in the Locating scenario and by 7.4% in the Building scenario, respectively. We release our code and benchmark at https://github.com/myeon9h/PlanRAG.

An Analysis of "PlanRAG: A Plan-then-Retrieval Augmented Generation for Generative LLMs as Decision Makers"

The research paper “PlanRAG: A Plan-then-Retrieval Augmented Generation for Generative LLMs as Decision Makers” by Myeonghwa Lee, Seonho An, and Min-Soo Kim offers a detailed exploration into enhancing LLMs for the task of complex decision making. The authors introduce a new methodology, PlanRAG, to tackle Decision QA—a challenging task involving complex data analytics and reasoning.

Summary

Definition of Decision QA

Decision QA involves answering a decision-making question $Q$ using a set of business rules $R$ and a structured database $D$. The decision $d_{best}$ is derived after analyzing the relevant data and adhering to the business rules. To validate and benchmark Decision QA, the authors constructed a new dataset named DQA, inspired by decision-making scenarios in two strategy video games: Europa Universalis IV and Victoria 3.

Proposed Methodology: PlanRAG

PlanRAG is presented as an innovative augmentation of the Retrieval-Augmented Generation (RAG) framework. Unlike traditional iterative RAG methods, which primarily handle knowledge-based queries and are suboptimal for complex decision-making tasks, PlanRAG introduces a two-step iterative process:

1. Planning Step: The LLM generates an initial plan by examining the data schema and the query.
2. Retrieval Step: Queries are generated and data is fetched according to the plan.
3. Re-planning Step: If necessary, the LLM reassesses and refines the plan based on intermediate findings.

PlanRAG iteratively alternates between planning and retrieval until it reaches sufficient data to make a decision.

Experimental Setup and Evaluation

The experimental setup involved comparing several LLM configurations: a single-turn RAG model (SingleRAG-LM), an iterative RAG model (IterRAG-LM), and multiple variants of the PlanRAG model. All models utilized GPT-4 within a zero-shot setting to ensure generic applicability. Two scenarios from the DQA benchmark, Locating and Building, served as the example questions.

The proposed PlanRAG method outperformed the state-of-the-art RAG by 15.8% in the Locating scenario and 7.4% in the Building scenario. The significant accuracy improvement can be attributed to PlanRAG’s ability to understand the complexity of queries better and make effective data retrieval decisions.

Detailed Results

Main Results:

• Locating Scenario: PlanRAG achieved an accuracy of 64.3%, significantly higher than IterRAG's 48.5%.
• Building Scenario: PlanRAG achieved an accuracy of 45.0%, outperforming IterRAG's 37.6%.

Error Analysis:

• Failures were categorized into CAN (improper candidates), MIS (missed data analysis), DEEP (misuse of data/equations), QUR (query errors), and OTH (other errors). PlanRAG reduced CAN and MIS errors, illustrating its improved query understanding and data retrieval abilities.

Analysis for Single Retrieval (SR) and Multiple Retrieval (MR):

• PlanRAG showed a notable impact on SR questions, demonstrating its ability to handle underestimated yet complex queries effectively.

Implications and Future Directions

Practical Implications

The introduction of PlanRAG opens up opportunities for automating decision-making tasks in real-world scenarios such as business planning, resource allocation, and strategic decision making. The better performance in querying relevant data and iteratively refining the decision process suggests that PlanRAG can reduce the human oversight required in these domains.

Theoretical Implications

From a theoretical perspective, PlanRAG sets a precedent for integrating systematic planning within LLM frameworks, emphasizing the sophistication required for complex decision-making tasks. It underscores the importance of iterative refinement and query-efficient methods in improving the capabilities of generative models.

Future Developments

As the field progresses, the following future directions can be speculated:

1. Integration with Other Data Types: Future research could extend PlanRAG to handle hybrid databases and vector databases, enhancing its applicability in diverse scenarios.
2. Efficiency Enhancements: Developing fine-tuned models that better generate queries and optimize database interactions for specific tasks.
3. Multi-Model Frameworks: Evaluating PlanRAG within multi-model language frameworks could further refine its decision-making capabilities.
4. Addressing Bias and Hallucination: Handling inherent biases and hallucinations in LLMs remains crucial, particularly for sensitive decision-making tasks.

Conclusion

The paper contributes significantly to the intersection of LLM technology and decision-making tasks, offering a robust framework in PlanRAG. The enhanced performance in DQA tasks demonstrates its potential and paves the way for more nuanced applications of LLMs in complex and data-intensive decision-making scenarios.