Triad: A Framework Leveraging a Multi-Role LLM-based Agent to Solve Knowledge Base Question Answering (2402.14320v6)
Abstract: Recent progress with LLM-based agents has shown promising results across various tasks. However, their use in answering questions from knowledge bases remains largely unexplored. Implementing a KBQA system using traditional methods is challenging due to the shortage of task-specific training data and the complexity of creating task-focused model structures. In this paper, we present Triad, a unified framework that utilizes an LLM-based agent with three roles for KBQA tasks. The agent is assigned three roles to tackle different KBQA subtasks: agent as a generalist for mastering various subtasks, as a decision maker for the selection of candidates, and as an advisor for answering questions with knowledge. Our KBQA framework is executed in four phases, involving the collaboration of the agent's multiple roles. We evaluated the performance of our framework using three benchmark datasets, and the results show that our framework outperforms state-of-the-art systems on the LC-QuAD and YAGO-QA benchmarks, yielding F1 scores of 11.8% and 20.7%, respectively.
- Dbpedia: A nucleus for a web of open data. In The Semantic Web, pages 722–735, Berlin, Heidelberg. Springer Berlin Heidelberg.
- Knowledge-augmented language model prompting for zero-shot knowledge graph question answering. arXiv preprint arXiv:2306.04136.
- Large language model for science: A study on p vs. np. arXiv preprint arXiv:2309.05689.
- A survey on in-context learning. arXiv preprint arXiv:2301.00234.
- Chatdb: Augmenting llms with databases as their symbolic memory. arXiv preprint arXiv:2306.03901.
- An empirical study of pre-trained language models in simple knowledge graph question answering. World Wide Web, pages 1–32.
- Answering natural language questions by subgraph matching over knowledge graphs. IEEE Transactions on Knowledge and Data Engineering, page 824–837.
- Edg-based question decomposition for complex question answering over knowledge bases. In The Semantic Web – ISWC 2021: 20th International Semantic Web Conference, ISWC 2021, Virtual Event, October 24–28, 2021, Proceedings, page 128–145, Berlin, Heidelberg. Springer-Verlag.
- Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213.
- Graphix-t5: Mixing pre-trained transformers with graph-aware layers for text-to-sql parsing. arXiv preprint arXiv:2301.07507.
- Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. arXiv preprint arXiv:2305.03111.
- Bolaa: Benchmarking and orchestrating llm-augmented autonomous agents. arXiv preprint arXiv:2308.05960.
- A universal question-answering platform for knowledge graphs. Proceedings of the ACM on Management of Data, 1(1):1–25.
- Chatgpt versus traditional question answering for knowledge graphs: Current status and future directions towards knowledge graph chatbots. arXiv preprint arXiv:2302.06466.
- Art: Automatic multi-step reasoning and tool-use for large language models. arXiv preprint arXiv:2303.09014.
- Yago 4: A reason-able knowledge base. In The Semantic Web, pages 583–596, Cham. Springer International Publishing.
- Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761.
- Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2303.17580.
- Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.11366.
- Make a choice! knowledge base question answering with in-context learning. arXiv preprint arXiv:2305.13972.
- Can chatgpt replace traditional kbqa models? an in-depth analysis of the question answering performance of the gpt llm family.
- Evaluation of chatgpt as a question answering system for answering complex questions. arXiv preprint arXiv:2303.07992.
- Lc-quad: A corpus for complex question answering over knowledge graphs. In The Semantic Web – ISWC 2017, pages 210–218, Cham. Springer International Publishing.
- 9th challenge on question answering over linked data (qald-9) (invited paper). In Semdeep/NLIWoD@ISWC.
- A survey on large language model based autonomous agents. arXiv preprint arXiv:2308.11432.
- React: Synergizing reasoning and acting in language models.