Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

KnowledgeNavigator: Leveraging Large Language Models for Enhanced Reasoning over Knowledge Graph (2312.15880v2)

Published 26 Dec 2023 in cs.CL

Abstract: LLM has achieved outstanding performance on various downstream tasks with its powerful natural language understanding and zero-shot capability, but LLM still suffers from knowledge limitation. Especially in scenarios that require long logical chains or complex reasoning, the hallucination and knowledge limitation of LLM limit its performance in question answering (QA). In this paper, we propose a novel framework KnowledgeNavigator to address these challenges by efficiently and accurately retrieving external knowledge from knowledge graph and using it as a key factor to enhance LLM reasoning. Specifically, KnowledgeNavigator first mines and enhances the potential constraints of the given question to guide the reasoning. Then it retrieves and filters external knowledge that supports answering through iterative reasoning on knowledge graph with the guidance of LLM and the question. Finally, KnowledgeNavigator constructs the structured knowledge into effective prompts that are friendly to LLM to help its reasoning. We evaluate KnowledgeNavigator on multiple public KGQA benchmarks, the experiments show the framework has great effectiveness and generalization, outperforming previous knowledge graph enhanced LLM methods and is comparable to the fully supervised models.

PDF HTML Abstract

The paper, "KnowledgeNavigator: Leveraging LLMs for Enhanced Reasoning over Knowledge Graph," introduces a novel framework called KnowledgeNavigator to enhance reasoning capabilities in LLMs when dealing with knowledge graph question answering (KGQA) tasks. The paper addresses the limitations of LLMs, such as inaccuracies and hallucinations arising from their struggle with complex logical sequences and constrained knowledge environments. KnowledgeNavigator aims to improve LLM reasoning by retrieving external knowledge from knowledge graphs to provide a structured approach to problem-solving.

Key Components of KnowledgeNavigator:

  1. Question Analysis:
    • This stage predicts the number of reasoning hops required in the knowledge graph, helping to refine and direct the retrieval process. It uses a pre-trained LLM (PLM) fine-tuned for estimating the reasoning depth needed to answer a question.
    • Similar questions are generated from the original query to guide the subsequent retrieval process, adding robustness to the retrieval of related entities and relations.
  2. Knowledge Retrieval:
    • A multi-hop retrieval process is conducted over the knowledge graph, where KnowledgeNavigator iteratively selects relations and entities pertinent to the question.
    • Filtering is performed using LLM to ensure only necessary information is gathered, taking advantage of weighted voting mechanisms that consider both the original and generated similar questions.
  3. Reasoning:
    • The retrieved knowledge is synthesized into natural language prompts, effectively avoiding the limitations of LLMs in processing graph-structured data.
    • LLMs, using this refined external knowledge, generate answers enhanced by accurate context and reduced redundancy.

Experimental Evaluation:

  • KnowledgeNavigator was evaluated using KGQA benchmarks, specifically the MetaQA and WebQSP datasets. The framework demonstrated competitive performance, surpassing models using similar approaches that integrate LLMs with knowledge retrieval. It outperformed several fully supervised models on multi-hop KGQA tasks.
  • It achieved significant improvements by extracting and utilizing structured knowledge graph data, reflecting its capability to handle complex reasoning tasks.

Ablation Studies and Error Analysis:

  • By testing different numbers of generated similar questions and various formats of knowledge representation, the paper showed the importance of natural language formatting and relation voting for efficient retrieval and reasoning.
  • Error analysis identified key areas such as relation selection and reasoning errors, providing insights into potential improvements for optimizing LLM performance on KGQA tasks.

Conclusion:

The paper concludes that enhancing LLMs with knowledge graphs can significantly curb limitations in complex reasoning and question-answering domains. KnowledgeNavigator, by integrating multi-hop reasoning and knowledge synthesis, leverages both structured knowledge graphs and LLMs to push the envelope in KGQA performance, demonstrating improvement over existing methods and revealing pathways for future enhancements in domain-specific applications.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (41)
  1. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023.
  2. Knowledge-augmented language model prompting for zero-shot knowledge graph question answering. arXiv preprint arXiv:2306.04136, 2023.
  3. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023.
  4. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544, 2013.
  5. Temporal knowledge graph completion: A survey. arXiv preprint arXiv:2201.08236, 2022.
  6. Introduction to neural network-based question answering over knowledge graphs. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 11(3):e1389, 2021.
  7. Systems engineering issues for industry applications of large language model. Applied Soft Computing, page 111165, 2023.
  8. Selection-inference: Exploiting large language models for interpretable logical reasoning. arXiv preprint arXiv:2205.09712, 2022.
  9. Go for a walk and arrive at the answer: Reasoning over paths in knowledge bases using reinforcement learning. arXiv preprint arXiv:1711.05851, 2017.
  10. Revisit input perturbation problems for llms: A unified robustness evaluation framework for noisy slot filling task. In CCF International Conference on Natural Language Processing and Chinese Computing, pages 682–694. Springer, 2023.
  11. Factkb: Generalizable factuality evaluation using language models enhanced with factual knowledge. arXiv preprint arXiv:2305.08281, 2023.
  12. Google. Freebase data dumps. https://developers.google.com/freebase/data, 2023.
  13. Improving multi-hop knowledge base question answering by learning intermediate supervision signals. In Proceedings of the 14th ACM international conference on web search and data mining, pages 553–561, 2021.
  14. Knowledge graph embedding based question answering. In Proceedings of the twelfth ACM international conference on web search and data mining, pages 105–113, 2019.
  15. A survey on knowledge graphs: Representation, acquisition, and applications. IEEE transactions on neural networks and learning systems, 33(2):494–514, 2021.
  16. Rho (ρ𝜌\rhoitalic_ρ): Reducing hallucination in open-domain dialogues with knowledge grounding. arXiv preprint arXiv:2212.01588, 2022.
  17. Unikgqa: Unified retrieval and reasoning for solving multi-hop question answering over knowledge graph. arXiv preprint arXiv:2212.00959, 2022.
  18. Structgpt: A general framework for large language model to reason over structured data. arXiv preprint arXiv:2305.09645, 2023.
  19. Attributed description logics: Reasoning on knowledge graphs. In IJCAI, pages 5309–5313, 2018.
  20. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
  21. Endo-sirna deficiency results in oocyte maturation failure and apoptosis in porcine oocytes. Reproduction, Fertility and Development, 29(11):2168–2174, 2017.
  22. Knowledge injection to counter large language model (llm) hallucination. In European Semantic Web Conference, pages 182–185. Springer, 2023.
  23. Skill: structured knowledge infusion for large language models. arXiv preprint arXiv:2205.08184, 2022.
  24. Unifying large language models and knowledge graphs: A roadmap. arXiv preprint arXiv:2306.08302, 2023.
  25. Improving multi-hop question answering over knowledge graphs using knowledge base embeddings. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 4498–4507, 2020.
  26. Open domain question answering using early fusion of knowledge bases and text. arXiv preprint arXiv:1809.00782, 2018.
  27. Think-on-graph: Deep and responsible reasoning of large language model with knowledge graph. arXiv preprint arXiv:2307.07697, 2023.
  28. Knowledge graph and knowledge reasoning: A systematic review. Journal of Electronic Science and Technology, 20(2):100159, 2022.
  29. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
  30. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
  31. Retrieve-rewrite-answer: A kg-to-text enhanced llms framework for knowledge graph question answering. arXiv preprint arXiv:2309.11206, 2023.
  32. Improving question answering over incomplete kbs with knowledge-aware reader. arXiv preprint arXiv:1905.07098, 2019.
  33. Enhancing key-value memory neural networks for knowledge based question answering. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2937–2947, 2019.
  34. Embedding entities and relations for learning and inference in knowledge bases. arXiv preprint arXiv:1412.6575, 2014.
  35. Semantic parsing for single-relation question answering. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 643–648, 2014.
  36. The value of semantic parse labeling for knowledge base question answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 201–206, 2016.
  37. Jaket: Joint pre-training of knowledge graph and language understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11630–11638, 2022.
  38. Variational reasoning for question answering with knowledge graph. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
  39. Ernie: Enhanced language representation with informative entities. arXiv preprint arXiv:1905.07129, 2019.
  40. Siren’s song in the ai ocean: A survey on hallucination in large language models. arXiv preprint arXiv:2309.01219, 2023.
  41. Multi-modal knowledge graph construction and application: A survey. IEEE Transactions on Knowledge and Data Engineering, 2022.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (8)
  1. Tiezheng Guo (2 papers)
  2. Qingwen Yang (2 papers)
  3. Chen Wang (599 papers)
  4. Yanyi Liu (6 papers)
  5. Pan Li (164 papers)
  6. Jiawei Tang (16 papers)
  7. Dapeng Li (32 papers)
  8. Yingyou Wen (2 papers)
Citations (12)
View on Semantic Scholar