Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

keqing: knowledge-based question answering is a nature chain-of-thought mentor of LLM (2401.00426v1)

Published 31 Dec 2023 in cs.CL and cs.AI

Abstract: LLMs have exhibited remarkable performance on various NLP tasks, especially for question answering. However, in the face of problems beyond the scope of knowledge, these LLMs tend to talk nonsense with a straight face, where the potential solution could be incorporating an Information Retrieval (IR) module and generating response based on these retrieved knowledge. In this paper, we present a novel framework to assist LLMs, such as ChatGPT, to retrieve question-related structured information on the knowledge graph, and demonstrate that Knowledge-based question answering (Keqing) could be a nature Chain-of-Thought (CoT) mentor to guide the LLM to sequentially find the answer entities of a complex question through interpretable logical chains. Specifically, the workflow of Keqing will execute decomposing a complex question according to predefined templates, retrieving candidate entities on knowledge graph, reasoning answers of sub-questions, and finally generating response with reasoning paths, which greatly improves the reliability of LLM's response. The experimental results on KBQA datasets show that Keqing can achieve competitive performance and illustrate the logic of answering each question.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (44)
  1. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  2. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021a.
  3. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022.
  4. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
  5. A survey of large language models. arXiv preprint arXiv:2303.18223, 2023.
  6. Exploring the limits of large scale pre-training. arXiv preprint arXiv:2110.02095, 2021.
  7. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560, 2022.
  8. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019.
  9. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
  10. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021.
  11. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837, 2022.
  12. Sources of hallucination by large language models on inference tasks. arXiv preprint arXiv:2305.14552, 2023.
  13. Augmented language models: a survey. arXiv preprint arXiv:2302.07842, 2023.
  14. Unik-qa: Unified representations of structured and unstructured knowledge for open-domain question answering. arXiv preprint arXiv:2012.14610, 2020.
  15. Few-shot in-context learning for knowledge base question answering. arXiv preprint arXiv:2305.01750, 2023a.
  16. In-context retrieval-augmented language models. arXiv preprint arXiv:2302.00083, 2023.
  17. Binding language models in symbolic languages. arXiv preprint arXiv:2210.02875, 2022.
  18. Question-answer sentence graph for joint modeling answer selection. arXiv preprint arXiv:2203.03549, 2022.
  19. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389, 2009.
  20. One question answering model for many languages with cross-lingual dense passage retrieval. Advances in Neural Information Processing Systems, 34:7547–7560, 2021.
  21. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213, 2022.
  22. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
  23. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://lmsys.org/blog/2023-03-30-vicuna/.
  24. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2303.17580, 2023.
  25. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.
  26. Knowledge base question answering by case-based reasoning over subgraphs. In International conference on machine learning, pages 4777–4793. PMLR, 2022.
  27. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.
  28. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  29. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906, 2020.
  30. Variational reasoning for question answering with knowledge graph. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
  31. The value of semantic parse labeling for knowledge base question answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 201–206, 2016.
  32. Structgpt: A general framework for large language model to reason over structured data. arXiv preprint arXiv:2305.09645, 2023.
  33. Don’t generate, discriminate: A proposal for grounding language models to real-world environments. arXiv preprint arXiv:2212.09736, 2022.
  34. Flexkbqa: A flexible llm-powered framework for few-shot knowledge base question answering. arXiv preprint arXiv:2308.12060, 2023b.
  35. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019.
  36. Key-value memory networks for directly reading documents. arXiv preprint arXiv:1606.03126, 2016.
  37. Open domain question answering using early fusion of knowledge bases and text. arXiv preprint arXiv:1809.00782, 2018.
  38. Pullnet: Open domain question answering with iterative retrieval on knowledge bases and text. arXiv preprint arXiv:1904.09537, 2019.
  39. Improving multi-hop question answering over knowledge graphs using knowledge base embeddings. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 4498–4507, 2020.
  40. Improving multi-hop knowledge base question answering by learning intermediate supervision signals. In Proceedings of the 14th ACM international conference on web search and data mining, pages 553–561, 2021.
  41. Query graph generation for answering multi-hop complex questions from knowledge bases. Association for Computational Linguistics, 2020.
  42. Retrack: A flexible and efficient framework for knowledge base question answering. In Proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing: system demonstrations, pages 325–336, 2021b.
  43. Tiara: Multi-grained retrieval for robust question answering over large knowledge bases. arXiv preprint arXiv:2210.12925, 2022.
  44. Decaf: Joint decoding of answers and logical forms for question answering over knowledge bases. arXiv preprint arXiv:2210.00063, 2022.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (8)
  1. Chaojie Wang (28 papers)
  2. Yishi Xu (11 papers)
  3. Zhong Peng (6 papers)
  4. Chenxi Zhang (17 papers)
  5. Bo Chen (309 papers)
  6. Xinrun Wang (39 papers)
  7. Lei Feng (190 papers)
  8. Bo An (127 papers)
Citations (16)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets