Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
166 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Reliable Academic Conference Question Answering: A Study Based on Large Language Model (2310.13028v2)

Published 19 Oct 2023 in cs.CL and cs.AI

Abstract: As the development of academic conferences fosters global scholarly communication, researchers consistently need to obtain accurate and up-to-date information about academic conferences. Since the information is scattered, using an intelligent question-answering system to efficiently handle researchers' queries and ensure awareness of the latest advancements is necessary. Recently, LLMs have demonstrated impressive capabilities in question answering, and have been enhanced by retrieving external knowledge to deal with outdated knowledge. However, these methods fail to work due to the lack of the latest conference knowledge. To address this challenge, we develop the ConferenceQA dataset, consisting of seven diverse academic conferences. Specifically, for each conference, we first organize academic conference data in a tree-structured format through a semi-automated method. Then we annotate question-answer pairs and classify the pairs into four different types to better distinguish their difficulty. With the constructed dataset, we further propose a novel method STAR (STructure-Aware Retrieval) to improve the question-answering abilities of LLMs, leveraging inherent structural information during the retrieval process. Experimental results on the ConferenceQA dataset show the effectiveness of our retrieval method. The dataset and code are available at https://github.com/zjukg/ConferenceQA.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (44)
  1. Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answering. arXiv preprint arXiv:2306.04136 (2023).
  2. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 (2022).
  3. A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity. arXiv:2302.04023 [cs.CL]
  4. Improving language models by retrieving from trillions of tokens. In International conference on machine learning. PMLR, 2206–2240.
  5. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901.
  6. Retrieval Augmented via Execution Guidance in Open-domain Table QA. In Proceedings of the 2022 5th International Conference on Algorithms, Computing and Artificial Intelligence. 1–6.
  7. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 (2022).
  8. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416 (2022).
  9. CiteSeer: An automatic citation indexing system. In Proceedings of the third ACM conference on Digital libraries. 89–98.
  10. Retrieval augmented language model pre-training. In International conference on machine learning. PMLR, 3929–3938.
  11. ChatDB: Augmenting LLMs with Databases as Their Symbolic Memory. arXiv:2306.03901 [cs.AI]
  12. Atlas: Few-shot Learning with Retrieval Augmented Language Models. arXiv:2208.03299 [cs.CL]
  13. Few-shot learning with retrieval augmented language models. arXiv preprint arXiv:2208.03299 (2022).
  14. Survey of hallucination in natural language generation. Comput. Surveys 55, 12 (2023), 1–38.
  15. Active retrieval augmented generation. arXiv preprint arXiv:2305.06983 (2023).
  16. Large language models are zero-shot reasoners. Advances in neural information processing systems 35 (2022), 22199–22213.
  17. Internet-augmented language models through few-shot prompting for open-domain question answering. arXiv preprint arXiv:2203.05115 (2022).
  18. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems 33 (2020), 9459–9474.
  19. S2ORC: The semantic scholar open research corpus. arXiv preprint arXiv:1911.02782 (2019).
  20. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837 (2022).
  21. RET-LLM: Towards a General Read-Write Memory for Large Language Models. arXiv:2305.14322 [cs.CL]
  22. Large dual encoders are generalizable retrievers. arXiv preprint arXiv:2112.07899 (2021).
  23. OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL]
  24. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35 (2022), 27730–27744.
  25. End-to-End Table Question Answering via Retrieval-Augmented Generation. arXiv preprint arXiv:2203.16714 (2022).
  26. Unifying Large Language Models and Knowledge Graphs: A Roadmap. arXiv preprint arXiv:2306.08302 (2023).
  27. RocketQA: An optimized training approach to dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2010.08191 (2020).
  28. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends® in Information Retrieval 3, 4 (2009), 333–389.
  29. Cristóbal Romero and Sebastián Ventura. 2017. Educational data science in massive open online courses. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 7, 1 (2017), e1187.
  30. Tarek Saier and Michael Färber. 2020. unarXive: a large scholarly data set with publications’ full-text, annotated in-text citations, and links to metadata. Scientometrics 125, 3 (2020), 3085–3108.
  31. Toolformer: Language Models Can Teach Themselves to Use Tools. arXiv:2302.04761 [cs.CL]
  32. Data science in data librarianship: Core competencies of a data librarian. Journal of Librarianship and Information Science 51, 3 (2019), 771–780.
  33. Knowledge graph-augmented language models for complex question answering. (2023).
  34. Replug: Retrieval-augmented black-box language models. arXiv preprint arXiv:2301.12652 (2023).
  35. LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971 [cs.CL]
  36. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv:2307.09288 [cs.CL]
  37. Rat-sql: Relation-aware schema encoding and linking for text-to-sql parsers. arXiv preprint arXiv:1911.04942 (2019).
  38. Microsoft academic graph: When experts are not enough. Quantitative Science Studies 1, 1 (2020), 396–413.
  39. KnowledGPT: Enhancing Large Language Models with Retrieval and Storage Access on Knowledge Bases. arXiv preprint arXiv:2308.11761 (2023).
  40. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems 35 (2022), 24824–24837.
  41. Retrieve-Rewrite-Answer: A KG-to-Text Enhanced LLMs Framework for Knowledge Graph Question Answering. arXiv preprint arXiv:2309.11206 (2023).
  42. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:2304.05302 (2023).
  43. The Effect of Metadata on Scientific Literature Tagging: A Cross-Field Cross-Model Study. In Proceedings of the ACM Web Conference 2023. 1626–1637.
  44. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint arXiv:1709.00103 (2017).

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - zjukg/ConferenceQA (2 stars)
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube