Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

SGSH: Stimulate Large Language Models with Skeleton Heuristics for Knowledge Base Question Generation (2404.01923v1)

Published 2 Apr 2024 in cs.CL and cs.AI

Abstract: Knowledge base question generation (KBQG) aims to generate natural language questions from a set of triplet facts extracted from KB. Existing methods have significantly boosted the performance of KBQG via pre-trained LLMs (PLMs) thanks to the richly endowed semantic knowledge. With the advance of pre-training techniques, LLMs (e.g., GPT-3.5) undoubtedly possess much more semantic knowledge. Therefore, how to effectively organize and exploit the abundant knowledge for KBQG becomes the focus of our study. In this work, we propose SGSH--a simple and effective framework to Stimulate GPT-3.5 with Skeleton Heuristics to enhance KBQG. The framework incorporates "skeleton heuristics", which provides more fine-grained guidance associated with each input to stimulate LLMs to generate optimal questions, encompassing essential elements like the question phrase and the auxiliary verb.More specifically, we devise an automatic data construction strategy leveraging ChatGPT to construct a skeleton training dataset, based on which we employ a soft prompting approach to train a BART model dedicated to generating the skeleton associated with each input. Subsequently, skeleton heuristics are encoded into the prompt to incentivize GPT-3.5 to generate desired questions. Extensive experiments demonstrate that SGSH derives the new state-of-the-art performance on the KBQG tasks.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (35)
  1. Knowledge-augmented language model prompting for zero-shot knowledge graph question answering. arXiv preprint arXiv:2306.04136.
  2. Knowledge-enriched, type-constrained and grammar-guided question generation over knowledge bases. In Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, pages 2776–2786.
  3. Language models are few-shot learners. In Proceedings of the 34th Conference on Neural Information Processing Systems, NeurIPS 2020, pages 1877–1901.
  4. Toward subgraph-guided knowledge graph question generation with graph neural networks. IEEE Transactions on Neural Networks and Learning Systems, pages 1–12.
  5. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, pages 1126–1135.
  6. A survey on neural question generation: Methods, applications, and prospects. arXiv preprint arXiv:2402.18267.
  7. Diversifying question generation over knowledge base via external natural questions. arXiv preprint arXiv:2309.14362.
  8. Dsm: Question generation over knowledge base via modeling diverse subgraphs with meta-learner. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, pages 4194–4207.
  9. Improving multi-hop knowledge base question answering by learning intermediate supervision signals. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, WSDM 21, pages 553–561.
  10. Jointgt: Graph-text joint representation learning for text generation from knowledge graphs. In Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, pages 2526–2538.
  11. Difficulty-controllable multi-hop question generation from knowledge graphs. In Proceddings of the 18th International Semantic Web Conference, ISWC 2019, pages 382–398.
  12. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, pages 3045–3059.
  13. BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, pages 7871–7880.
  14. Self-checker: Plug-and-play modules for fact-checking with large language models. arXiv preprint arXiv:2305.14623.
  15. Jinggui Liang and Lizi Liao. 2023. Clusterprompt: Cluster semantic enhanced prompt learning for new intent discovery. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pages 10468–10481.
  16. Prompting large language models with chain-of-thought for few-shot knowledge base question generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, pages 4329–4343.
  17. Looking for a few good metrics: Rouge and its evaluation. In Ntcir workshop, pages 1–8.
  18. Generating questions for knowledge bases via incorporating diversified contexts and answer-aware loss. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, pages 2431–2441.
  19. Improving chatgpt prompt for code generation. arXiv preprint arXiv:2305.08360.
  20. What makes good in-context examples for gpt-3? In Proceedings of Deep Learning Inside Out: The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, DeeLIO@ACL 2022, pages 100–114.
  21. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, pages 11048–11064.
  22. Enhancing few-shot text-to-sql capabilities of large language models: A study on prompt design strategies. arXiv preprint arXiv:2305.12586.
  23. Training language models to follow instructions with human feedback. In Proceedings of the 36th Conference on Neural Information Processing Systems, NeurIPS 2022, pages 27730–27744.
  24. What in-context learning "learns" in-context: Disentangling task recognition and task learning. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8298–8319.
  25. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, ACL 2002, pages 311–318.
  26. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67.
  27. Interpretation of natural language rules in conversational machine reading. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018, pages 2087–2097.
  28. Open domain question answering using early fusion of knowledge bases and text. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018, pages 4231–4242.
  29. Alon Talmor and Jonathan Berant. 2018. The web as a knowledge-base for answering complex questions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, pages 641–651.
  30. Attention is all you need. In Proceedings of the 31st Conference on Neural Information Processing Systems, NeurIPS 2017, pages 5998–6008.
  31. Math word problem generation with mathematical consistency and problem context constraints. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, pages 5986–5999.
  32. Autoqgs: Auto-prompt for low-resource knowledge-based question generation from SPARQL. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pages 2250–2259.
  33. The value of semantic parse labeling for knowledge base question answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, pages 201–206.
  34. Calibrate before use: Improving few-shot performance of language models. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, pages 12697–12706.
  35. An interpretable reasoning network for multi-relation question answering. In Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, pages 2010–2022.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Shasha Guo (11 papers)
  2. Lizi Liao (44 papers)
  3. Jing Zhang (731 papers)
  4. Yanling Wang (14 papers)
  5. Cuiping Li (42 papers)
  6. Hong Chen (230 papers)
Citations (3)
View on Semantic Scholar