Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
80 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

FlexKBQA: A Flexible LLM-Powered Framework for Few-Shot Knowledge Base Question Answering (2308.12060v3)

Published 23 Aug 2023 in cs.CL and cs.AI

Abstract: Knowledge base question answering (KBQA) is a critical yet challenging task due to the vast number of entities within knowledge bases and the diversity of natural language questions posed by users. Unfortunately, the performance of most KBQA models tends to decline significantly in real-world scenarios where high-quality annotated data is insufficient. To mitigate the burden associated with manual annotation, we introduce FlexKBQA by utilizing LLMs as program translators for addressing the challenges inherent in the few-shot KBQA task. Specifically, FlexKBQA leverages automated algorithms to sample diverse programs, such as SPARQL queries, from the knowledge base, which are subsequently converted into natural language questions via LLMs. This synthetic dataset facilitates training a specialized lightweight model for the KB. Additionally, to reduce the barriers of distribution shift between synthetic data and real user questions, FlexKBQA introduces an executionguided self-training method to iterative leverage unlabeled user questions. Furthermore, we explore harnessing the inherent reasoning capability of LLMs to enhance the entire framework. Consequently, FlexKBQA delivers substantial flexibility, encompassing data annotation, deployment, and being domain agnostic. Through extensive experiments on GrailQA, WebQSP, and KQA Pro, we observe that under the few-shot even the more challenging zero-shot scenarios, FlexKBQA achieves impressive results with a few annotations, surpassing all previous baselines and even approaching the performance of supervised models, achieving a remarkable 93% performance relative to the fully-supervised models. We posit that FlexKBQA represents a significant advancement towards exploring better integration of large and lightweight models. The code is open-sourced.

PDF HTML Abstract

An Academic Overview of FlexKBQA: A Flexible LLM-Powered Framework for Few-Shot Knowledge Base Question Answering

This paper presents FlexKBQA, a novel framework leveraging LLMs, specifically designed to address the challenges of few-shot knowledge base question answering (KBQA). KBQA is a complex task, demanding the conversion of natural language questions into structured queries executable on a knowledge base. The traditional requirement for extensive manual annotation poses significant impediments for deploying KBQA systems in real-world scenarios, particularly when high-quality labeled data is sparse.

Framework Design and Methodology

FlexKBQA mitigates the annotation burden by utilizing LLMs as program translators, converting synthetic programs from the knowledge base into natural language questions, which facilitates the training of lightweight models. This novel approach doctrinally contrasts with conventional methods that rely on LLMs for in-context learning of question-to-program mapping. The framework consists of key components designed to improve performance and flexibility:

  • Automatic Program Sampling: This component employs automated algorithms to create diverse program templates, ensuring a comprehensive sample space for program conversion. It uses a step-wise grounding technique to iteratively determine variable values, enabling the derivation of executable programs from structured query templates.
  • Low-Resource Program Translation: LLMs are employed to translate sampled programs into natural language questions, thereby utilizing their generative capabilities. The translation process is facilitated through a structured prompt containing directives and seed pairs of question-program exemplars.
  • Execution-Guided Self-Training (EGST): To address the distribution shift between synthetic data and real-world user queries, EGST uses iterative self-training with unlabeled user questions, leveraging execution-guided filtering mechanisms to enhance data purity and model robustness.
  • Inherent Reasoning Augmentation: This augmentation utilizes the internal knowledge capabilities of LLMs, providing a secondary method to syntactically generated programs, enhancing both training data quality and inference accuracy.

Experimental Results and Contributions

FlexKBQA was extensively evaluated on diverse datasets including GrailQA, WebQSP, and KQA Pro, demonstrating robust performance in few-shot settings, often surpassing existing baselines and achieving near-parity with supervised methods. Notably, it achieved an impressive 93% of supervised performance, indicating its efficacy in real-world applications with limited annotated data.

The paper emphasizes three dimensions of flexibility offered by FlexKBQA:

  1. Data Annotation Flexibility: Requiring minimal annotated pairs, the framework provides a scalable solution for diverse KBs.
  2. Domain-Agnostic Flexibility: Applicable across different KBs and program formats, FlexKBQA alleviates distribution shift challenges through EGST.
  3. Deployment Flexibility: Lightweight models ensure deployability advantages over large, closed-source LLMs, enabling seamless integration of domain-specific knowledge through fine-tuning.

Theoretical and Practical Implications

The paper enhances the theoretical framework of KBQA by demonstrating a synergistic relationship between LLMs and lightweight models. The practical implications extend to efficient deployment in low-resource environments, where the integration of LLM-generated synthetic data provides a feasible path for robust KBQA system development.

Future Directions

FlexKBQA opens avenues for future exploration into zero-shot KBQA settings, enriching the methodological toolkit for knowledge base interactions. Further research into batch prompting and advanced semantic filtering methods could refine its capabilities and extend its applicability across broader language understanding tasks.

Overall, FlexKBQA represents a significant step toward advancing KBQA methodology, particularly in scenarios constrained by limited training annotations, ensuring enhanced adaptability, and deployment efficiency across knowledge-rich environments.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (38)
  1. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, 1533–1544.
  2. KQA Pro: A Dataset with Explicit Compositional Programs for Complex Question Answering over Knowledge Base. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 6101–6119.
  3. Program transfer for answering complex questions over knowledge bases. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 8128–8140.
  4. Evaluating Large Language Models Trained on Code. arXiv:2107.03374.
  5. ReTraCk: A flexible and efficient framework for knowledge base question answering. In Proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing: system demonstrations, 325–336.
  6. Batch Prompting: Efficient Inference with Large Language Model APIs. arXiv:2301.08721.
  7. Binding Language Models in Symbolic Languages. arXiv:2210.02875.
  8. Case-based Reasoning for Natural Language Queries over Knowledge Bases. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 9594–9611.
  9. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171–4186. Minneapolis, Minnesota: Association for Computational Linguistics.
  10. FewRel 2.0: Towards More Challenging Few-Shot Relation Classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 6250–6255.
  11. Don’t Generate, Discriminate: A Proposal for Grounding Language Models to Real-World Environments. arXiv:2212.09736.
  12. Beyond iid: three levels of generalization for question answering on knowledge bases. In Proceedings of the Web Conference 2021, 3477–3488.
  13. Knowledge Base Question Answering: A Semantic Parsing Perspective. arXiv:2209.04994.
  14. ArcaneQA: Dynamic Program Induction and Contextualized Encoding for Knowledge Base Question Answering. In Proceedings of the 29th International Conference on Computational Linguistics, 1718–1731.
  15. Distilling the Knowledge in a Neural Network. arXiv:1503.02531.
  16. A Survey on Complex Knowledge Base Question Answering: Methods, Challenges and Solutions. arXiv:2105.11644.
  17. Query Graph Generation for Answering Multi-hop Complex Questions from Knowledge Bases. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 969–974.
  18. Efficient One-Pass End-to-End Entity Linking for Questions. arXiv:2010.02413.
  19. Few-shot In-context Learning for Knowledge Base Question Answering. arXiv:2305.01750.
  20. Effective few-shot named entity linking by meta-learning. In 2022 IEEE 38th International Conference on Data Engineering (ICDE), 178–191. IEEE.
  21. Toward a Unified Framework for Unsupervised Complex Tabular Reasoning. In 2023 IEEE 39th International Conference on Data Engineering (ICDE), 1691–1704. IEEE.
  22. Generating Training Data with Language Models: Towards Zero-Shot Language Understanding. arXiv:2202.04538.
  23. Lever: Learning to verify language-to-code generation with execution. In International Conference on Machine Learning, 26106–26128. PMLR.
  24. OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774.
  25. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. arXiv:1908.10084.
  26. CLASP: Few-Shot Cross-Lingual Data Augmentation for Semantic Parsing. arXiv:2210.07074.
  27. LINGUIST: Language Model Instruction Tuning to Generate Annotated Utterances for Intent Classification and Slot Tagging. arXiv:2209.09900.
  28. Generating Datasets with Pretrained Language Models. arXiv:2104.07540.
  29. Modeling relational data with graph convolutional networks. In The Semantic Web: 15th International Conference, ESWC 2018, Heraklion, Crete, Greece, June 3–7, 2018, Proceedings 15, 593–607. Springer.
  30. TIARA: Multi-grained Retrieval for Robust Question Answering over Large Knowledge Bases. arXiv:2210.12925.
  31. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), 11523–11530. IEEE.
  32. On generating characteristic-rich question sets for qa evaluation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 562–572.
  33. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35: 24824–24837.
  34. ZeroGen: Efficient Zero-shot Learning via Dataset Generation. arXiv:2202.07922.
  35. Generating Data for Symbolic Language with Large Language Models. arXiv:2305.13917.
  36. RNG-KBQA: Generation Augmented Iterative Ranking for Knowledge Base Question Answering. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 6032–6043.
  37. The value of semantic parse labeling for knowledge base question answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 201–206.
  38. DecAF: Joint Decoding of Answers and Logical Forms for Question Answering over Knowledge Bases. arXiv:2210.00063.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (8)
  1. Zhenyu Li (120 papers)
  2. Sunqi Fan (4 papers)
  3. Yu Gu (218 papers)
  4. Xiuxing Li (11 papers)
  5. Zhichao Duan (8 papers)
  6. Bowen Dong (27 papers)
  7. Ning Liu (199 papers)
  8. Jianyong Wang (38 papers)
Citations (47)
View on Semantic Scholar