Papers
Topics
Authors
Recent
Search
2000 character limit reached

Aligning Large Language Models to a Domain-specific Graph Database for NL2GQL

Published 26 Feb 2024 in cs.CL, cs.AI, and cs.DB | (2402.16567v3)

Abstract: Graph Databases (Graph DB) find extensive application across diverse domains such as finance, social networks, and medicine. Yet, the translation of Natural Language (NL) into the Graph Query Language (GQL), referred to as NL2GQL, poses significant challenges owing to its intricate and specialized nature. Some approaches have sought to utilize LLMs to address analogous tasks like text2SQL. Nonetheless, in the realm of NL2GQL tasks tailored to a particular domain, the absence of domain-specific NL-GQL data pairs adds complexity to aligning LLMs with the graph DB. To tackle this challenge, we present a well-defined pipeline. Initially, we utilize ChatGPT to generate NL-GQL data pairs, leveraging the provided graph DB with self-instruction. Subsequently, we employ the generated data to fine-tune LLMs, ensuring alignment between LLMs and the graph DB. Moreover, we find the importance of relevant schema in efficiently generating accurate GQLs. Thus, we introduce a method to extract relevant schema as the input context. We evaluate our method using two carefully constructed datasets derived from graph DBs in the finance and medicine domains, named FinGQL and MediGQL. Experimental results reveal that our approach significantly outperforms a set of baseline methods, with improvements of 5.90 and 6.36 absolute points on EM, and 6.00 and 7.09 absolute points on EX for FinGQL and MediGQL, respectively.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (57)
  1. Qwen technical report. arXiv preprint arXiv:2309.16609.
  2. Demystifying graph databases: Analysis and taxonomy of data organization, system designs, and graph queries. ACM Computing Surveys, 56(2):1–40.
  3. Shuaichen Chang and Eric Fosler-Lussier. 2023. How to prompt llms for text-to-sql: A study in zero-shot, single-domain, and cross-domain settings. arXiv preprint arXiv:2305.11853.
  4. Huatuogpt-ii, one-stage training for medical adaption of llms.
  5. Bianque: Balancing the questioning and suggestion ability of health llms with multi-turn health conversations polished by chatgpt.
  6. Editing factual knowledge in language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6491–6506, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  7. Active prompting with chain-of-thought for large language models. arXiv preprint arXiv:2302.12246.
  8. A survey on in-context learning.
  9. Emilio Ferrara. 2023. Should chatgpt be biased? challenges and risks of bias in large language models. arXiv preprint arXiv:2304.03738.
  10. Text-to-sql empowered by large language models: A benchmark evaluation. arXiv preprint arXiv:2308.15363.
  11. Text-to-sql empowered by large language models: A benchmark evaluation.
  12. Retrieval-augmented generation for large language models: A survey.
  13. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997.
  14. Beyond iid: three levels of generalization for question answering on knowledge bases. In Proceedings of the Web Conference 2021, pages 3477–3488.
  15. Few-shot text-to-sql translation using structure and content prompt learning. Proceedings of the ACM on Management of Data, 1(2):1–28.
  16. Spcql: A semantic parsing dataset for converting natural language into cypher. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pages 3973–3977.
  17. A formal basis for the heuristic determination of minimum cost paths. IEEE transactions on Systems Science and Cybernetics, 4(2):100–107.
  18. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.
  19. Longquan Jiang and Ricardo Usbeck. 2022. Knowledge graph question answering datasets and their generalizability: Are they enough for future research? In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’22, page 3209–3218, New York, NY, USA. Association for Computing Machinery.
  20. Challenges and applications of large language models. arXiv preprint arXiv:2307.10169.
  21. Knowledge graph-augmented language models for knowledge-grounded dialogue generation. arXiv preprint arXiv:2305.18846.
  22. Knowledge graph-augmented language models for knowledge-grounded dialogue generation.
  23. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213.
  24. Flexkbqa: A flexible llm-powered framework for few-shot knowledge base question answering. arXiv preprint arXiv:2308.12060.
  25. Knowledge-aware graph-enhanced GPT-2 for dialogue state tracking. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7871–7881, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  26. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950–1965.
  27. Scalability and performance evaluation of graph database systems: A comparative study of neo4j, janusgraph, memgraph, nebulagraph, and tigergraph. In 2023 Second International Conference On Smart Technologies For Smart Nation (SmartTechCon), pages 537–542. IEEE.
  28. Unified structure generation for universal information extraction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5755–5772, Dublin, Ireland. Association for Computational Linguistics.
  29. Chatkbqa: A generate-then-retrieve framework for knowledge base question answering with fine-tuned large language models. arXiv preprint arXiv:2310.08975.
  30. Llm-assisted knowledge graph engineering: Experiments with chatgpt. arXiv preprint arXiv:2307.06917.
  31. Recent advances in natural language processing via large pre-trained language models: A survey. ACM Computing Surveys, 56(2):1–40.
  32. Ret-llm: Towards a general read-write memory for large language models. arXiv preprint arXiv:2305.14322.
  33. Experimental evaluation of graph databases: Janusgraph, nebula graph, neo4j, and tigergraph. Applied Sciences, 13(9):5770.
  34. Enhancing text-to-SQL capabilities of large language models: A study on prompt design strategies. In The 2023 Conference on Empirical Methods in Natural Language Processing.
  35. Jeff Z Pan. 2009. Resource description framework. In Handbook on ontologies, pages 71–90. Springer.
  36. Unifying large language models and knowledge graphs: A roadmap. IEEE Transactions on Knowledge and Data Engineering.
  37. Mohammadreza Pourreza and Davood Rafiei. 2023. Din-sql: Decomposed in-context learning of text-to-sql with self-correction.
  38. Improving multi-hop question answering over knowledge graphs using knowledge base embeddings. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4498–4507, Online. Association for Computational Linguistics.
  39. Knowledge graph-augmented language models for complex question answering. In Proceedings of the 1st Workshop on Natural Language Reasoning and Structured Explanations (NLRSE).
  40. Multilingual translation with extensible multilingual pretraining and finetuning.
  41. An empirical study on recent graph database systems. In Knowledge Science, Engineering and Management: 13th International Conference, KSEM 2020, Hangzhou, China, August 28–30, 2020, Proceedings, Part I 13, pages 328–340. Springer.
  42. Knowledgpt: Enhancing large language models with retrieval and storage access on knowledge bases. arXiv preprint arXiv:2308.11761.
  43. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560.
  44. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682.
  45. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824–24837. Curran Associates, Inc.
  46. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837.
  47. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.07864.
  48. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498, Online. Association for Computational Linguistics.
  49. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305.
  50. Llm-based sparql generation with selected schema from large scale knowledge base. In China Conference on Knowledge Graph and Semantic Computing, pages 304–316. Springer.
  51. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. arXiv preprint arXiv:1809.08887.
  52. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414.
  53. A survey of large language models. arXiv preprint arXiv:2303.18223.
  54. Cyspider: A neural semantic parsing corpus with baseline models for property graphs. In AI 2023: Advances in Artificial Intelligence, pages 120–132, Singapore. Springer Nature Singapore.
  55. Natural language query for technical knowledge graph navigation. In Australasian Conference on Data Mining, pages 176–191. Springer.
  56. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625.
  57. r3superscript𝑟3r^{3}italic_r start_POSTSUPERSCRIPT 3 end_POSTSUPERSCRIPT-nl2gql: A hybrid models approach for for accuracy enhancing and hallucinations mitigation.
Citations (3)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up