Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
110 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

KC-GenRe: A Knowledge-constrained Generative Re-ranking Method Based on Large Language Models for Knowledge Graph Completion (2403.17532v1)

Published 26 Mar 2024 in cs.AI

Abstract: The goal of knowledge graph completion (KGC) is to predict missing facts among entities. Previous methods for KGC re-ranking are mostly built on non-generative LLMs to obtain the probability of each candidate. Recently, generative LLMs have shown outstanding performance on several tasks such as information extraction and dialog systems. Leveraging them for KGC re-ranking is beneficial for leveraging the extensive pre-trained knowledge and powerful generative capabilities. However, it may encounter new problems when accomplishing the task, namely mismatch, misordering and omission. To this end, we introduce KC-GenRe, a knowledge-constrained generative re-ranking method based on LLMs for KGC. To overcome the mismatch issue, we formulate the KGC re-ranking task as a candidate identifier sorting generation problem implemented by generative LLMs. To tackle the misordering issue, we develop a knowledge-guided interactive training method that enhances the identification and ranking of candidates. To address the omission issue, we design a knowledge-augmented constrained inference method that enables contextual prompting and controlled generation, so as to obtain valid rankings. Experimental results show that KG-GenRe achieves state-of-the-art performance on four datasets, with gains of up to 6.7% and 7.7% in the MRR and Hits@1 metric compared to previous methods, and 9.0% and 11.1% compared to that without re-ranking. Extensive analysis demonstrates the effectiveness of components in KG-GenRe.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (55)
  1. TuckER: Tensor factorization for knowledge graph completion. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5185–5194. Association for Computational Linguistics.
  2. Translating embeddings for modeling multi-relational data. In Proceedings of the Advances in Neural Information Processing Systems 26: Annual Conference on Neural Information Processing Systems, NIPS’13, page 2787–2795. Curran Associates Inc.
  3. COMET: commonsense transformers for automatic knowledge graph construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4762–4779. Association for Computational Linguistics.
  4. Language models are few-shot learners. In Proceedings of the Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems.
  5. OKGIT: Open knowledge graph link prediction with implicit types. In Proceedings of the Findings of the Association for Computational Linguistics, pages 2546–2559. Association for Computational Linguistics.
  6. PairRE: Knowledge graph embeddings via paired relation vectors. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 4360–4369. Association for Computational Linguistics.
  7. Knowledge is flat: A seq2seq generative framework for various knowledge graph completion. In Proceedings of the 29th International Conference on Computational Linguistics, pages 4005–4017. International Committee on Computational Linguistics.
  8. Convolutional 2d knowledge graph embeddings. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence and 30th Innovative Applications of Artificial Intelligence Conference and 8th AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI’18/IAAI’18/EAAI’18, pages 1811–1818. AAAI Press.
  9. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314.
  10. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171–4186. Association for Computational Linguistics.
  11. Identifying relations for open information extraction. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1535–1545.
  12. Facc1: Freebase annotation of clueweb corpora, version 1 (release date 2013-06-26, format version 1, correction level 0). Note: http://lemurproject. org/clueweb09/FACC1/Cited by, 5:140.
  13. Care: Open knowledge graph embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 378–388.
  14. Lora: Low-rank adaptation of large language models. In Proceedings of the 10th International Conference on Learning Representations.
  15. Text-augmented open knowledge graph completion via pre-trained language models. In Proceedings of the Findings of the Association for Computational Linguistics, pages 11161–11180. Association for Computational Linguistics.
  16. Multi-task learning for knowledge graph completion with pre-trained language models. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1737–1743.
  17. Large language models are zero-shot reasoners. Proceedings of the Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems, 35:22199–22213.
  18. Cear: Cross-entity aware reranker for knowledge base completion. arXiv preprint arXiv:2104.08741.
  19. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 61–68.
  20. Robust knowledge graph completion with stacked convolutions and a student re-ranking network. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 1016–1029. Association for Computational Linguistics.
  21. Differentiating concepts and instances for knowledge graph embedding. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1971–1979. Association for Computational Linguistics.
  22. Do pre-trained models benefit knowledge graph completion? a reliable evaluation and a reasonable approach. In Proceedings of the Findings of the Association for Computational Linguistics, pages 3570–3581. Association for Computational Linguistics.
  23. Zero-shot listwise document reranking with a large language model. arXiv preprint arXiv:2305.02156.
  24. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048–11064. Association for Computational Linguistics.
  25. Training language models to follow instructions with human feedback. Proceedings of the Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems, 35:27730–27744.
  26. Rankvicuna: Zero-shot listwise document reranking with open-source large language models. arXiv preprint arXiv:2309.15088.
  27. Translating embeddings for knowledge graph completion with relation attention mechanism. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 4286–4292. International Joint Conferences on Artificial Intelligence Organization.
  28. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(1).
  29. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3982–3992. Association for Computational Linguistics.
  30. Sequence-to-sequence knowledge graph completion and question answering. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 2814–2828. Association for Computational Linguistics.
  31. Improving multi-hop question answering over knowledge graphs using knowledge base embeddings. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4498–4507. Association for Computational Linguistics.
  32. Modeling relational data with graph convolutional networks. In Proceedings of the 15th Extended Semantic Web Conference, pages 593–607. Springer International Publishing.
  33. JointLK: Joint reasoning with language models and knowledge graphs for commonsense question answering. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5049–5060. Association for Computational Linguistics.
  34. Rotate: Knowledge graph embedding by relational rotation in complex space. In Proceedings of the 7th International Conference on Learning Representations.
  35. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  36. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  37. Complex embeddings for simple link prediction. In Proceedings of the 33nd International Conference on Machine Learning, ICML’16, page 2071–2080. JMLR.org.
  38. Unsupervised embedding enhancements of knowledge graphs using textual associations. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 5218–5225. International Joint Conferences on Artificial Intelligence Organization.
  39. Structure-augmented text representation learning for efficient knowledge graph completion. In Proceedings of the Web Conference, pages 1737–1748.
  40. SimKGC: Simple contrastive knowledge graph completion with pre-trained language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 4281–4294. Association for Computational Linguistics.
  41. Instructuie: Multi-task instruction tuning for unified information extraction. arXiv preprint arXiv:2304.08085.
  42. A canonicalization-enhanced known fact-aware framework for open knowledge graph link prediction. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, pages 2332–2342. International Joint Conferences on Artificial Intelligence Organization.
  43. Knowledge base completion via search-based question answering. In Proceedings of the 23rd International Conference on World Wide Web, WWW ’14, page 515–526, New York, NY, USA. Association for Computing Machinery.
  44. Representation learning of knowledge graphs with entity descriptions. In Proceedings of the 30th AAAI Conference on Artificial Intelligence, AAAI’16, page 2659–2665. AAAI Press.
  45. Representation learning of knowledge graphs with hierarchical types. In Proceedings of the 25th International Joint Conference on Artificial Intelligence, IJCAI’16, page 2965–2971. AAAI Press.
  46. Open knowledge graph link prediction with segmented embedding. In Proceedings of the 2022 International Joint Conference on Neural Networks, pages 1–8.
  47. From discrimination to generation: Knowledge graph completion with generative transformer. In Companion Proceedings of the Web Conference, WWW ’22, page 162–165. Association for Computing Machinery.
  48. Can text-based knowledge graph completion benefit from zero-shot large language models? arXiv preprint arXiv:2310.08279.
  49. Knowledge graph contrastive learning for recommendation. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’22, page 1434–1443. Association for Computing Machinery.
  50. End-to-end case-based reasoning for commonsense knowledge base completion. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3491–3504. Association for Computational Linguistics.
  51. Kg-bert: Bert for knowledge graph completion. arXiv preprint arXiv:1909.03193.
  52. Exploring large language models for knowledge graph completion. arXiv preprint arXiv:2308.13916.
  53. Pretrain-KGE: Learning knowledge representation from pretrained language models. In Proceedings of the Findings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 259–266. Association for Computational Linguistics.
  54. Llms for knowledge graph construction and reasoning: Recent capabilities and future opportunities. arXiv preprint arXiv:2305.13168.
  55. Large language models for information retrieval: A survey. arXiv preprint arXiv:2308.07107.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Yilin Wang (156 papers)
  2. Minghao Hu (14 papers)
  3. Zhen Huang (114 papers)
  4. Dongsheng Li (240 papers)
  5. Dong Yang (163 papers)
  6. Xicheng Lu (7 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now