Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Editing Language Model-based Knowledge Graph Embeddings (2301.10405v8)

Published 25 Jan 2023 in cs.CL, cs.AI, cs.DB, cs.IR, and cs.LG

Abstract: Recently decades have witnessed the empirical success of framing Knowledge Graph (KG) embeddings via LLMs. However, LLM-based KG embeddings are usually deployed as static artifacts, making them difficult to modify post-deployment without re-training after deployment. To address this issue, we propose a new task of editing LLM-based KG embeddings in this paper. This task is designed to facilitate rapid, data-efficient updates to KG embeddings without compromising the performance of other aspects. We build four new datasets: E-FB15k237, A-FB15k237, E-WN18RR, and A-WN18RR, and evaluate several knowledge editing baselines demonstrating the limited ability of previous models to handle the proposed challenging task. We further propose a simple yet strong baseline dubbed KGEditor, which utilizes additional parametric layers of the hypernetwork to edit/add facts. Our comprehensive experimental results reveal that KGEditor excels in updating specific facts without impacting the overall performance, even when faced with limited training resources. Code and datasets are available in https://github.com/zjunlp/PromptKG/tree/main/deltaKG.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (63)
  1. Translating Embeddings for Modeling Multi-relational Data. In NeurIPS.
  2. Language Models are Few-Shot Learners. In NeurIPS.
  3. Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases. In ACL.
  4. Editing Factual Knowledge in Language Models. In EMNLP.
  5. Knowledge Is Flat: A Seq2Seq Generative Framework for Various Knowledge Graph Completion. In COLING.
  6. KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction. In WWW.
  7. Evaluating the Ripple Effects of Knowledge Editing in Language Models. CoRR, abs/2307.12976.
  8. Knowledge Neurons in Pretrained Transformers. In ACL.
  9. Convolutional 2D Knowledge Graph Embeddings. In AAAI.
  10. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Burstein, J.; Doran, C.; and Solorio, T., eds., NAACL.
  11. Calibrating Factual Knowledge in Pretrained Language Models. In EMNLP, Findings of EMNLP.
  12. Online Updates of Knowledge Graph Embedding. In Complex Networks 2021, volume 1016 of Studies in Computational Intelligence, 523–535. Springer.
  13. Dissecting Recall of Factual Associations in Auto-Regressive Language Models. CoRR, abs/2304.14767.
  14. Pre-trained models: Past, present and future. AI Open.
  15. A divide and conquer framework for Knowledge Editing. Knowledge-Based Systems, 110826.
  16. Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models. CoRR, abs/2301.04213.
  17. Semi-Supervised Classification with Graph Convolutional Networks. In ICLR.
  18. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In ACL.
  19. Learning Entity and Relation Embeddings for Knowledge Graph Completion. In Bonet, B.; and Koenig, S., eds., AAAI.
  20. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys.
  21. INDIGO: GNN-Based Inductive Knowledge Graph Completion Using Pair-Wise Encoding. In NeurIPS, 2034–2045.
  22. P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks. In ACL.
  23. Do Pre-trained Models Benefit Knowledge Graph Completion? A Reliable Evaluation and a Reasonable Approach. In Findings of ACL.
  24. Locating and Editing Factual Knowledge in GPT. In NeurIPS.
  25. Fast Model Editing at Scale. In ICLR.
  26. Holographic Embeddings of Knowledge Graphs. In AAAI.
  27. A Three-Way Model for Collective Learning on Multi-Relational Data. In ICML.
  28. Unifying Large Language Models and Knowledge Graphs: A Roadmap. CoRR, abs/2306.08302.
  29. Language Models as Knowledge Bases? In EMNLP.
  30. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. J. Mach. Learn. Res., 21: 140:1–140:67.
  31. Sequence-to-Sequence Knowledge Graph Completion and Question Answering. In ACL.
  32. Editable Neural Networks. In ICLR.
  33. RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space. In ICLR.
  34. Stanford Alpaca: An Instruction-following LLaMA model. https://github.com/tatsu-lab/stanford˙alpaca.
  35. Representing Text for Joint Embedding of Text and Knowledge Bases. In EMNLP.
  36. LLaMA: Open and Efficient Foundation Language Models. CoRR, abs/2302.13971.
  37. Composition-based Multi-Relational Graph Convolutional Networks. In ICLR.
  38. Graph Attention Networks. In ICLR.
  39. Structure-Augmented Text Representation Learning for Efficient Knowledge Graph Completion. In WWW.
  40. SimKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language Models. In ACL.
  41. Knowledge Graph Embedding: A Survey of Approaches and Applications. TKDE.
  42. K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters. In Findings of ACL.
  43. KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation. Trans. Assoc. Comput. Linguistics, 9: 176–194.
  44. Language Models as Knowledge Embeddings. In IJCAI.
  45. Knowledge Graph Embedding by Translating on Hyperplanes. In AAAI.
  46. Incremental Update of Knowledge Graph Embedding by Rotating on Hyperplanes. In ICWS 2021, 516–524. IEEE.
  47. KC-GEE: Knowledge-based Conditioning for Generative Event Extraction.
  48. Towards relation extraction from speech. In Goldberg, Y.; Kozareva, Z.; and Zhang, Y., eds., Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, 10751–10762. Association for Computational Linguistics.
  49. Representation Learning of Knowledge Graphs with Entity Descriptions. In Schuurmans, D.; and Wellman, M. P., eds., AAAI.
  50. From Discrimination to Generation: Knowledge Graph Completion with Generative Transformer. In WWW.
  51. Embedding Entities and Relations for Learning and Inference in Knowledge Bases. In ICLR.
  52. Context-Aware Attentive Multilevel Feature Fusion for Named Entity Recognition. IEEE Transactions on Neural Networks and Learning Systems.
  53. HiTRANS: A Hierarchical Transformer Network for Nested Named Entity Recognition. In Findings of ACL.
  54. KG-BERT: BERT for Knowledge Graph Completion. CoRR, abs/1909.03193.
  55. Editing Large Language Models: Problems, Methods, and Opportunities. CoRR, abs/2305.13172.
  56. GLM-130B: An Open Bilingual Pre-trained Model. CoRR, abs/2210.02414.
  57. OntoProtein: Protein Pretraining With Gene Ontology Embedding. In ICLR.
  58. Relation Adversarial Network for Low Resource Knowledge Graph Completion. In WWW.
  59. Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learners. In ICLR.
  60. Learning Hierarchy-Aware Knowledge Graph Embeddings for Link Prediction. In AAAI.
  61. Pretrain-KGE: Learning Knowledge Representation from Pretrained Language Models. In Findings of EMNLP.
  62. Rethinking Graph Convolutional Networks in Knowledge Graph Completion. In WWW.
  63. A Survey of Large Language Models. CoRR, abs/2303.18223.
Citations (19)
View on Semantic Scholar

Summary

  • The paper introduces KGEditor, a novel baseline method that uses hypernetwork techniques to edit and add facts in KG embeddings without full retraining.
  • It constructs four specialized datasets to evaluate knowledge reliability, locality, and efficiency in updating KG embeddings.
  • Experimental results demonstrate that KGEditor improves update accuracy and preserves existing knowledge, enabling scalable and adaptive AI systems.

Editing LLM-based Knowledge Graph Embeddings

This paper introduces a novel approach to editing LLM-based Knowledge Graph Embeddings (KGEs), addressing the constraints of current methodologies that treat these embeddings as static artifacts. The proposed approach facilitates rapid, data-efficient updates to KG embeddings without necessitating a complete retraining, thus maintaining overall system performance while enabling flexible knowledge updates. The authors systematically develop novel datasets to evaluate the effectiveness of these edits, presenting both the challenges and potential of this innovative task in the field of knowledge representation.

Key Contributions and Methodology

The authors make three primary contributions to the field:

  1. Task Definition for Editing KGEs: The paper introduces new tasks—EDIT and ADD—aimed at modifying or augmenting KGEs seamlessly after deployment. These tasks address changing facts or integrating emerging knowledge, reflecting the dynamic nature of real-world data.
  2. Dataset Construction: The development of four datasets—E-FB15k237, A-FB15k237, E-WN18RR, and A-WN18RR—provides a robust foundation for evaluating the efficacy of KGE edits. These datasets allow for rigorous testing of the proposed methodologies, focusing on principles such as Knowledge Reliability, Knowledge Locality, and Knowledge Efficiency.
  3. KGEditor Implementation: The authors propose KGEditor, a baseline method utilizing additional parametric layers and hypernetwork-based techniques to edit/add facts within KGEs. KGEditor stands out for its ability to efficiently perform updates with minimal impact on the broader knowledge framework.

The paper meticulously compares the KGEditor with both existing and newly adapted methods, such as KE and MEND, each employing various mechanisms for knowledge modification. These comparisons highlight KGEditor’s improved performance with respect to both the accuracy of knowledge edits and the stability of unaffected knowledge.

Experimental Evaluation

The experimental results elucidate the advantages of using KGEditor for dynamic knowledge updates:

  • Knowledge Reliability: KGEditor demonstrates consistent success in updating specific facts without introducing significant errors, as shown by its superior Success@1 and Success@3 scores when tested against other baseline models.
  • Knowledge Locality and Efficiency: The paper discusses the importance of preserving existing knowledge (Knowledge Locality) while making updates, with KGEditor keeping a high RK@3 score, indicating minimal disturbance to non-updated facts.
  • Parameter Efficiency: By utilizing fewer tunable parameters, KGEditor efficiently balances the need for model flexibility with computational resource constraints, presenting a scalable solution for real-world applications.

Implications and Future Directions

The theoretical implications of this research are significant, suggesting that dynamic and efficient knowledge updates can enhance the applicability of KGEs in diverse AI applications, from information retrieval to recommendation systems. The practical implications involve the potential for deploying more adaptive and resilient AI systems that can quickly respond to evolving data landscapes.

Future developments and research could further explore complex knowledge structures and many-to-many relationship handling, as identified limitations of current methods. Additionally, integrating large-scale generative models like LLaMA or ChatGLM into this framework could further expand the scalability and applicability of these techniques.

In conclusion, the proposed editing task for LLM-based KGEs is a meaningful step forward, offering practical solutions and frameworks that align with the dynamic demands of modern machine learning environments. The ability to efficiently and accurately update knowledge graphs post-deployment represents a crucial advancement in the evolution of AI systems, promising enhanced adaptability and utility in real-world applications.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown