Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Knowledge Graph Enhanced Large Language Model Editing (2402.13593v1)

Published 21 Feb 2024 in cs.CL

Abstract: LLMs are pivotal in advancing NLP tasks, yet their efficacy is hampered by inaccuracies and outdated knowledge. Model editing emerges as a promising solution to address these challenges. However, existing editing methods struggle to track and incorporate changes in knowledge associated with edits, which limits the generalization ability of postedit LLMs in processing edited knowledge. To tackle these problems, we propose a novel model editing method that leverages knowledge graphs for enhancing LLM editing, namely GLAME. Specifically, we first utilize a knowledge graph augmentation module to uncover associated knowledge that has changed due to editing, obtaining its internal representations within LLMs. This approach allows knowledge alterations within LLMs to be reflected through an external graph structure. Subsequently, we design a graph-based knowledge edit module to integrate structured knowledge into the model editing. This ensures that the updated parameters reflect not only the modifications of the edited knowledge but also the changes in other associated knowledge resulting from the editing process. Comprehensive experiments conducted on GPT-J and GPT-2 XL demonstrate that GLAME significantly improves the generalization capabilities of post-edit LLMs in employing edited knowledge.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (27)
  1. Knowledgeable or educated guess? revisiting language models as knowledge bases. In Annual Meeting of the Association for Computational Linguistics, pages 1860–1874.
  2. Evaluating the ripple effects of knowledge editing in language models. arXiv preprint arXiv:2307.12976.
  3. Knowledge neurons in pretrained transformers. In Annual Meeting of the Association for Computational Linguistics, pages 8493–8502.
  4. Editing factual knowledge in language models. In Conference on Empirical Methods in Natural Language Processing, pages 6491–6506.
  5. Dissecting recall of factual associations in auto-regressive language models. In Conference on Empirical Methods in Natural Language Processing, page 12216–12235.
  6. Transformer feed-forward layers are key-value memories. In Conference on Empirical Methods in Natural Language Processing, pages 5484–5495.
  7. A survey on knowledge graphs: Representation, acquisition, and applications. IEEE transactions on neural networks and learning systems, 33(2):494–514.
  8. Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations.
  9. Feed-forward blocks control contextualization in masked language models. arXiv preprint arXiv:2302.00456.
  10. Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In International Conference on Learning Representations.
  11. Are we really making much progress? revisiting, benchmarking and refining heterogeneous graph neural networks. In Conference On Knowledge Discovery and Data Mining, page 1150–1160.
  12. Locating and editing factual associations in gpt. Annual Conference on Neural Information Processing Systems, 35:17359–17372.
  13. Mass-editing memory in a transformer. In International Conference on Learning Representations.
  14. Fast model editing at scale. In International Conference on Learning Representations.
  15. Memory-based model editing at scale. In International Conference on Machine Learning, pages 15817–15831. PMLR.
  16. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Annual Conference on Neural Information Processing Systems, pages 8024–8035.
  17. Judea Pearl. 2022. Direct and indirect effects. In Probabilistic and causal inference: the works of Judea Pearl, pages 373–392.
  18. Language models are unsupervised multitask learners.
  19. Modeling relational data with graph convolutional networks. In Extended Semantic Web Conference, pages 593–607.
  20. Attention is all you need. Annual Conference on Neural Information Processing Systems.
  21. Investigating gender bias in language models using causal mediation analysis. Annual Conference on Neural Information Processing Systems, 33:12388–12401.
  22. Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/mesh-transformer-jax.
  23. Deep graph library: Towards efficient and scalable deep learning on graphs. International Conference on Learning Representations Workshop on Representation Learning on Graphs and Manifolds.
  24. Editing large language models: Problems, methods, and opportunities. In Conference on Empirical Methods in Natural Language Processing, pages 10222–10240.
  25. A survey of large language models. arXiv preprint arXiv:2303.18223.
  26. MQuAKE: Assessing knowledge editing in language models via multi-hop questions. In Conference on Empirical Methods in Natural Language Processing, page 15686–15702.
  27. Modifying memories in transformer models. arXiv preprint arXiv:2012.00363.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Mengqi Zhang (48 papers)
  2. Xiaotian Ye (6 papers)
  3. Qiang Liu (405 papers)
  4. Pengjie Ren (95 papers)
  5. Shu Wu (109 papers)
  6. Zhumin Chen (78 papers)
Citations (10)
View on Semantic Scholar