Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Leveraging Logical Rules in Knowledge Editing: A Cherry on the Top (2405.15452v2)

Published 24 May 2024 in cs.CL, cs.AI, and cs.LG

Abstract: Multi-hop Question Answering (MQA) under knowledge editing (KE) is a key challenge in LLMs. While best-performing solutions in this domain use a plan and solve paradigm to split a question into sub-questions followed by response generation, we claim that this approach is sub-optimal as it fails for hard to decompose questions, and it does not explicitly cater to correlated knowledge updates resulting as a consequence of knowledge edits. This has a detrimental impact on the overall consistency of the updated knowledge. To address these issues, in this paper, we propose a novel framework named RULE-KE, i.e., RULE based Knowledge Editing, which is a cherry on the top for augmenting the performance of all existing MQA methods under KE. Specifically, RULE-KE leverages rule discovery to discover a set of logical rules. Then, it uses these discovered rules to update knowledge about facts highly correlated with the edit. Experimental evaluation using existing and newly curated datasets (i.e., RKE-EVAL) shows that RULE-KE helps augment both performances of parameter-based and memory-based solutions up to 92% and 112.9%, respectively.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (52)
  1. Prompt-saw: Leveraging relation-aware graphs for textual prompt compression. arXiv preprint arXiv:2404.00489, 2024.
  2. Editing factual knowledge in language models. In Conference on Empirical Methods in Natural Language Processing, 2021.
  3. Recall and learn: Fine-tuning deep pretrained language models with less forgetting. In Conference on Empirical Methods in Natural Language Processing, 2020.
  4. Multi-hop question answering under temporal knowledge editing. arXiv preprint arXiv:2404.00492, 2024.
  5. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3, pages 265–284. Springer, 2006.
  6. Pokemqa: Programmable knowledge editing for multi-hop question answering. arXiv preprint arXiv:2312.15194, 2023.
  7. Improving sequential model editing with fact retrieval. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 11209–11224, Singapore, December 2023. Association for Computational Linguistics.
  8. Do language models have beliefs? methods for detecting, updating, and visualizing model beliefs. ArXiv, abs/2111.13654, 2021.
  9. Faithful question answering with monte-carlo planning. In Annual Meeting of the Association for Computational Linguistics, 2023.
  10. Wilke: Wise-layer knowledge editor for lifelong knowledge editing. ArXiv, abs/2402.10987, 2024.
  11. Differentially private natural language models: Recent advances and future directions. arXiv preprint arXiv:2301.09112, 2023.
  12. Improving faithfulness for vision transformers. arXiv preprint arXiv:2311.17983, 2023.
  13. Seat: stable and explainable attention. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37(11), pages 12907–12915, 2023.
  14. High dimensional differentially private stochastic optimization with heavy-tailed data. In Proceedings of the 41st ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of Database Systems, pages 227–236, 2022.
  15. Privacy-preserving sparse generalized eigenvalue problem. In International Conference on Artificial Intelligence and Statistics, pages 5052–5062. PMLR, 2023.
  16. Meta-learning online adaptation of language models. ArXiv, abs/2305.15076, 2023.
  17. Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey. ArXiv, abs/2212.10403, 2022.
  18. See the unseen: Better context-consistent knowledge-editing by noises. ArXiv, abs/2401.07544, 2024.
  19. Cheonsu Jeong. A study on the implementation of generative ai services using an enterprise data-based llm application architecture. arXiv preprint arXiv:2309.01105, 2023.
  20. Dense passage retrieval for open-domain question answering. ArXiv, abs/2004.04906, 2020.
  21. Decomposed prompting: A modular approach for solving complex tasks. ArXiv, abs/2210.02406, 2022.
  22. End-to-end neural entity linking. arXiv preprint arXiv:1808.07699, 2018.
  23. Faithful vision-language interpretation via concept bottleneck models. In The Twelfth International Conference on Learning Representations, 2023.
  24. Fast and exact rule mining with amie 3. The Semantic Web, 12123:36 – 52, 2020.
  25. A survey on complex knowledge base question answering: Methods, challenges and solutions. ArXiv, abs/2105.11644, 2021.
  26. Pmet: Precise model editing in a transformer. ArXiv, abs/2308.08742, 2023.
  27. A survey on multi-hop question answering and generation. ArXiv, abs/2204.09140, 2022.
  28. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359–17372, 2022.
  29. Mass-editing memory in a transformer. In The Eleventh International Conference on Learning Representations, 2022.
  30. Memory-based model editing at scale. ArXiv, abs/2206.06520, 2022.
  31. Training language models to follow instructions with human feedback. ArXiv, abs/2203.02155, 2022.
  32. Rnnlogic: Learning logic rules for reasoning on knowledge graphs. ArXiv, abs/2010.04029, 2020.
  33. J. Ross Quinlan. Learning logical definitions from relations. Machine Learning, 5:239–266, 1990.
  34. A survey of hallucination in large foundation models. arXiv preprint arXiv:2309.05922, 2023.
  35. Faster rates of private stochastic convex optimization. In International Conference on Algorithmic Learning Theory, pages 995–1002. PMLR, 2022.
  36. Rule: Neural-symbolic knowledge graph reasoning with rule embedding. ArXiv, abs/2210.14905, 2022.
  37. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
  38. Wikidata: a free collaborative knowledgebase. Communications of the ACM, 57(10):78–85, 2014.
  39. Differentially private (gradient) expectation maximization algorithm with statistical guarantees. arXiv preprint arXiv:2010.13520, 2020.
  40. Estimating smooth glm in non-interactive local differential privacy model with public unlabeled data. In Algorithmic Learning Theory, pages 1207–1213. PMLR, 2021.
  41. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. In Annual Meeting of the Association for Computational Linguistics, 2023.
  42. Chain of thought prompting elicits reasoning in large language models. ArXiv, abs/2201.11903, 2022.
  43. Transformers: State-of-the-art natural language processing. In Conference on Empirical Methods in Natural Language Processing, 2019.
  44. How does selection leak privacy: Revisiting private selection and improved results for hyper-parameter tuning. arXiv preprint arXiv:2402.13087, 2024.
  45. Practical differentially private and byzantine-resilient federated learning. Proceedings of the ACM on Management of Data, 1(2):1–26, 2023.
  46. An llm can fool itself: A prompt-based adversarial attack. arXiv preprint arXiv:2310.13345, 2023.
  47. Moral: Moe augmented lora for llms’ lifelong learning. arXiv preprint arXiv:2402.11260, 2024.
  48. Human-ai interactions in the communication era: Autophagy makes large models achieving local optima. arXiv preprint arXiv:2402.11271, 2024.
  49. Dialectical alignment: Resolving the tension of 3h and security threats of llms. arXiv preprint arXiv:2404.00486, 2024.
  50. Can we edit factual knowledge by in-context learning? ArXiv, abs/2305.12740, 2023.
  51. Mquake: Assessing knowledge editing in language models via multi-hop questions. arXiv preprint arXiv:2305.14795, 2023.
  52. Modifying memories in transformer models. ArXiv, abs/2012.00363, 2020.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (10)
  1. Keyuan Cheng (9 papers)
  2. Muhammad Asif Ali (18 papers)
  3. Shu Yang (178 papers)
  4. Haoyang Fei (2 papers)
  5. Ke Xu (309 papers)
  6. Lu Yu (87 papers)
  7. Lijie Hu (50 papers)
  8. Di Wang (407 papers)
  9. Yuxuan zhai (2 papers)
  10. Gang Lin (3 papers)
Citations (5)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets