Language Modeling with Editable External Knowledge (2406.11830v1)

Abstract: When the world changes, so does the text that humans write about it. How do we build LLMs that can be easily updated to reflect these changes? One popular approach is retrieval-augmented generation, in which new documents are inserted into a knowledge base and retrieved during prediction for downstream tasks. Most prior work on these systems have focused on improving behavior during prediction through better retrieval or reasoning. This paper introduces ERASE, which instead improves model behavior when new documents are acquired, by incrementally deleting or rewriting other entries in the knowledge base each time a document is added. In two new benchmark datasets evaluating models' ability to answer questions about a stream of news articles or conversations, ERASE improves accuracy relative to conventional retrieval-augmented generation by 7-13% (Mixtral-8x7B) and 6-10% (Llama-3-8B) absolute. Code and data are available at https://github.com/belindal/ERASE

• The paper presents erase, a system that updates external knowledge bases on document acquisition to prevent stale information during predictions.
• It leverages dense vector retrieval and language model classification to decide whether to retain, edit, or delete facts.
• Experiments on clark-News and clark-Conversations demonstrate 7–13% accuracy improvements in factual QA tasks with state-of-the-art models.

LLMing with Editable External Knowledge

The paper entitled "LLMing with Editable External Knowledge" addresses the limitations of contemporary retrieval-augmented generation (RAG) models, which simply append new documents to an existing knowledge base (KB) but fail to update or maintain the consistency of the KB. This can result in the retrieval of stale or contradictory documents during prediction, leading to inaccurate LLM (LM) outputs. The authors propose a novel approach, named erase, which continuously updates the KB by rewriting or deleting outdated entries whenever new documents are added. This process ensures that the KB reflects the current state of the world, thereby enhancing the accuracy of the LLM during prediction.

Key Contributions

1. Problem Definition and Motivation:
   • The world and the textual descriptions of it are constantly evolving. Thus, a static KB, which is common in many RAG systems, can quickly become outdated and unreliable.
   • erase tackles this issue by updating the KB at the time of document acquisition rather than during prediction, thereby preventing the usage of stale information.
2. erase Methodology:
   • The erase system operates through three main steps: retrieving relevant facts from the KB (Step 1), deciding whether to keep, edit, or delete these facts based on the new document (Step 2), and adding any new facts extracted from the document (Step 3).
   • For retrieval, erase employs dense vector retrieval to fetch relevant facts from the KB.
   • For updating, it uses a LLM to classify whether a fact should be reinforced, kept unchanged, marked as false, or rewritten based on the new information. This ensures that the KB only retains facts that align with the current state.
3. Benchmark Datasets:
   • The authors introduce a new dataset, clark, encompassing two domains: clark-News and clark-Conversations. The clark-News dataset comprises timestamped news articles paired with factual QA pairs, while clark-Conversations includes long conversations where facts about participants evolve.
4. Experimental Results:
   • The experiments demonstrate that erase significantly improves accuracy compared to conventional RAG approaches, achieving absolute improvements of 7-13% with Mixtral-8x7B and 6-10% with Llama-3-8B in the factual QA domain.
   • Additionally, erase shows comparable results to baselines in multi-hop scenarios, which involve more complex inferences, indicating potential areas for future enhancement.

Practical and Theoretical Implications

The practical implications of this research are profound. By maintaining an up-to-date KB, downstream applications such as search engines, chatbots, and personal assistants can provide more reliable and accurate responses. Furthermore, the development of editable KB methodologies like erase can inform the future of adaptive AI systems that evolve alongside the data they process.

Theoretically, this work bridges a gap in the literature by not only focusing on improving retrieval and reasoning during prediction but emphasizing the importance of maintaining a continuously relevant and accurate KB. It challenges the notion that static KBs are sufficient for dynamic environments and highlights the role of continual learning in the context of LLMs.

Future Work

Future developments could concentrate on enhancing multi-hop reasoning capabilities of the retrieval and editing models, as the current implementation encounters challenges in this regard. Another promising direction is to refine methodologies for targeted interventions to ensure consistent accuracy across extensive timeframes and larger data scales. Additionally, improving the retrieval model to account for downstream facts affected by new inputs can further optimize the process.

Conclusion

This paper sets a new precedent in handling dynamic knowledge within LLMs by introducing erase. The method ensures that KBs are not only updated at the time of document acquisition but also that stale information is systematically erased, resulting in improved prediction accuracy. The introduction of two novel datasets for evaluating changing world facts reinforces the robustness of erase. Subsequent research can build upon these insights to develop even more sophisticated mechanisms for managing evolving knowledge bases.